George Davey Smith challenges the findings of a new BMJ paper claiming that eating dark chocolate protects against Type 2 diabetes.
Xmas is supposed to be a time of goodwill, and medical journals like the BMJ print seasonal papers not intended to be taken too seriously. The topic of one of the latest papers – chocolate – has been the topic of Xmas BMJ papers in the past, but it appears this is intended to be a serious paper; it is certainly not entertaining. The paper suggests that dark chocolate can protect against type 2 diabetes. At the risk of being a Grinch at Christmas, it has to be said that this is, frankly, a meaningless paper. Indeed it is surprising to see such a paper published in 2024 in a supposedly reputable journal.
As with the vast array of dietary factors investigated in the studies used in this paper the authors find something which appears to be protective of future disease, in this case dark chocolate supposedly preventing type 2 diabetes. The authors carry out a range of statistical analyses supposedly adjusting for confounding factors, and report that a residual “statistically significant” finding remains, and then go on to imply this could indicate protection against diabetes by increasing dark chocolate consumption. In the same studies when such findings have been followed by randomized controlled trials (RCTs) – which is relatively rarely – they have consistently found no actual effect. In those studies it is clear that the observations just reflected residual confounding and selection bias.
For examples of these see the first few minutes of the recent inaugural Bennett Lecture at Ben Goldacre’s Bennett Institute for Applied Data Science. This shows how in the same studies and using the same methods more than 30 years ago (and with some of the same authors) showed a large protective effect of vitamin E supplement use for coronary heart disease. This was later followed by a substantial number of RCTs finding absolutely no such effect. The same sad story has been repeated again and again, as documented in the beginning of the lecture and the first couple of pages of this paper and this old cartoon (reproduced from this paper).
That consumers of milk chocolate differ, on average, from those of dark chocolate will not surprise anyone. As much research has shown (e.g. this paper), dark chocolate consumption is higher among individuals in higher-income groups. Studies attribute this pattern to the greater availability of premium and fine chocolate varieties in affluent markets and the appeal of dark chocolate’s perceived health benefits. These perceived benefits align with the lifestyle preferences of wealthier (and healthier) demographics.
Given this, it is unsurprising that, in the study under discussion, participants consuming five servings or more per week of milk chocolate were twice as likely to be current smokers than those reporting one or fewer serving per week. Conversely, for dark chocolate, it was those reporting one serving or fewer per month who were most likely to be current smokers.
Whilst current smoking was relatively uncommon in the study population, the point is there will clearly be a large difference in many health-related factors that are not measured. This cannot be taken into account with a handful of confounders, and measured with error, as it is simple to demonstrate, for example in this paper and in this excellent recent talk by the eminent epidemiologist Sir Richard Peto.
The repeated failure of RCTs that test hypothesis from these studies, mentioned above, confirms that this is the case. Indeed, a large RCT has failed to demonstrate any protection at all from cocoa extract supplementation in type 2 diabetes
The interesting questions regarding this paper relate to the sociology of science: why do journals like the BMJ continue to publish such likely spurious findings, and why do authors continue to turn them out? Presumably newsworthiness and citations are at least part of the answer, but this tendency is harmful to nutritional science (and to science in general), and should be seriously combatted.
The paper could provide excellent teaching material in elementary epidemiology classes, it demonstrates so many problematic aspects in much current population health research. It is yet another contribution to “Today’s random medical news”.