You can read the Worst US Science Stories of 2007, winners of the STATS Dubious Data Awards. Trevor Butterworth of the Statistical Assessment Service
reports:
2008 seemed obsessed with the coming of the chemicapocalypse – the increasing fear that everything in the modern world is toxic and out to make us ill, disrupt our hormones, bend our genders, and make us infertile. In 2007, activists and journalists attributed all manner of health problems to absorbing tiny amounts of chemicals from everyday objects, even though the science is tentative, the evidence thin, and the risk of something materially bad happening to you hypothetical.
If it sounds suspicious, ban itIn June, San Francisco’s mayor, Gavin Newsom, decided to ban plastic water bottles, in part because of concerns about recycling, which was reasonable enough, and in part because they contained “toxic” vinyl softeners known as phthalates, which was, at least metaphorically, garbage. The mayor – and the journalists who dutifully conveyed his fears to the public – seemed oblivious to the fact that plastic bottles do not contain phthalates; they are, instead, made with a polyester called polyethylene terephthalate, which is something quite different even though it seems to sound similar. But that’s chemistry for you. Poylethylene terephthalate, or PET for short, is not considered a health hazard by any regulatory agency in the world.
Perhaps a refresher course in puberty?Phthalatophobia, a subcategory of chemophobia (the fear of chemicals), led the media to make all sorts of remarkable claims in 2007, none more ballsy, perhaps, than Time magazine’s decision to advance puberty beyond the bounds of biological plausibility with the claim, in September, that inhaling phthalates from air fresheners could decrease sperm levels in infants.
Perhaps, Time was demonstrating that the mere act of reporting on toxic chemicals can cause mental derangement, as a) infants don’t produce sperm and b), the author of the study on phthalates in air fresheners, Dr. Gina Solomon of the Natural Resources Defense Council, admitted that had no “clear cut evidence here for health effects.” This comment was something of a let down from urgent wording of the NRDC press release, which claimed that phthalates were “particularly dangerous for young children and unborn babies.”
According to Google News, there were, on average, 7.7 news stories about or referencing phthalates every day during 2007. And yet despite such concern, it was far from clear what people should really be concerned about in terms of an actual risk of something – anything – happening. California banned phthalates in children’s toys in October, even though the Consumer Product Safety Commission in the U.S. and the European Union’s scientific risk assessment found no cause for worry (Europe, however, had already ignored its scientists and banned phthalates in toys too). More to the point, the National Institutes for Health found that children’s exposure to phthalates was overwhelmingly through food and dust, so banning toys will have no discernible effect on exposure regardless of whether there’s a risk or not.
Some phthalates have been shown at very high levels to harm laboratory animals, but then you can make rodents sick if you give them too much of anything. One study has drawn a statistical association between exposure to some phthalates in the womb and borderline changes in genital development. But contrary to the way the media have reported this study, the children were all healthy and had normal reproductive functioning. Even the Guardian newspaper, which is ardently pro green, concluded in its “Bad Science” column (written by an actual doctor) that the data on phthalates was being “overstated.”
As for air fresheners, the NRDC only measured the presence of phthalates inside the product. As to how much evaporated into the air and was likely to be absorbed by a passing human, there was nothing. The Environmental Protection Agency has since turned down the NRDC’s petition to examine the safety of air fresheners, although the agency does note that they are highly flammable – and will likely kill you if you eat one.
So, um, don’t eat air fresheners.
iFear AppleIf one was to pick one of the key flaws in the way the media reported the risk from chemicals it would have to be the absence of any meaningful measurement. Few journalists, when faced with a press release or a study by some group claiming some new threat to our collective well being, say, “show me the numbers.” But without numbers, it’s impossible to assess just what the risk is.
Such penetrating questioning might have spared Apple computers from a whirlwind of negative publicity after Greenpeace ranked the company bottom in its list of enviro-friendly computer companies because its phones and computers contained toxic chemicals, like brominated fire retardants.
Most reporters failed to note that Greenpeace didn’t measure how much of these chemicals were in Apple’s products, and whether they leached out in a way that could be dangerous. Moreover, in the case of the fire retardant deca-BDE – the most widely used flame-retardant in consumer products – Greenpeace conveniently overlooked the fact that the European Union conducted a 10-year risk assessment, evaluating 588 studies on the chemical, and found it posed no health risk. The press didn’t catch them on this either.
Yes, there are health and environmental concerns over two other brominated fire retardants, “penta” and “octa,” but their use was phased out in the U.S. several years ago. A special exemption is required from the EPA to import any item containing these chemicals.
Attack of the Killer MattressesFire-retardants are also a great example of how risk assessment requires balancing the risk of using certain chemicals against the risk of not using those chemicals. This is what the Consumer Product Safety Commission did when it imposed new fire safety standards on mattresses in July to delay the onset of flashover, one of the leading causes of fatal house fires. The CPSC estimated that as many as 270 people would be saved each year. Which, you would imagine is a good thing.
But what about the chemicals needed to meet the standards? The CPSC pointed to a huge body of research saying they were safe and, besides, they were buried inside the mattress. Nevertheless, the CPSC simulated the effects of sleeping on the mattresses in adverse conditions, such as bedwetting, and still the dermal or inhalation exposure was negligible. But that didn’t cut if for local TV news affiliates around the country, which began running stories about the toxic risks from the new standards, spurred on by an outraged consumer group, “People for Clean Beds.”
What news stations failed to make clear was that “People for Clean Beds” was, in effect, a “person for clean beds” – namely Mark Strobel, who just happens to make mattresses which are exempt from the standards and can only be bought with a doctor’s prescription.
Strobel, without any scientific justification, argued that the new fire safety standard might end up harming or killing 300 million people – which would put mattresses far ahead of the Black Death (approximately 40 million dead, one of the worst pandemics in human history), AIDS (approximately 25 million dead to date) and on a par with the total number of people worldwide who will, in the absence of any breakthroughs in treatment, cumulatively die from cancer over the next forty years (approx 240 million, based on World Health Organization estimates).
Hot air on ‘Fresh Air’The chemicapocalypse reached a fever pitch on NPR’s Fresh Air in November, when Terry Gross interviewed Mark Shapiro, an investigative journalist who had just written a book on the topic: Exposed: The Toxic Chemistry of Everyday Products and What’s at Stake for American Power.”
Shapiro claimed that a litany of epidemiological evidence showed that chemical exposure might be behind “spikes” in the incidence of breast cancer, reproductive problems among young women, endocrine troubles, mutagenic effects in young children, and declining sperm counts.
The only problem is that all of these problems can be explained much more easily by other factors or dismissed as false or misleading: breast cancer rates, for instance, are falling; the spikes in incidence can be explained by such known factors as the surges in hormone replacement therapy use, increased fertility treatments, later age of first birth, alcohol consumption and increased numbers of women having mammograms. The rise in fertility issues is almost certainly a matter of greater reporting and more intervention, say doctors, especially in the marketing of high-end fertility treatments. Birth defect rates are also stable - and the claims for declining sperm counts have been widely attacked for poor methodology.
What Gross, Shapiro, and many other journalists during the year failed to do when it came to the evaluating the risk from trace amounts of chemicals was to think critically about the numbers. Epidemiology can show associations between any number of things; you could, in theory, show an association between green health products and increasing cancer rates if you mined the data enough; but a correlation is not evidence of causation. To establish a causal relationship between common chemicals in parts per billion and health problems, you have to prove that the correlation does not have other, better, and possibly really obvious, explanations.
Not every over-hyped, goof-ridden story in 2007 involved chemicals, the media also stumbled over sex, drugs, race and math.
Single women left at the altar by statisticsTime Out New York sent a wave of panic through the city’s single women in June by reporting in a cover story that there are 185,000 more single women than men looking for love. It turns out, however, that the excess number of single women is due to men dying at younger ages than women. If you look at the male/female numbers in the younger age groups, in most, there are significantly more men. For example, there are 211,590 men aged 18 and 19 in the NY Metro area – but only 201,282 women.
Husbandless teens languish at homeThe New York Times went further and claimed in a front-page article in January that more women are living in the United States without a husband than with one. The claim could only be supported by counting women between the ages of 15 and 17, 90 percent of whom live at home with their parents.
Will one joint make you schizoid?In July, the Associated Press – and many other news organizations – reported that “Using marijuana seems to increase the chance of becoming psychotic… even infrequent use could raise the small but real risk of this serious mental illness by 40 percent.” Since marijuana use rates have skyrocketed since the 1940’s and 50’s, going from single digit percentages of the population trying it to a peak of some 60 percent of high school seniors trying it in 1979 (stabilizing thereafter at roughly 50 percent of each high school class), we would expect to see this trend have some visible effect on the prevalence of schizophrenia and other psychoses.
Roughly one to two percent of the population has schizophrenia (and another two percent or so have other psychotic disorders), and this percentage does not vary much with the region within the U.S. Over time, diagnosis of schizophrenia has changed, making it almost impossible to evaluate whether low-level exposure to pot could increase the risk by as much as 40 percent.
Second generation diversity only please.In March, The Washington Post reported on a new study by researchers at Princeton and the University of Pennsylvania under the provocative headline “In Diversity Push, Top Universities Enrolling More Black Immigrants: Critics Say Effort Favors Elite Foreigners, Leaves Out Americans” While 13 percent of the U.S. black population is foreign-born, researchers at Princeton and University of Pennsylvania found that 27 percent of blacks in American universities were “elite foreigners”. The only problem is that American-born children of black immigrants were counted as "elite foreigners" who gained college admissions over American blacks, despite the fact that they are, uh, American. The study didn’t explain what percentage of the “elite foreigners” were actually American citizens.
Analyzing Avandia Possibly the most damaging use of statistics in 2007 occurred when the New England Journal of Medicine published a study claiming that the risk of heart attack increased 43 percent among diabetics taking the drug Avandia. The study’s author, Steven Nissen, told ABC News that the deaths from Avandia may dwarf 9/11, and scare headlines led thousands of patients to stop taking their medicine.
On closer inspection, many of the cases of heart attack driving the statistical increase in risk came from one study where the diabetic participants already had congestive heart disease. Worse, Nissen excluded studies where there were no heart attacks. If you want to look at the safety of a drug, studies that don't find adverse outcomes are as statistically important as studies that found adverse outcomes. Nissen also wasn't able to control for the amount of time patients were exposed to Avandia relative to when they had a heart attack, which further undermined the certainly of his result. Even though a number of studies were published in medical journals criticizing the methodology, the media ignored them and continued to report the magic increased risk of 43 percent.
As Brian Strom, chair of the Department of Biostatistics and Epidemiology and director of the Center for Clinical Epidemiology and Biostatistics at the University of Pennsylvania noted: “You should not draw any conclusion based on this analysis. I would not recommend taking patients off of Avandia because of this."
The Food and Drug Administration and the European Medicines Agency both looked at all the data and decided, in July and October respectively, that the benefits of Avandia outweighed the risk and the drug shouldn’t be removed. Yet by the end of 2007, news organizations were still citing Nissen’s data without any qualification.
