Lying with data

Lying with data

2020, Jul 30    

It's not hard to spin data to tell the story you want it to. Is ivermectin an effective therapy for Covid-19? The simple truth is that we don't know.

For more on ivermectin and Covid-19, check out this article.

Carlos Chaccour, Assistant Research Professor at ISGlobal and Chief Scientific Officer of BOHEMIA

Joe Brew, Data scientist at Hyfe and Databrew

Summary: Ivermectin may or may not be an effective therapy for Covid-19. We don’t know. And we probably won’t know with any degree of confidence until the results of some randomized, controlled trials come in. In the meantime, the data can be twisted to conform with one’s prior beliefs. This is an article about confirmation bias, using the case of ivermectin.

What’s going on with ivermectin in Latin America?

Although the very limited evidence supports the use of ivermectin against COVID-19, this antiparasitic has been included in the national therapeutic guidelines for COVID-19 in Peru and Bolivia and in sub-national guidelines in Brazil. There is also broadspread interest in its use in Chile. This surge in demand has caused some serious issues like mass administration of veterinary formulations to thousands of vulnerable populations and the selling of poor-quality/counterfeit formulations. Mass use of ivermectin is not exempt from risks, including a false sense of security that may hamper social distancing or even more complex immunological changes that might affect the way our body responds to viruses.

So, is ivermectin in Latin America saving lives?

Some are arguing that ivermectin is reducing mortality among Covid-19 patients. Proponents are pointing out that the ratio of deaths per confirmed cases (aka “case fatality rate” or CFR) appears lower in those countries where ivermectin is being promoted. The below chart shows CFR among 5 ivermectin-promoting countries (Brazil, Peru, Dominican Republic, Honduras, Bolivia) vs. CFR among 10 countries not promoting ivermectin (Mexico, Canada, Chile, Cuba, Argentina, Nicaragua, Colombia, Panama, Uruguay, Venezuela)

Deaths per confirmed cases

Much of the above is driven by the large, most affected countries, namely Brazil (a country apparently “promoting” ivermectin) versus Mexico (ivermectin-averse): the former has seen a recent decline in the CFR, whereas the latter’s CFR remains high.

Deaths per confirmed cases

Wow, it looks like ivermectin is really great! We should be administering ivermectin everywhere, right?

Not quite. These country-level comparisons are subject to all sorts of biases, making it difficult to deduce meaningful inference. Testing strategies and rates are very different by country; the percentage of true Covid-19 deaths that different countries’ data systems capture are variable; we don’t know the extent to which ivermectin is actually being used in each country; and perhaps most importantly, there is “ecological fallacy”.

What is ecological fallacy?

Ecological fallacy is when you make some conclusion about something at the individual level (ie, that ivermectin reduces Covid-19 mortality) using non-individual data. It’s a classic “correlation does not mean causation” issue. That is, just because things are associated does not necessarily mean that one thing causes the other.

Maybe an example would be useful: one might stumble upon a Brazil vs. Mexico chart online and conclude that ivermectin is an effective therapy. But, by the same reasoning, one might look at the same chart and conclude that speaking Portuguese reduces mortality, and speaking Spanish increases it. The point is, we can’t really know about causality without individual-level data. And the best individual-level data is the kind generated from experiments, not observations.

But “ecological” data is better than no data, right?

Maybe. Maybe not. We need to tell the whole story, and the case fatality rate is only part of the story. And it’s a part of the story that makes ivermectin look good. There are other indicators that make ivermectin look bad.

Okay, show me some data that makes ivermectin look bad.

Here you go. Instead of looking at case fatality rate (which is subject to testing rates), let’s instead just look at Covid-19 deaths per population.

Daily deaths per million - Brazil vs. Mexico

The blue line (ivermectin-promoting Brazil) no longer looks so much better than the orange line (ivermectin-averse Mexico). Why is it that this chart is so different from the previous one? It’s partly due to the fact that Brazil has been apparently finding many cases, whereas Mexico has been missing them (see below).

New daily cases per million

Could it simply be that the scale of the outbreak (rather than just testing capacity) is much greater in Brazil? Sure. But we don't know that. What we do know is that the test positivity rate in Mexico is exceptionally high, suggesting that there are many undetected cases.

But that’s just 2 countries. What about the countries from the original chart?

Here you go.

New daily deaths per million

Wow. It looked like ivermectin decreases mortality. Now it looks like ivermectin increases mortality. I’m confused.

You should be.

Do you see what just happened? None of the above is a lie, but not all of it can be true.

Doctors and public health authorities are facing an unprecedented surge in deaths due to a single disease. Their urgency to act is not to be judged harshly as it is based on an honest desire to alleviate suffering. Compassionate use of drugs outside their approved label is a common figure for desperate circumstances, including patients with untreatable diseases, COVID-19 being one of them. However, even compassionate use in patients with stage IV cancer is heavily regulated under the fundamental primum non nocere (first do no harm) principle. No drug is exempt of potential harm, and when used at population level, this potential harm increases proportionally with the number of people treated.

This pandemic has (rightfully!) brought a strong sense or urgency upon us. But this rush has led to some mistakes that have caused great confusion and sometimes harm. This includes the rise and downfall of hydroxychloroquine, fraudulent data shaping some important global health action or suggestions to use “miraculous” medicines based on disinfectants. A recent editorial in The Lancet Global Health mentions what appears to be a paradoxical lesson learned a need to slow down, “When research, writing, and peer review are rushed, the consequences can be damaging”.

There is urgent need for good quality information, there is urgent need for rapid randomized controlled trials that can and must be done in the current epicenter of the pandemic which allows for quick recruitment and completion. There is additional need to answer basic science questions and above all, there is an urgent need to avoid the temptation of thinking that “bigger” data means “better” data.