Unreliable data: how doubt snowballed over Covid-19 drug research that swept the world

Dr Carlos Chaccour had just woken up in Barcelona when he opened his laptop to read the latest Covid-19 research.

Usually, he would start the day by meditating, but that was proving difficult in the middle of a global pandemic.

Overnight, several colleagues had emailed him a large study that had just been published online which examined the effect of the anti-parasite drug ivermectin on Covid-19 hospital patients around the world.

Chaccour is known for his work with the research institute ISGlobal in Spain examining parasites and microbes, exploring how these vectors spread disease and what works to treat the infections they transmit. He is particularly interested in mosquito-killing drugs, especially ivermectin. So he was intrigued by the study, which was published on 14 April in a version known as “preprint”, which means it was made available online before it had been peer-reviewed or accepted by a medical journal.

“I saw the researchers had looked at this huge database … they included 169 hospitals in Asia, Europe, Africa, North America and South America and 1,900 Covid-19 patients seen by hospitals in those countries by 1 March,” Chaccour says.

The study methodology said its data had been obtained from Surgisphere. The Surgisphere website says it owns a data analytics system called QuartzClinical which monitors global healthcare in real time through data collection from 1,200 international hospitals. Promotional material says the database “has led to advances in care for kidney failure, aneurysms, lymphedema, peripheral artery disease, colon cancer, and cardiovascular disease”.

But as Chaccour and other researchers began to look more closely, they quickly found some concerning anomalies. Over the next weeks those doubts would only increase. Surgisphere itself came under greater scrutiny, culminating in two of the world’s most prestigious medical journals reconsidering studies based on its data, an about-turn from the World Health Organization on research into a potential Covid-19 treatment, and a Guardian investigation that uncovered worrying inconsistencies in the Surgisphere story.

Chaccour’s first surprise was that the study had found 52 Covid-19 patients who had received ivermectin. At the time, ivermectin was not being widely discussed as a potential Covid-19 treatment. Yet the study said patients around the world had already been receiving it.

Chaccour, who has worked throughout Africa and knows African healthcare systems well, believes many hospitals there are not equipped with the electronic health systems required to be a part of such a database.

“And they’re supposed to be connected to a fancy automatic thing that gives all this data to a corporation in the US? It was so weird.”

US data in the study also raised questions. The paper found mortality of the Covid-19 patients on mechanical ventilation and in the control group was 2%. By contrast, a paper published in the journal JAMA about New York state’s largest health system found almost 25% of those on ventilators died. Only the most critically ill patients require ventilation, so this high death rate wasn’t a surprise. The low death rate from the preprint study was.

“So these guys have a control group with 10-fold less deaths?” Chaccour says. “But it was a preprint, and prepublication, and I said to myself, ‘well this is just another non-reviewed thing, whatever, it doesn’t make much sense’.” Busy with his own work, he tried to forget the study.

“They talk about propensity scores, this huge database from 169 hospitals, five continents, it sounds fancy, and people started latching to this study out of hope,” Chaccour says. “Doctors were desperate for something to treat Covid-19.”

On 2 May, two weeks after the study appeared online, a doctor in Peru wrote a white paper for the government about the use of ivermectin to treat Covid-19, heavily citing the preprint Surgisphere study as evidence. Peru reported its first Covid-19 case on 6 March, but by the beginning of May it was in a state of emergency, having recorded 42,000 cases and about 1,200 deaths. Less than a week after the white paper was published, the Peruvian government included ivermectin in its national Covid-19 therapeutic guidelines. Projects involving ivermectin around the world received thousands of dollars in grants.

Chaccour was concerned. He knew ivermectin well and was shocked at how quickly it was being adopted as part of treatment protocols without rigorous research to back up its use. He believes more studies should be done first, as he wrote in an editorial in which he was lead author, published on 16 April. He sent the lead researcher on the preprint paper an email with some questions and concerns about the data, which were forwarded to a co-author of the paper, the Surgisphere founder and chief executive, Dr Sapan Desai.

Instead of answering his questions about the data, Chaccour says, Desai flattered him and spoke enthusiastically about potential collaboration.

“Let’s just say my concerns about the study did not reduce whatsoever,” Chaccour says.

Then in May, the most respected medical journal in the world, the New England Journal of Medicine, published a study featuring two of the same authors as the pre-print ivermectin study. The renowned Harvard University vascular surgeon Dr Mandeep Mehra was the lead author, Desai the second author. The study also based its results on the Surgisphere QuartzClinical database, including data from Covid-19 patients from 169 hospitals in 11 countries in Asia, Europe and North America. It found common drugs given for heart disease were not associated with a higher risk of death in Covid-19 patients.

Chaccour thought this might explain his concerns about the ivermectin study. “I thought, [maybe the ivermectin study] was just a side project while they were busy working on this big New England Journal of Medicine study. I didn’t look closely at the study, because cardiovascular disease is not my area.”

In the meantime, another drug, an anti-malarial called hydroxychloroquine, a derivative of chloroquine, was gaining traction as a potential Covid-19 treatment in the US after Donald Trump described it as possibly “one of the biggest game changers in the history of medicine,” adding, “it’s not going to kill anybody”. Soon after, a US man died after he drank chloroquine found in fish tank cleaner because he was scared of getting sick. Nigeria also reported chloroquine poisonings. Drug regulators around the world urged people not to attempt to source and ingest the drug, emphasising it was unproven as a Covid-19 treatment and came with potentially toxic side-effects.

“It was a royal mess,” Chaccour says. “There was huge political polarisation about hydroxychlorioquine, politics became mixed in with policy. So there’s people defending hydroxychloroquine because they like Donald Trump, and people opposing it because they don’t like Donald Trump. This should be about data, not opinions, and absolutely not about politics. The world had gone crazy.”

Meanwhile, ivermectin had continued gaining momentum throughout Latin America. The ivermectin preprint paper had been downloaded more than 15,000 times, its abstract viewed 90,000 times. Bolivia went a step further than Peru, announcing on 19 May that 350,000 doses of the drug would be distributed. “Demand goes up hugely, so much so that a blackmarket of ivermectin emerges,” Chaccour says. “So ivermectin threatens to become the new hydroxychloroquine in Latin America.” Researchers in Latin America who found hydroxychloroquine was not effective for Covid-19 began receiving death threats after their findings were published in a US medical journal.

On 22 May, the Lancet published a hydroxychloroquine study involving 96,000 patients around the world which found the drug was associated with a higher risk of heart problems and death in those with Covid-19. The authors again included Mehra and Desai, and again, Surgisphere’s QuartzClinical global database was used, this time to obtain the data from 1,200 hospitals. The study involved so many hospitals and people that its findings, to many, seemed definitive. Hydroxychloroquine did not work for Covid-19, and in fact could be dangerous in those patients.

“The Lancet study was published on a Friday. In less than 24 hours, the World Health Organization [WHO] had stopped the hydroxychloroquine arm of the trial they’re sponsoring,” Chaccour says. “This had massive consequences. There are 131 hydroxychloroquine Covid-19 trials registered. Many national funding agencies stopped or paused these trials. Many patients read the news about the study. Thousands of patients enrolled in these trials feel fear and anguish. How can you continue these trials now? When patients just read this drug is not good?”

On Thursday, following growing concern from the medical community about the Lancet study and the database, the WHO reversed the decision to halt hydroxychloroquine trials. It too had reviewed the study, and found there was no reason to discontinue trials based on the data. However, WHO officials also reiterated there was as yet no evidence that hydroxychloroquine, or any drug, reduces the mortality in patients who have Covid-19.

Chaccour was baffled that Mehra and Desai were publishing these large studies in prestigious journals every few weeks. It was research that would take months, at least, to write up, longer than the pandemic has been around.

This time, Chaccour was not alone in his concerns. His questions about the paper were echoed by scientists around the world who had also identified serious and potentially fatal flaws in the study. Guardian Australia revealed errors in the Australian data, which overestimated the number of deaths. It led to the Lancet issuing a correction, but Desai insisted the error did not change the overall findings of the study – that hydroxychloroquine was ineffective and potentially dangerous in the treatment of Covid-19 patients.

A private ambulance team brings in a probable COVID-19 patient to the emergency room of the Elmhurst Hospital Center in the Queens borough of New York City.

An investigation by Guardian Australia found the database the studies were based on contained seriously questionable data. Desai did not adequately answer questions from Guardian Australia about how Surgisphere, once listed as a medical education company, came seemingly out of nowhere to become a data analytics company implementing a global database using artificial intelligence and cloud sharing in a matter of months, and with just 11 staff.

Surgisphere’s science editor, listed on LinkedIn, appears to have no science or data credentials. Instead, searches of her name and photo suggest she is a full-time science-fiction author and artist. Searches of the company’s director of sales and marketing, also listed on LinkedIn, suggest she is an adult model and brand ambassador, again with no scientific background or references to a Surgisphere job on her websites. Most of the Linkedin profiles for Surgisphere staff were created only two months ago.

Desai told Guardian Australia: “Surgisphere has 11 employees and has been in business since 2008. Our healthcare data analytics services started about the same time and have continued to grow since that time.” However, information about the database has appeared publicly only since 2019. Before that Surgisphere appeared to publish medical education tools, such as textbooks. Of the database, Desai said: “We use a great deal of artificial intelligence and machine learning to automate this process as much as possible, which is the only way a task like this is even possible.”

He did not explain how the artificial intelligence worked to gather the data, or how hospitals went about putting the system in place and feeding data into it. Desai later said hospital staff were responsible for de-identifying patient data before feeding it into the database. Not one major Australian hospital spoken to by Guardian Australia had heard of Surgisphere or the database.

Doctors from hospitals around the world scoffed at the idea that staff would have time to de-identify patient data and contribute to a US database in the middle of a pandemic. Desai said the way Surgisphere obtained data was “always done in compliance with local laws and regulations. We never receive any protected health information or individually identifiable information.”

Chaccour had the same questions about Surgisphere. How was it collecting its data? And which hospitals were taking part? “My to-do list was completely on hold, for a week I looked at nothing else,” he says. “And as a result I have serious concerns with the database that I believe may have major flaws.” He found race data was reported in the studies that used the database. “It is not clear how Surgisphere gets race data, since collecting it is uncommon in most countries, and illegal in some,” Chaccour says. Surgisphere did not respond to questions from the Guardian about how race data was collected.

It also surprised Chaccour that the company behind one of the largest and fastest hospital databases in the world had almost no online presence. Its Twitter handle had fewer than 170 followers when he checked it, with no posts between October 2017 and 2020; its LinkedIn page has fewer than 100 followers and six employees, which now appears to have reduced to three, with no posts before March 2020; its YouTube page has few subscribers, two videos, and no videos posted in the last decade.

It is also unclear where the company is based. Its address is listed in Chicago’s John Hancock Center. The privacy policy of QuartzClinical refers to a residential address in Illinois. Nor is its ethics policy transparent. Despite the Surgisphere database collecting patient data including lab data and physical exam findings, the study published in the Lancet said “the data collection and analyses are deemed exempt from ethics review”.

A history of ambitious claims

But why do it? Why would Desai allow questionable data to be relied on by prestigious journals and run the risk that the errors would be picked up by editors, or by other researchers around the world once published?

“This guy is flying high,” Chaccour speculates. “And he’s enjoying the view from up there. He doesn’t realise the wax in his wings is melting. He’s getting integrity, and interviews. And I think perhaps his co-authors were fooled into this.”

Desai appears to have a history of making ambitious claims. Eight years ago, he launched a crowdfunding campaign on the platform Indiegogo promoting a device called Neurodynamics Flow. The device, Desai wrote on the campaign page, “is a next generation human augmentation device that can help humans achieve what was never thought possible. Unlock human creativity. With over a decade of research into fine-tuning the device, Flow’s sophisticated microprocessor precisely stimulates various areas of the brain to create a higher level of function.”

For a donation of US$100, backers of the project were promised an exclusive, signed 8×10 fine art print of the blueprint for the actual prototype. For US$500 project backers were offered a “Neurodynamics Flow in your choice of colour with suede microfiber carrying case and accessories. All electronics and microprocessor are sewn into a wearable device. Machine washable.”

The campaign raised about US$300 of its US$10,000 goal. One of Desai’s project backers asked on the website: “What happened to the project? Where is my perk?” There was no visible response on the site from Desai. It appears the device never eventuated.

Surgisphere’s Covid-19 “Mortality Prediction Tool” has also come under fire. Surgisphere claims that its tools are based on machine-learning algorithms and derived from real-time data. But when different ages are entered into the tool, it appears to simply divide them by 20 to get the predicted mortality, rounded up to the nearest 0.1%. For example, a 10-year-old would have a 0.5% risk of death, according to the tool.

A senior London infectious diseases doctor concerned by the tool told the Guardian: “We know that this is not what happens in Covid-19, where there is an exponential relationship between age and mortality.

A health worker wearing a protective medical equipment extracts a blood sample from a patient’s finger to make an antibody test for patients suspected of being infected with for the coronavirus (COVID-19) at the Center Health Teixeira de Freitas, in Niteroi, Brazil.

“Unfortunately Surgisphere’s tool appears to be a public-facing shell without real data driving it,” he said. “Given the clear data flaws pointed out by many readers of the Lancet article, it is reasonable to conclude that that publication also suffers from the same fundamental problem.” Desai did not respond to questions abo

Desai is also facing larger issues. In November 2019, two medical negligence claims were filed to the Cook County court. Desai said in an interview with the Scientist that he deemed any lawsuit against him to be “unfounded”. He told the Guardian and the Scientist that he stands by the studies, but did not explain how hospitals join and sync their data.

‘There are definite doubts’

James Heathers, a research scientist with Northeastern University in the US, said major results in large medical journals can affect medical policies in a matter of days, both at the level of changing local hospital practice and changing governmental health policy. That means it is critical that data is transparent and easy to clarify, especially amid a global pandemic.

“If serious problems are found with data in a paper such as this, apart from the potentially catastrophic consequences for the authors, it speaks to a problem with how we aggregate and understand evidence,” Heathers said.

“It is more crucial than [at] any other point … in modern history that results are transparent, fully assessable, and accurate. This means open analytical code, well-specified methods, and inspectable data set – even if they are proprietary – should be considered mandatory as conditions of publication at present.”

Stephen Evans, a professor of pharmacoepidemiology from the London School of Hygiene and Tropical Medicine, said: “There are definite doubts about the integrity of the Lancet paper. In retrospect many readers and decision-makers may well have placed too much reliance on that paper.”

On Wednesday, the Lancet conceded that “important scientific questions have been raised about data reported in the paper”.

“Although an independent audit of the provenance and validity of the data has been commissioned by the authors not affiliated with Surgisphere and is ongoing, with results expected very shortly, we are issuing an Expression of Concern to alert readers to the fact that serious scientific questions have been brought to our attention,” the editors said. The NEJM issued a similar expression of concern stating: “We have asked the authors to provide evidence that the data are reliable.”

Surgisphere data is used for retrospective observational studies, a type of study which can be problematic because of the presence of variables that were not controlled for, and selection bias. Scientists worldwide and the World Health Organization have repeatedly said randomised control trials are needed to show which drugs, if any, are effective for Covid-19 and that these should be made public. Randomised control trials are seen as the gold standard of science due to limited opportunities for bias.

Mehra has called for more such trials before conclusions are drawn from the hydroxychloroquine study. In response to questions from Guardian Australia, Desai said the Lancet paper should not be overinterpreted and that the study itself recommended that randomised control trials be “urgently completed”.

But in a YouTube video talking about the Lancet study, Desai also said of Surgisphere: “The real question here is, with data like this, do we even need a randomised control trial?”

Chaccour found the comment baffling. But he is most disappointed that Desai’s questionable data rapidly changed public policy around the world. Despite widespread criticisms of the studies, they remain available online.

“Worldwide policy, funding and clinical trials have changed fast,” he said. “We now go into a slow process of validating the concerns we have.”