Messy, incomplete U.S. data are hobbling the pandemic response

The contentious and confusing debate in recent weeks over coronavirus booster shots has exposed a fundamental weakness in America’s ability to respond to a public health crisis: The data are a mess.

How many people have been infected at this point? No one knows for sure, in part because of insufficient testing and incomplete reporting. How many fully vaccinated people have had breakthrough infections? The Centers for Disease Control and Prevention decided to track only a fraction of them. When do inoculated people need booster shots? American officials trying to answer that have had to rely heavily on data from overseas.

Critically important data on vaccinations, infections, hospitalizations and deaths are scattered among local health departments, often out of date, hard to aggregate at the national level — and simply not up to the job of battling a highly transmissible and stealthy pathogen.

"We are flying blind," said Ali Mokdad, an epidemiologist at the University of Washington's Institute for Health Metrics and Evaluation, who spent two decades working for the CDC. "With all our money, with all our know-how, we have dropped the ball. ... We don't have the data. We don't have the good surveillance system to keep us informed."

The dearth of timely, comprehensive data complicated the ability of the nation's top public health officials and infectious-disease experts to reach a consensus on the need for booster shots. The experts looked at conflicting data from Israel, the United Kingdom and the U.S., and came up with a bewildering set of recommendations. The debate seemed to confuse more than clarify the necessity of an additional shot.

"We are pulling data in from all different sources," said a senior administration official, who spoke on the condition of anonymity because they were not authorized to discuss the issue publicly. "We're trying to put it all together to see ... what is the vaccine efficacy? And there's this wide divergence. It's not reconcilable."

Data is key to an effective pandemic response — and the lack of it has hobbled the U.S. response again and again. The lack of testing, and then of standardized reporting of cases and deaths, left U.S. officials slow to grasp the scale of the crisis when the virus first began to spread. Insufficient data also meant supplies arrived too late in hard-hit cities. State and federal officials made decisions about travel restrictions and reopening policies with an incomplete picture of what was happening.

Many places were forced to shut down before they had substantial outbreaks, former FDA Commissioner Scott Gottlieb told The Washington Post, and when the virus finally arrived, some resisted a return to restrictions.

"Early on, CDC couldn't even tell us how many people were being hospitalized for COVID," Gottlieb said.

Multiple factors underlie this data deficit. First and foremost: The U.S. does not have a national health system like Israel or the U.K., and in a pandemic, must rely on a vast and decentralized public health infrastructure that is notoriously underfunded and full of holes. As a result, there is no simple way to track infections or outcomes across a wide swath of the population.

» READ MORE: N.J. school nurses juggle COVID with cuts

Another obstacle to data aggregation may be the siloed computer systems and the self-interest of medical institutions. Some hospital systems want to hang onto their data, said Michael Kurilla, director of the division of clinical innovation at the National Institutes of Health's National Center for Advancing Translational Sciences.

"They don't necessarily want to give up all that data because they see that as a potential future revenue stream," Kurilla said.

The CDC compiles national statistics by collecting data from every state and locality, but these jurisdictions often have different ways of counting tests, infections and even deaths. The data may not be submitted to the CDC for days or weeks. Many smaller jurisdictions still share that data via outdated fax machines.

"We're still operating on a largely 19th-century system," Kurilla said. "Who exactly is to blame is really hard to point a finger at. There are systems where things are done on paper, some information is being faxed, so it's being transcribed. There isn't any way to seamlessly upload information."

The Biden administration recently rolled out a pandemic preparedness plan that aspires to "fundamentally transform our capabilities to protect the nation." One element would be the modernization of digital health data, with standardized software that would enable different jurisdictions to share and analyze data.

The data problem has been recognized by federal officials and outside experts for many years, Biden's science adviser, Eric Lander, said in an interview Wednesday.

"It's a question that pertains to the whole health care and public health system. In the United States our data systems are not interoperable. They don't talk to one another," Lander said.

The task of gathering and analyzing data is currently too laborious, and lowers situational awareness in a crisis, he said.

"If it takes weeks to clean the data ... it means you're going to be running weeks behind the war that you're fighting. That's just no way to take on a pandemic," Lander said.

» READ MORE: At last minute, Pa. lawmakers vote to extend dozens of COVID-19 waivers

Solving this won't happen overnight, but should not take more than two or three years, he estimated. The underfunding of public health departments is at the core of the problem, he said. Giving them access to affordable, standardized software for handling data "is going to be useful not just in the next pandemic — it's going to be useful in the next flu season, it's going to be useful in the next measles outbreak."

The CDC is charged with making sense of the patchwork of state data, and regularly issues reports on outbreaks. But critics say the CDC operates at too slow a pace, as if it were an academic institution and not a first responder in a crisis. The long-standing rap is that the agency focuses heavily on retrospective studies, and does not share those results quickly — even with other health agencies. The CDC traditionally has emphasized empirical rigor over speed, an aspiration in conflict with the demands of a rapidly evolving health emergency.

"They're out there putting reports from three months ago, and you can't do that in a pandemic when things move so fast," said Walid Gellad, an associate professor of medicine at the University of Pittsburgh School of Medicine who criticized as premature the administration's initial push for boosters for all adults.

"In a span of three months, we had super high cases, then the lowest cases we've ever seen, and now we're back up again. You can't use old data in a health emergency that is changing as quickly as COVID," he said.

In response to questions from The Post, the CDC said it has shared the results from numerous vaccine effectiveness studies over the last two months that helped shape the discussion on the need for booster shots. CDC spokeswoman Kristen Nordlund said in a statement those studies showed that vaccines are effective at reducing the risk of severe disease, hospitalization and death, but that protection may decrease over time and may be less able to protect against the delta variant. She added the agency is publishing all of its data on vaccine effectiveness in one place on Thursday.

"Even highly effective vaccines often become less effective over time and tracking this can take time," Nordlund said. "We relied on the data from colleagues in Israel and the UK because the epidemiology of their outbreaks, experience with the delta variant, and use of boosters preceded what happened in the United States."

Many at the CDC recognize the agency needs to move faster. In August, CDC Director Rochelle Walensky announced plans to develop a new forecasting and outbreak analytics center to analyze data in real time to better predict disease threats, which is expected to be up and running early next year.

CDC data played only a small part in the booster decision, senior administration officials said in interviews, in part because Israel vaccinated its population faster than the United States and began experiencing a delta wave several weeks sooner, giving it a data set that extended over a longer period.

» READ MORE: Judge denies Tredyffrin/Easttown parents’ request to halt school mask mandate

But administration officials and outside experts said the CDC should have shared its own findings on vaccine effectiveness more quickly, rather than waiting until its results were publication ready in the late summer and early fall.

The CDC also drew criticism for its decision in the spring to stop tracking all breakthrough infections, and instead follow only those that resulted in hospitalization. The agency has said it could compile more accurate and complete data from its studies tracking thousands of people who are regularly tested and monitored to see if they develop breakthrough infections. Critics of the decision say policymakers need as much real-time information as possible about new variants that may cause even minor breakthrough infections, but some experts contend these more focused studies offer clearer and more reliable results, and are a better use of the agency's resources.

But even the results of those focused studies were often hard to come by. As administration officials debated in July and August about whether the U.S. would need to administer boosters, they repeatedly implored the CDC to share what it was learning, said several people familiar with the conversations who spoke on the condition of anonymity because of the sensitivity of the conversations. As a result, officials initially relied on data from Israel, as well as studies conducted at the Mayo Clinic in New York state and by Kaiser Permanente.

"There is no way nationally in this giant country to connect who's been vaccinated and what their outcomes are, and that's the underlying problem," said Gellad. "Whenever you have to piece lots of data together, a lot of which is contradictory, it can be confusing."

Better data by itself would not have resolved all the disagreements among disease experts and policymakers. Experts on infectious disease will look at identical data sets and reach different conclusions about who needs an additional shot.

Some scientists — including several in the administration — believe data from Israel, the United Kingdom and elsewhere showing waning immunity against infection over time strongly support the need for millions of people to get booster shots.

They argue that these breakthrough cases, even if they are not classified as "severe," translate into shuttered classrooms, lost income, and continued widespread transmission of the virus. As long as infections are still circulating in large numbers, several administration officials and scientists argue, the country cannot crush the pandemic.

Many of these scientists also believe that waning immunity is an early warning sign that will inevitably lead to increased hospitalizations, an outcome they hoped to avoid by administering boosters early.

But many members of the FDA and CDC advisory committees, who are also scientists and public health experts, took a starkly different approach. They focused on the risk of hospitalization among vaccinated individuals, which has not increased significantly in the U.S.

Some of these scientists believe the public has unrealistic expectations of vaccines. The fact that hospitalizations are not dramatically increasing among the vaccinated — especially those under 65 — indicates the vaccines are functioning as designed.

They also questioned whether new data from Israel, which showed a rise in "severe" disease, was fully applicable to the U.S. Israel uses a different definition of "severe," basing it on such measurements as oxygen saturation and elevated respiration rate, rather than on hospitalization. And Israel isn't America: It's a much smaller country, less diverse demographically, and it doesn't have as great a burden of chronic health problems, such as obesity and diabetes.

The data shortfall is not simply an issue for the crafters of national vaccine policy. It's also a conundrum for individuals trying to figure out their own risk calculus.

Even Mokdad, the IHME epidemiologist who studies the data for a living, is uncertain about his continuing level of protection from vaccines.

"I'm a healthy 59-year-old person," he said. "I'm not obese, I'm very healthy, I don't have any chronic condition. The only thing I have against me is age."

It has been more than six months since he got his second shot.

“Do we know how much immunity I have against hospitalization in the U.S.? No,” he said. “Do we know how much immunity I have against death? We don’t.”