Behind the scenes at Penn Medicine during the global tech meltdown

Top executives at Penn Medicine were woken up by frantic calls at 2:30 a.m. on Friday.

Across the vast health system, staff were trying to log onto their computers, only to be met with a bright blue screen with a few lines of text, known in tech circles as the Blue Screen of Death.

Something had rendered nearly 50,000 computers at Penn inoperable. And by 2:30 a.m., the health system’s IT service desks had fielded so many calls that it was clear the problem was widespread.

John Keogh, an anesthesiologist who oversees all surgeries and procedures at the system’s flagship Hospital of the University of Pennsylvania, called into HUP’s control center around 3 a.m. to get a sense of what was happening on the ground.

“I said, ‘Tell me — is Epic down?’” he said, referring to the electronic medical records system that Penn and many other local hospitals use.

“And they’re like, ‘You won’t believe it. We don’t have computers. It’s all blue screens showing up here,’” Keogh recalled being told.

The IT outage was unprecedented: “We’ve simulated plans for the event that our electronic medical records become inoperable. But we were less prepared for the complete inability to use a computer.”

Dozens of Penn staffers signing into an emergency, middle-of-the-night Zoom call began to realize their systems had been caught up in a global technology meltdown.

“Naturally, we first thought it was ransomware” — a malicious cyberattack that locks computer systems and demands a ransom to restore them, Penn Medicine CEO Kevin Mahoney said. But it soon became clear that the problem had originated from CrowdStrike, the cybersecurity company that Penn relies on to protect tens of thousands of servers and PCs.

An error in its update for Windows operating systems had brought the Blue Screen of Death to millions of PCs worldwide, upending government operations, forcing airlines to cancel flights, and hampering access to patient records and other crucial hospital programs.

Penn was one of the most impacted health systems in the Philadelphia area, with the outage forcing the rescheduling of nearly an entire day’s worth of nonessential procedures and appointments and restricting doctors to caring for only their sickest patients, while taking medical notes on paper. In the aftermath, Penn has extended evening and weekend hours at some clinics, vowing to get most canceled appointments rescheduled within two weeks.

Penn’s experience was due, in part, to caution about restoring access to affected systems too soon and potentially compromising patient data, leaders now say. And because Penn had invested heavily in a leading cybersecurity provider, Mahoney said, it was similarly heavily impacted when its system erred.

“We planned for hacks. We planned for bad actors. We hadn’t planned for the security vendor to push out the daily patch that wasn’t right,” Mahoney said.

The hospital’s first patients with scheduled appointments were set to arrive around 5:30 a.m. No one knew how long the outages would last, so hospital executives at first decided to allow patients in, and planned to keep them for up to an hour and a half, hoping the situation resolved. Later, when it became clear the outages would not be resolved that day, they began canceling appointments.

IT staff had security concerns about rushing the process.

“We took a very conservative approach to making sure that this wasn’t a breach where we had been compromised,” said John Donohue, the vice president of entity services at Penn’s Information Services department. “You can make a mistake by just starting to turn things back on again.”

In the meantime, Penn staff prepared paper records for physicians to document cases and worked to determine whether the imaging machines that treat “the true emergencies” — strokes, heart attacks, and other health conditions where time is of the essence — could still function.

Those machines use a different operating system and were still functional, Keogh said, so emergency departments remained open. More than 2,400 inpatients, including those in intensive care, continued to receive treatment.

At Penn Presbyterian Medical Center, doctors treated patients who needed care immediately, including a patient with a serious leg infection and a person whose fractured limb had an exposed bone.

“These are the kinds of things we have to get done. We can’t wait and hope they get better on their own,” said Mark Alan Pizzini, the vice president of preoperative services at Presbyterian.

But with patients’ electronic medical records out of reach, Penn used backup PDFs of patient schedules to triage those with appointments.

Surgeons were asked whether a patient’s health would be negatively impacted if they put off surgery. Those not in immediate danger were asked to wait.

Alison Loren had a patient who couldn’t wait. Loren, who directs the blood and bone marrow transplant program at the Abramson Cancer Center, was preparing to conduct a bone marrow harvest, taking stem cells produced by a donor’s healthy bone marrow and transplanting them into a patient with blood cancer.

The hope is that the stem cells will then produce healthy marrow for the patient, slowing or stopping the cancer’s progress.

“A patient’s life is hanging in the balance,” Loren said. “Even though our operating room was telling all of the cases that they were on hold, this particular patient needed to stay right where he was.”

All told, Abramson staff had to juggle about 150 patients receiving care in the hospital and hundreds more scheduled for outpatient appointments. Some would still need immediate care or even an ER visit on Friday; others had less pressing needs. The bone marrow harvest — from a son who was donating marrow to his father — was deemed essential, and went smoothly, Loren said.

» READ MORE: Penn researchers are studying how to prevent breast cancer recurrence, which is often fatal. Their work just got a $10 million grant.

By that evening, Penn IT staff were bringing systems back online — though many affected computers still needed to be rebooted. Abramson has about 160 computers between four clinic floors, so Loren and a member of Penn’s operations staff began walking from floor to floor, rebooting every one by hand.

Mahoney said a top priority for the system is to facilitate surgeries and appointments that were canceled within a week or two. On Friday, several patients had taken to X, formerly known as Twitter, to express their disappointment with canceled procedures.

Keogh said most patients will get appointments within two or three weeks, but some might take longer than others: Cardiology patients and surgery patients, for example, will likely be rescheduled sooner, but patients waiting for an endoscopy might see longer waits because of the high demand for appointments.

“We know it impacted patients, and it’s a huge inconvenience, and we want to get them accommodated,” he said.

Some departments, like Loren’s at the cancer center, are staying open late to see rescheduled patients.

Penn has long had plans for how to get back to work after a major disruption like Friday’s outages. Leadership there will review how well they followed those plans on Friday and discuss how to improve their response in the future, said Anna Schoenbaum, Penn’s vice president of applications and digital health.

Among the ideas already floated: using more iOS devices, because the CrowdStrike problem affected mostly computers with Windows; subjecting new updates to increased scrutiny; and checking for errors first on a small number of computers, cut off from the rest of Penn’s systems.

Still, there’s no way to eliminate risks entirely, Mahoney and other Penn executives say.

“This was an outlier event,” said Donohue, the IT executive. “But it opened our eyes in terms of how we need to protect ourselves from things like this.”