Pa.’s voter website crash stems from a Unisys-run data center that broke down last weekend in Virginia

The state of Pennsylvania and Unisys, the Montgomery County company that played a key role in the development of the modern computer, depend on each other.

But a breakdown last weekend has tested their nearly $100 million-a-year relationship and renewed questions about the reliability of the commonwealth’s computer systems.

Much of the data that holds Pennsylvania together as a state — voter registrations, tax payments, orders from the state liquor monopoly, and cash assistance information — aren’t kept or processed by clerks at the State Capitol these days.

The information is held in a corner of a 150,000-square-foot building that looks like a pile of boxes, one of many lining the streets that locals call Data Center Alley in Ashburn, Loudon County, Va.. That data center sits 30 miles west of Washington and 100 miles south of Harrisburg. It’s in a neighborhood where computer equipment purchases are exempt from sales tax.

The center is run for Pennsylvania by Unisys, the 20,000-employee computer maker-turned-security and tech services company in Blue Bell that has made a specialty of government data processing and protection. Unisys can trace its roots to the Philadelphia company founded by the inventors who built the pioneering ENIAC computer at the University of Pennsylvania during World War II.

» READ MORE: Pennsylvania’s voter services website went down over the weekend as deadlines for mail ballots loom

Pennsylvania’s state government accounts for nearly 5% of Unisys’ yearly sales. And Unisys lags behind only Deloitte in terms of state IT spending, according to state Rep. Seth Grove (R. York), chairman of the lower chamber’s Government Oversight committee.

But something went wrong in Ashburn on Saturday night at about dinnertime.

Computer equipment failed at the Unisys-run data center, Michael Newsome, Gov. Tom Wolf’s secretary of administration, said in a statement.

(Wolf administration officials did not disclose the location of the center, but the site, no secret to Harrisburg IT professionals, was confirmed by other state officials.)

Why is it so far from Pennsylvania? “It is a standard industry practice to geographically disperse a primary data center” from secondary or backup locations so they aren’t as likely to go down all at once, said state spokesman Dan Egan.

The company won’t say exactly what happened to its hardware. Public officials say they are still trying to understand the impact and how to avoid a repeat.

“Some hardware went down and they don’t know why – there were no telltale signs,” Grove told me. “Normally, [Unisys computers] will communicate and let you know something’s up. According to them, they didn’t see that with the hardware” this time.

For the next 40 hours, Pennsylvanians couldn’t go to the state voting website to register for next month’s election, sending a shiver through a battleground state in the presidential race.

Taxpayers, including retailers who check liabilities and payments online, were unable to access the state’s tax review website.

Systems also went down for a time at the state Department of Human Services, which runs programs for poor people.

Those systems were brought back up by early this week, but on Tuesday bar and restaurant owners who needed to order liquor were still being turned away from the state Liquor Control Board website and had to social-distance in line at a state-run liquor stores or risk running out of whiskey or vodka.

"Orders cannot be accepted at this time,” visitors at the system’s FWGS.com — fine wine and good spirits — website. (Store operations weren’t disrupted, said Wendell Young, president of the union local for liquor store clerks.)

When the company data center suffered its “infrastructure issue,” Unisys “migrated critical functions for some agencies to another system to allow for continued operations,” Unisys spokesman Brian Daly said.

What failed, and why? The state and the company were more interested in telling what didn’t go wrong.

It wasn’t “malicious activity or cybersecurity concerns," Unisys' Daly said.

The “unanticipated equipment failure” was not related to “updates or maintenance,” state spokesman Egan said.

They also wouldn’t say how long some services might be down, or how they were able to bring some services back up while repair work continues. “To protect the security of our IT infrastructure, we will not disclose details of backup operations and recovery plans,” Egan said.

Unisys holds two long-running relationships with the state. It has long been a “master IT services” contractor for equipment and services. That designation, renewed last year, enables Unisys to supply all Pennsylvania state agencies. In 2014 it was awarded a cloud services, security and disaster recovery contract through 2024.

Grove, the lawmaker from York County, noted that Pennsylvania spends at least $1 billion a year on computer systems. He said the state has a tough time keeping track of IT spending. “Information technology budgets are really buried" within agency budgets, and “augmented” through no-bid contracts, often filled by small vendors led by former state employees, he noted.

He pointed out that Pennsylvania’s infamously difficult unemployment compensation system relies on 1970s-era COBOL programming. He added that a state system has difficulty processing the hyphens in Social Security numbers.

Wolf has tried to streamline procurement by grouping departments for IT services. Some contractors complain this has added a new level to the bureaucracy.

Unlike in Congress, the two parties in the Pennsylvania House still have separate technology systems, Grove said.

He’s backing a legislative measure that he says would force branches of state government to cooperate on IT purchases and professionalize decisions on hardware and software. “My favorite part is the ‘transparency portal,’ where they have to list all the projects going, and green-light or red-light them on time and budget performance,” Grove said.

While the state’s reliance on computers has grown explosively, Grove said, so have digital problems that have gone unaddressed.

“This is a huge expenditure for us, and now we see how important it is, right before an election, with election-year theories of people trying to manipulate the election,” Grove said. “And our restaurants can’t buy alcohol."