This article was posted in a local Minneapolis paper last week. Sit back and think about your own company’s data and how much that is worth. Think about liability and what that might cost. What happened at Fairview Health Services isn’t uncommon.
There isn’t a one size fits all solution, but research what fits best for your company. Read this article and think about what a disaster like this could cost you.
Whistleblower: Fairview Health Services’ IT system keeps crashing
Fairview Health Services started small.
Founded in 1906 by a group of Minneapolis Lutherans, the hospital provided care to the city’s Norwegian immigrant community.
Over the past century, Fairview’s grown into a behemoth. Still headquartered in Minneapolis, the nonprofit healthcare organization today employs almost 25,000 staffers at various hospitals, dozens of clinics, 50-plus senior housing locations, and nearly 30 retail pharmacies.
That kind of expansion doesn’t come without growing pains. And not just when it comes to its clunky impending merger with University of Minnesota Physicians, or the game of musical chairs playing out at its president and CEO position.
The hospital system’s IT department is regularly straining to keep its systems online — and sometimes scrambling to get them working again at all.
For many years the nonprofit used global conglomerate Hitachi’s computer storage system. Hitachi served as an electronic warehouse for the volumes of medical records generated by 70,000 inpatients and 6.5 million outpatient visits each year.
The importance of a health care provider’s computer storage cannot be overstated.
It’s the foundation of the inverted IT triangle, with streams of data funneling downward through applications, to the server, to the network. The storage system is assigned with receiving, compressing, and saving all that information, like a patient’s medications history or the latest lab test results.
“Storage is critical,” says an IT professional familiar with Fairview’s system who spoke to City Pages on the condition of anonymity because he’s still employed in the field. “In compressing all that information up front, it’s working super hard and must be 100 percent active and performing functionally. Otherwise, you can have problems.”
In the fall of 2015 Fairview installed a new storage system. Hitachi’s successor, EMC, a company owned by the multinational corporation Dell, supposedly would be a state-of-the-art replacement. But the Dell EMC system is having stubborn problems that are affecting other crucial IT components.
According to internal Fairview documents, glitches related to the EMC storage system are limiting care givers’ access to Epic, a data system in use at Fairview and many other American hospitals. Epic’s applications are responsible for everything from registering a patient and scheduling blood work to fulfilling pharmacy orders.
In some instances, Fairview staff have intermittent access to the software. In others, chronic issues cause the entire system to be shut down.
And that, in turn, is creating issues for Fairview and its patient caretakers, according to documents obtained by City Pages, and interviews with current employees at the hospital system, who all agreed to speak only on the condition of anonymity for fear of professional repercussions.
One employee has reached out to former Minnesota Attorney General Mike Hatch.
“I can confirm I have met with one of the employees, the whistleblower, if you will, who is pursuing the whistleblower matter,” Hatch says.
Hatch added that he was not personally handling the employee’s case, and said he had “forwarded on the [employee’s] message to people in the state government.”
That employee, a veteran Fairview IT worker, says under its old storage system with Hitachi, the hospital chain had one across-the-board IT system outage in 12 years. Since switching to EMC in fall of 2015, it’s had three crashes in one year.
The staffer gives an example of a random hospital patient who checks in at a Fairview hospital. The patient’s name is introduced to the system, where health care professionals can access or add to his or her medical records. If there’s a hiccup somewhere within the larger IT system, Epic can often take the brunt of it. If Epic’s not available, Fairview staff are back to pen and paper.
“So in other words,” the staffer says, “you can’t pull people’s information who are at the hospitals or clinics. So there’s the potential for an impact for whatever they’re going to have done.”
The story of Fairview’s IT problems begins almost two years ago. With its existing Hitachi system needing an upgrade, Fairview was in the market for a new deal. Tasked with finding it was the nonprofit’s newly hired vice president of infrastructure Don Tierney.
This was no small decision. Nor would it come cheap. The system, for instance, would have to fluently interface with Fairview’s more than 1,500 computer applications. The storage hardware and accompanying software had a total price tag of roughly $3 million.
Various companies courted Fairview and Tierney: Hitachi, IBM, Computex and Pure Storage, and a company called EMC. The nonprofit’s IT staff favored sticking with Hitachi and springing for an upgrade.
Tierney awarded the contract to EMC.
“[Tierney] basically said to us, ‘This is what we’re going to get, and you guys don’t have a choice,'” says a Fairview employee. “I have to think they now, at least somewhat, regret that decision. Because the product that they bought wasn’t ready, wasn’t fully baked to handle what it was purchased to do.”
Among the incidents seen since the storage switch was a mid-April ordeal lasting parts of two days, in which “several of our technology systems, including Epic… were behaving inconsistently and a major outage was declared,” according to an April 22 email from Fairview Chief Information Officer Jacques Alistair, Tierney, and another Fairview vice president, Julie Flaschenriem.
The group email, addressed to the Physician and Ambulatory Informatics committee and Nursing leadership, among others, says “intermittent access problems” began “around 2:30 p.m.” It goes on to say that “[a]t 4:50 p.m. access to Epic was disabled for all users; for patient care, it was riskier to have inconsistent access versus no access to Epic.”
The problems began to get reconciled “at 6:30 p.m. and the last hospital finished their reconciliation processes around 11 p.m.,” the email continues.
In this episode, the “major outage” resulted in “access to Epic to freeze” — meaning doctors and nurses couldn’t open the software program they use almost constantly — according to an internal email, which also cites problems with “users’ access, inability to log in and system slowness.”
Tierney would admit as much months later in a Fairview document, which begins, “When systems — Epic or otherwise — are down, taking care of patients becomes more difficult.”
He continues: “IT fully recognizes just how disruptive outages are for everyone, especially to those providing patient care.”
An unreliable IT system raises the potential of compromised care, according to a former Fairview nurse, who worked for the hospital system for eight years starting in the mid-2000’s.
“In the [Epic] system, it does everything for us,” she says. “For the office visit, we enter in all the patient’s vitals, the history, the lab orders are ordered that way, any types of scans are ordered. It’s all electronic surgery scheduling.”
She gives the example of waiting for blood readings for enzymes on a patient who might have experienced a “cardiac event.” In that situation, there’s not a moment to spare.
“If you can’t get that reported to you right away,” the nurse says, “that the patient had a cardiac event — and the Epic is down, and you can’t see it and the person in the lab can’t see it — it wastes time. And it puts the life of the patient in danger.”
Adds a current Fairview employee, “However many years ago, everything was put down on paper. When there’s outages, every clinic, every hospital has downtime procedures when everything is written down by hand. So basically after the computer systems come back up, [staff has] has to go back and key in all that information. But if something gets missed, something gets thrown away, a paper gets lost, it’s kind of a bad deal.”
Hatch, who has reviewed some of the same internal documents obtained by City Pages, agrees.
“You’ve got a major hospital with 20,000 people working there,” he says. “You want to make sure everything is operating in the patients’ best interests. These communications and failures should raise concern.”
In recent months Fairview’s IT issues haven’t improved.
On September 1, “a major outage was declared” just after 9 a.m., an email written later that same day by Tierney acknowledges.
“I’d like to begin by recognizing and apologizing for the difficulties this — and all — system outages cause,” it says. “We know outages cause tremendous complications related to patient care and satisfaction, and for many of you, they make your jobs more difficult.
“Today’s event was a result of too much activity occurring on recently implemented storage system.”
Internal documents show the “event” started “around 8:30 a.m.”
Just after 9:00 a.m. that morning, Fairview IT cardiology manager Patty Vondlerstine wrote, “Users can’t access Epic,” tagging her email “High” importance.
The email chain in the ensuing hours instructs staff to contact Fairview “Operations” for any closing of departments such as “Clinics, OR’s, etc.” It also instructs Fairview’s pharmacies, “for patient safety, [that they] do not update medication records for patients who have moved location” since the outage began.
The issues lasted for hours. All Epic users weren’t granted full access to the system until 7:05 p.m. — more than 10 hours after issues were first reported — according to one of Tierney’s September 1 emails.
“Having to write everything down then input it into the system once it’s back up, I think, really opens you up for human error,” says the former Fairview nurse, who’s worked in the field for three decades. “You can’t order labs electronically so you have to pull a paper lab order sheet, write it down, send somebody to the lab to get this done. Then they’re writing this down. And you just hope everything will get re-entered the way it should be when it goes back up.”
The nurse calls her former employer’s IT problems “a huge deal” for those tasked with on-the-floor patient care.
“I can’t come up with a specific life-threatening situation off the top of my head,” she says, “but if you can’t verify who somebody is, their vitals, what medications they need, what the labs say, if somebody doesn’t get something they were supposed to or if they get something they weren’t supposed to, it sets you up for a huge liability and the possibility of a lawsuit.”
Fairview declined to get into specifics about its IT system and its outages during the past year. Camie Melton Hanily, director of communications and public affairs for the hospital system, sent the following statement in response to City Pages’ questions:
“Patient safety is always our top priority. Like other health care organizations, we have well established plans and processes for care continuity in instances when a particular tool or system is unavailable. It is not our policy to comment on specific patient or employee circumstances.”