This article was posted in a local Minneapolis paper last week. Sit back and think about your own company’s data and how much that is worth. Think about liability and what that might cost. What happened at Fairview Health Services isn’t uncommon.
There isn’t a one size fits all solution, but research what fits best for your company. Read this article and think about what a disaster like this could cost you.
Whistleblower: Fairview Health Services’ IT system keeps crashing
Fairview Health Services started small.
Founded in 1906 by a group of Minneapolis Lutherans, the hospital provided care to the city’s Norwegian immigrant community.
Over the past century, Fairview’s grown into a behemoth. Still headquartered in Minneapolis, the nonprofit healthcare organization today employs almost 25,000 staffers at various hospitals, dozens of clinics, 50-plus senior housing locations, and nearly 30 retail pharmacies.
That kind of expansion doesn’t come without growing pains. And not just when it comes to its clunky impending merger with University of Minnesota Physicians, or the game of musical chairs playing out at its president and CEO position.
The hospital system’s IT department is regularly straining to keep its systems online — and sometimes scrambling to get them working again at all.
For many years the nonprofit used global conglomerate Hitachi’s computer storage system. Hitachi served as an electronic warehouse for the volumes of medical records generated by 70,000 inpatients and 6.5 million outpatient visits each year.
The importance of a health care provider’s computer storage cannot be overstated.
It’s the foundation of the inverted IT triangle, with streams of data funneling downward through applications, to the server, to the network. The storage system is assigned with receiving, compressing, and saving all that information, like a patient’s medications history or the latest lab test results.
“Storage is critical,” says an IT professional familiar with Fairview’s system who spoke to City Pages on the condition of anonymity because he’s still employed in the field. “In compressing all that information up front, it’s working super hard and must be 100 percent active and performing functionally. Otherwise, you can have problems.”
In the fall of 2015 Fairview installed a new storage system. Hitachi’s successor, EMC, a company owned by the multinational corporation Dell, supposedly would be a state-of-the-art replacement. But the Dell EMC system is having stubborn problems that are affecting other crucial IT components.
According to internal Fairview documents, glitches related to the EMC storage system are limiting care givers’ access to Epic, a data system in use at Fairview and many other American hospitals. Epic’s applications are responsible for everything from registering a patient and scheduling blood work to fulfilling pharmacy orders.
In some instances, Fairview staff have intermittent access to the software. In others, chronic issues cause the entire system to be shut down.
And that, in turn, is creating issues for Fairview and its patient caretakers, according to documents obtained by City Pages, and interviews with current employees at the hospital system, who all agreed to speak only on the condition of anonymity for fear of professional repercussions.
One employee has reached out to former Minnesota Attorney General Mike Hatch.
“I can confirm I have met with one of the employees, the whistleblower, if you will, who is pursuing the whistleblower matter,” Hatch says.
Hatch added that he was not personally handling the employee’s case, and said he had “forwarded on the [employee’s] message to people in the state government.”
That employee, a veteran Fairview IT worker, says under its old storage system with Hitachi, the hospital chain had one across-the-board IT system outage in 12 years. Since switching to EMC in fall of 2015, it’s had three crashes in one year.
The staffer gives an example of a random hospital patient who checks in at a Fairview hospital. The patient’s name is introduced to the system, where health care professionals can access or add to his or her medical records. If there’s a hiccup somewhere within the larger IT system, Epic can often take the brunt of it. If Epic’s not available, Fairview staff are back to pen and paper.
“So in other words,” the staffer says, “you can’t pull people’s information who are at the hospitals or clinics. So there’s the potential for an impact for whatever they’re going to have done.”
The story of Fairview’s IT problems begins almost two years ago. With its existing Hitachi system needing an upgrade, Fairview was in the market for a new deal. Tasked with finding it was the nonprofit’s newly hired vice president of infrastructure Don Tierney.
This was no small decision. Nor would it come cheap. The system, for instance, would have to fluently interface with Fairview’s more than 1,500 computer applications. The storage hardware and accompanying software had a total price tag of roughly $3 million.
Various companies courted Fairview and Tierney: Hitachi, IBM, Computex and Pure Storage, and a company called EMC. The nonprofit’s IT staff favored sticking with Hitachi and springing for an upgrade.
Tierney awarded the contract to EMC.
“[Tierney] basically said to us, ‘This is what we’re going to get, and you guys don’t have a choice,'” says a Fairview employee. “I have to think they now, at least somewhat, regret that decision. Because the product that they bought wasn’t ready, wasn’t fully baked to handle what it was purchased to do.”
Among the incidents seen since the storage switch was a mid-April ordeal lasting parts of two days, in which “several of our technology systems, including Epic… were behaving inconsistently and a major outage was declared,” according to an April 22 email from Fairview Chief Information Officer Jacques Alistair, Tierney, and another Fairview vice president, Julie Flaschenriem.
The group email, addressed to the Physician and Ambulatory Informatics committee and Nursing leadership, among others, says “intermittent access problems” began “around 2:30 p.m.” It goes on to say that “[a]t 4:50 p.m. access to Epic was disabled for all users; for patient care, it was riskier to have inconsistent access versus no access to Epic.”
The problems began to get reconciled “at 6:30 p.m. and the last hospital finished their reconciliation processes around 11 p.m.,” the email continues.
In this episode, the “major outage” resulted in “access to Epic to freeze” — meaning doctors and nurses couldn’t open the software program they use almost constantly — according to an internal email, which also cites problems with “users’ access, inability to log in and system slowness.”
Tierney would admit as much months later in a Fairview document, which begins, “When systems — Epic or otherwise — are down, taking care of patients becomes more difficult.”
He continues: “IT fully recognizes just how disruptive outages are for everyone, especially to those providing patient care.”
An unreliable IT system raises the potential of compromised care, according to a former Fairview nurse, who worked for the hospital system for eight years starting in the mid-2000’s.
“In the [Epic] system, it does everything for us,” she says. “For the office visit, we enter in all the patient’s vitals, the history, the lab orders are ordered that way, any types of scans are ordered. It’s all electronic surgery scheduling.”
She gives the example of waiting for blood readings for enzymes on a patient who might have experienced a “cardiac event.” In that situation, there’s not a moment to spare.
“If you can’t get that reported to you right away,” the nurse says, “that the patient had a cardiac event — and the Epic is down, and you can’t see it and the person in the lab can’t see it — it wastes time. And it puts the life of the patient in danger.”
Adds a current Fairview employee, “However many years ago, everything was put down on paper. When there’s outages, every clinic, every hospital has downtime procedures when everything is written down by hand. So basically after the computer systems come back up, [staff has] has to go back and key in all that information. But if something gets missed, something gets thrown away, a paper gets lost, it’s kind of a bad deal.”
Hatch, who has reviewed some of the same internal documents obtained by City Pages, agrees.
“You’ve got a major hospital with 20,000 people working there,” he says. “You want to make sure everything is operating in the patients’ best interests. These communications and failures should raise concern.”
In recent months Fairview’s IT issues haven’t improved.
On September 1, “a major outage was declared” just after 9 a.m., an email written later that same day by Tierney acknowledges.
“I’d like to begin by recognizing and apologizing for the difficulties this — and all — system outages cause,” it says. “We know outages cause tremendous complications related to patient care and satisfaction, and for many of you, they make your jobs more difficult.
“Today’s event was a result of too much activity occurring on recently implemented storage system.”
Internal documents show the “event” started “around 8:30 a.m.”
Just after 9:00 a.m. that morning, Fairview IT cardiology manager Patty Vondlerstine wrote, “Users can’t access Epic,” tagging her email “High” importance.
The email chain in the ensuing hours instructs staff to contact Fairview “Operations” for any closing of departments such as “Clinics, OR’s, etc.” It also instructs Fairview’s pharmacies, “for patient safety, [that they] do not update medication records for patients who have moved location” since the outage began.
The issues lasted for hours. All Epic users weren’t granted full access to the system until 7:05 p.m. — more than 10 hours after issues were first reported — according to one of Tierney’s September 1 emails.
“Having to write everything down then input it into the system once it’s back up, I think, really opens you up for human error,” says the former Fairview nurse, who’s worked in the field for three decades. “You can’t order labs electronically so you have to pull a paper lab order sheet, write it down, send somebody to the lab to get this done. Then they’re writing this down. And you just hope everything will get re-entered the way it should be when it goes back up.”
The nurse calls her former employer’s IT problems “a huge deal” for those tasked with on-the-floor patient care.
“I can’t come up with a specific life-threatening situation off the top of my head,” she says, “but if you can’t verify who somebody is, their vitals, what medications they need, what the labs say, if somebody doesn’t get something they were supposed to or if they get something they weren’t supposed to, it sets you up for a huge liability and the possibility of a lawsuit.”
Fairview declined to get into specifics about its IT system and its outages during the past year. Camie Melton Hanily, director of communications and public affairs for the hospital system, sent the following statement in response to City Pages’ questions:
“Patient safety is always our top priority. Like other health care organizations, we have well established plans and processes for care continuity in instances when a particular tool or system is unavailable. It is not our policy to comment on specific patient or employee circumstances.”
For those of you that went to the cloud, or are thinking of moving to the cloud, can you answer with certainty who owns your data? This isn’t a new question or new controversy, as big data gets bigger, as more data is stored in the cloud, as more devices hit the market and as more hackers are getting into our banks and government servers do you own your data and if not who does and where is it?
Over the past few years there has been controversy over the “Cloud” and who owns what data where. For those willing to play “who’s data is it anyway”, the legal issues aren’t getting any clearer.
As a consumer user of the cloud -posting my so important pictures of my yellow lab Riley on Facebook
Riley, the best dog ever!
or using my Gmail account or the obsessive habit of using my Amazon prime account so I can feel like a kid at Christmas every day seeing a box on my doorstep, I fail to realize the pain of this issue: probably because it’s so convenient. I then suggested the cloud as a solution for one of my customers. That’s when it hit me.
As someone working in technology (ok I’m a sales geek) I need to really think about how real and complicated this issue is to better serve my customers by educating them in the pros and cons of using the cloud
For those thinking of going to the cloud, it seems like such and easy thing. So you call Mr. Cloud company and say “Mr. Cloud company, I want to put my data way up in the cloud so no one can get it.” Mr. Cloud says ok “we’ll store your data and all will be safe in the world forever and ever amen.” You sign the contract there, you’re in the cloud. You’re happy your data is safe, no one will ever get your data, you will have access to it at all times and you don’t have to hire and pay someone to support it.
What you don’t ask Mr. Cloud company is “what is the trail of your cloud”? Why would you ask that? What is the “trail of the cloud” Clouds don’t trail! Have you ever looked up and seen those long skinny clouds? Yeah, they trail.
In some cloud companies you give your data to a cloud provider who then outsources its work to another storage or process provider, who’s responsible if your information is lost or damaged? What if that outsourcing happens in another country? So if data is created in one country, but then stored in another the legal rules that apply become blurred. YIKES!!
Now you worry about your data. You call an attorney. What area of law is this? Cloud law isn’t a thing……yet. There are 3 main areas of law (and maybe more) that cover this: Copyright, Confidentiality and Contract. So do you need 3 attorneys? Also if your data is stored or outsourced in another country do their rules apply?
In speaking with a customer of mine from a University, he mentioned real concern about security in regard to the cloud. He mentioned his concern regarding student personal information as well as student loan information. Once student loan information is breached now we have a tax payer issue, and as he put it, “now we have a federal issue”.
There are guidelines from PTAC – the US Department of Education’s Privacy technical assistance center. But it gets a little “cloudy” regarding the cloud.
What it boils down to is Data Mining or Big Data. In the education world to use as an example; this is a huge no no as it violates the “no commercial use of student data” policy. According to Education Weekly when the litigation started in 2014, consent was not given to scan or index emails under the Google for education platform.
This issue isn’t any clearer in 2016 as UC Berkeley has this lawsuit pending for the same thing, called. “UC Berkeley students sue Google Alleging their emails were illegally scanned”.
I have only discussed a bit of the issue, but how about your industry? How about your data? Think about what you personally put out there? Your buying habits, your search habits. What about when you are in crisis? Is that something we want out in the “cloud”?
Should you decide to go to the cloud, read your contracts, ask some questions. Make sure the provider can specify who will be responsible for the data should it be lost or stolen. There should also be a provision in that contract as to who is responsible if the cloud company goes bankrupt, or is purchased by another cloud company.
If you need some help to get started in this process, help is here just give us a call.
Dell and EMC are getting married again. In this being their second marriage, they are taking Compellant and Powervault, but leaving Equallogic as the possession sold in the Saturday morning garage sale.
What does this mean for the current Equallogic users? The support is ending, to extend support if even possible, is really expensive. So you, the IT person says, “well I’ll limp along until my budget allows me new storage”. Oh wait, no support contract, no firmware upgrades you think to yourself. “Will my critical data be unstable with no firmware updates?” I’ll be able to get hardware for a while should something fail, but… that …software……sigh.
You don’t have to be held hostage by the hardware OEM’s for your storage. The 3 to 5 year rip and replace cycle of pain, agony and expense doesn’t have to continue to be part of your daily pain and suffering. The difficult marriage to your storage can be repaired with a little information, some trust and your current hardware. Yes your CURRENT hardware if you desire to keep it. Or, get a little crazy and mix your hardware up. Live dangerously, but keep your data safe.
What makes storage smart isn’t the hardware, it’s the software. So why not get the smartest software out there that can run on ANY OEM and out perform everyone else and why not get the best least expensive hardware to run it on. www.velocitytechsolutions.com
One word…… DATACORE. Datacore San Symphony V Software defind storage is hardware agnostic. It will run on any OEM hardware. If your need is for speed check out the SP1 RECORD BREAKING SPEED: https://www.datacore.com/best-price-performance-fastest-response-time. If your need is high availability or business continuity, there are real cases of years of zero downtime. Is managing data in one pane of glass a dream for you? They have that covered too. Latency is minimal with Datacore as their parallel i/o keeps those multi cores working as they can simultaneously handle compute, networking and i/o loads with minimal hardware.
Let’s talk dollars and “sense”. At $ .08 /SPC-1 IOPS Datacore blows away the $.41/SPC-1 IOPS of and EMC VNX8000 storage array. In real dollars we can say as an example an EMC VNX8000 will run about $177,000 for a mid range storage. Datacore $38K. And oh by the way, get ready to spend more than $177,000 in 3 to 5 years when support ends and you get ready for a rip and replace that EMC array. If you want to change your hardware with Datacore in 3 to 5 years aside from the hardware you want to purchase your cost: $0.00. You don’t have to EVER buy a new license. What makes sense to you?
Datacore really does what Nutanix does for Dell, what ScaleIO does for EMC , and what On Command does for Net Apps array. The difference is you no longer have to be bullied by the OEMS to spend excessive amounts of money just for it to do the same thing Datacore can do on a JBOD, or a DAS. So keep your old hardware, buy some new less expensive hardware, or go for recertified. Keep that return on investment to invest in your organization.
The partnership of Velocity Tech Solutions and Datacore Software gives you the best of all things storage. Low cost, high availability, speed and ease of use. Check us out, ask some questions, and don’t hesitate to ask for a demo.
Velocity Tech Solutions Partners with DataCore to Deliver Hyper-converged and Software-Defined Storage Solutions
World-Class Provider of Custom Servers joins DataCore as new Value-Added Reseller
ROSEVILLE, Minn. and FORT LAUDERDALE, Fla. – December 1, 2015 – DataCore Software, a leader in Software-Defined Storage and Hyper-converged Virtual SAN solutions, today announced that it has signed Velocity Tech Solutions as a new value-added reseller (VAR) serving the United States. Velocity Tech Solutions primarily resells server and storage hardware offerings from HP and Dell, including EqualLogic.
“We uncovered DataCore as part of a market research initiative we were conducting, and were drawn to the fact that DataCore is hardware-agnostic,” stated Kay Winchell, president, Velocity Tech Solutions. “It does not matter to DataCore what hardware is running, and what is even better – the hardware does not have to be brand new. DataCore helps to position our firm in the storage space, which was exactly what we wanted in becoming a DataCore VAR.”
Velocity owns thousands of servers which enable the firm to customize and deliver an order exactly the way that customer wants it and when they need it. Now, Velocity Tech Solutions’ engineers will be able to combine the benefits of Hyper-converged storage with central SAN functionality – supporting any storage, hypervisor, server, and topology.
The Answer to the Customer’s Storage Problem is DataCore
“Because DataCore offers a software-based approach to storage virtualization, the software lives beyond devices that ‘come and go’ – thereby extending the useful life of storage investments,” explained Devi Madhavan, vice president of sales channel enablement at DataCore Software. “DataCore enables customers to reduce storage costs by up to 75% in large part due to reducing operating expenses via market purchasing power and by removing disk vendor lock-in. This fact really resonated with Velocity Tech Solutions who valued that DataCore enables the user to only pay once for intelligent software and run it on the hardware of their choice.”
Customers use DataCore’s SANsymphony-V Software-Defined Storage solution to transform disparate storage resources into a unified and efficient infrastructure that is flexible, fast, reliable and scalable. The solution virtualizes and manages the capacity, performance and data protection capabilities of multiple instances of the same or different storage devices. DataCore Software also enables virtual disks or pooled storage resources to serve as a shared storage infrastructure for physical or virtual host systems.
The key benefits that prompted Velocity Tech Solutions to resell DataCore all concern the “empowerment’ DataCore gives to its users, including:
- Substantially reducing capital and operating expenses associated with storage
- Avoiding costly hardware lock-in and opening doors to more attractive alternatives from competing suppliers
- Extending the life of storage investments and enabling users to skip expensive refresh cycles (no more software throw-away)
No More Storage Vendor Lock-in
“Before, when we heard customers say things like ‘our maintenance support is running out on our storage platform,’ we did not have an answer to that,” stated Anne Tarantino, vice president of sales at Velocity Tech Solutions. “By not having a storage ‘answer’ – we could not do the whole project for our customers. Now, because DataCore gives users true empowerment, customers do not need to ‘rip-and-replace’ perfectly good equipment.”
Velocity Tech Solutions can now sell customers an entirely virtualized solution, whereas before their expertise stopped at server virtualization.
Added Tarantino, “It is great to be able to tell our customers that with DataCore they can scale to add-on whatever storage is needed. The added beauty of the DataCore solution is that once customers buy it, they own the solution.”
About Velocity Tech Solutions
Velocity Tech Solutions is a leader in delivering high quality, off-lease servers and solutions, customized to meet each business objective. VTS specializes in solutions that empower the customer to make decisions based on need and choices that fit the project’s budget. The company’s core values are: Excellence, Integrity, and Professionalism. We are customer-focused and our culture is second-to-none. VTS has the passion to build a total IT Solution and to help remove the pain associated with any IT project. The team lives by the mantra, “We Listen, We Support, and We Deliver!” For more, go to:
DataCore is the leading provider of Software-Defined Storage and Adaptive Parallel I/O Software – harnessing today’s powerful and cost-efficient server platforms to solve the IT industry’s biggest storage problem, the I/O bottleneck. The company’s comprehensive and flexible storage virtualization and hyper-converged virtual SAN solutions free users from the pain of labor-intensive storage management and provide customers true independence from storage solution vendors that cannot offer a hardware agnostic architecture. DataCore’s Software-Defined Storage platforms revolutionize storage infrastructure and serve as the cornerstone of the next-generation, software-defined data center – delivering greater value, performance, availability, and simplicity. Visit http://www.datacore.com or call (877) 780-5111 for more information.
Velocity Tech Solutions is pleased to have a partnership with Datacore. Datacore is the leader in Software Defined Storage (SDS) and optimizes existing investments and storage infrastructure in environments with diverse hardware.
What sets Datacore apart from other Virtual San Solutions:
- Simplified management and uniformity to your storage infrastructure
- Reduced Op Ex and Cap Ex
- Increased Asset Utilization
- Increased performance and flexibility
- Greater freedom of choice.
I’d like to share this informative Webinar on Hyper Converged Storage:
Contact me for a free test of Datacore VSan Symphony software. Whether you have one location or have storage in several locations, the ability to manage all of your storage in one seemless place will make life easier.