Spring Break is upon us. Each year around this time, I think about one of my favorite Velocity Tech Solutions “saving asses among the masses” stories.
This is Minnesota and spring break can feel more like winter solstice, so when families can actually get a room at the Water Park of America they feel like a Florida vacation without being in Florida. That is a BIG Deal!
I was the lucky one to carry the emergency phone that night. I happened to be at a social event with our President, Kay Winchell, which was a bonus for the customer since she is a network engineer in disguise.
I answered the phone at 10pm and I heard panic. A server was down and so was the Water Park of America! Kids were crying, parents were screaming, parents were weeping. It was pandemonium! The tech from Water Park of America was beside himself. He had spent the entire day dealing with another vendor and paid for parts to be couriered across the Twin Cities multiple times and the server was still not repaired. That vendor had not asked questions and just kept throwing parts at him without helping troubleshoot the issue.
We met him at the office and he was so exhausted he couldn’t remember the model of the server . We walked around our warehouse for an hour (It’s a big warehouse!) and he finally saw the server that was the same model.
The tech rushed back with the cache module that was needed and within an hour the Water Park of America was up and running and our customer was ecstatic! (And so were the parents that promised the kids the water slide that was as dry as a bone all day long!) We helped the tech become the Hero instead of the Villain. The only thing that could have gone better was if he had found us much earlier!
How important is cyber security to you? You folks in education, and healthcare how about you? IT professionals, I know you have your pretty little firewalls, but seriously does your software update every minute so it catches every threat out there? Perhaps its “good enough” It’s 2018, in case anyone hasn’t noticed and Cyber Security continues to be a huge issue for every business that has a network. That means EVERY business out there is subject to these threats.
Hi, I’m Nate with Velocity Tech Solutions in Roseville, MN. I’m here to teach you, how to clear the event log on your Dell R710, R810, and R910 servers, also if you need to get rid of that troubling Amber LCD screen on the front of your server.
When it comes to the 11th generation servers, there are 2 ways to clear the event log. 1 of them does NOT require a restart, which is nice if your server is up and running. If your server is not booting into the OS, or if the iDRAC web interface is not working, there is a 2nd way involving the hardware that requires a restart. Let’s go through each method one at a time.
1st Method: iDRAC web interface.
This method is great if you don’t want to restart your server, your iDRAC is configured with a known IP address, and your machine is up and running. You can do this method without internet access, as long as you can access your server via an IP address.
- To do this, the first step is to log into the IP address using your web browser. Mine is set to the default setting, which is 192.168.0.120.
- Now you’ll be prompted with your user ID and password for your iDRAC. You should know this information, but if this is your first time accessing your iDRAC this way, the defaults are “root” for the username, and “calvin”, all lowercase, for the password. Make sure the drop-down box says “this iDRAC” and then click submit.
- At the next screen, if you have never been to this web interface before, it will most likely ask you to change the default username and password. If you see that screen, do this now and go through your typical company’s protocol for storing and remembering passwords.
- After the login, you will come to this screen (main page for iDRAC web interface). You need to navigate to….The Dashboard.
- a) On the right side of the dashboard, there are a few quick launch tags.
Click on View system event log.
- Here’s a chance to see what is in your event log. Just make sure that nothing in the log is unexpected.
- Click “clear log”. At this point, your event log should be cleared. You can log out of the iDRAC if you have nothing else to do here. Wait a few minutes, and the LCD screen on the front of your machine should go from Amber to the standard blue, indicating that there are no persistent errors at the moment. If after a few minutes, the screen is still amber, make sure to go through the errors using the buttons on the screen. If you are still getting an error, it could be that the problem is persistent and something in your machine is not ideal and needs to be fixed before clearing the event log will bring the screen back to standard blue. An example of this would be if your raid cables were missing or plugged into the wrong ports. In that instance, the LCD Amber error light will not go away until the machine has detected new Raid cables in the machine and then the machine is rebooted again.
One last thing to note here is that if you open the lid on your server, but have no other errors when your machine boots up, you will get an Amber LCD screen for only a minute while the machine boots and the error will say “intrusion”, but this will go away after about a minute and the LCD screen will go back to blue.
2nd Method: Hardware way, using Ctrl + E on bootup
- For this method, the first thing we need to do is restart the server. Make sure you have your company’s permission before you continue.
- Once you are rebooting, you will see a splash screen for either Dell or the maker of your machine.
- Then you will get to the next POST screen, which displays all the information of your machine and starts listing options. The option you are looking for will say “Press Ctrl + E to enter remote access setup within 5 seconds…” at the bottom of your screen. Press Ctrl + E immediately when you see that.
- If it worked, you will enter this screen. Simply use the down arrow key to navigate all the way to the bottom of the list where it says (THIS). Hit enter.
- After 10 seconds, you will be given two options, to either view or clear the event log. You can clear it if you want, but this is a GREAT opportunity to see what is in the event log. If you are having issues with your server hardware, this is a great place to start looking, but if you simply need to clear it, just use the clear option and hit enter. Clearing the log should be instantaneous.
- Once it’s cleared, hit escape until you exit the Remote Access screen. At this point, your machine will continue to boot up as normal. Your LCD screen should go back to the standard blue soon if there are no persistent errors. If it remains Amber after a minute, use the arrows on the LCD screen to see what errors are still coming up.
This finalizes the steps you should take to clear the event log on the Dell PowerEdge Models: R710, R810, R910
Stay tuned to see our video on these methods! Check out our website:
This article was posted in a local Minneapolis paper last week. Sit back and think about your own company’s data and how much that is worth. Think about liability and what that might cost. What happened at Fairview Health Services isn’t uncommon.
There isn’t a one size fits all solution, but research what fits best for your company. Read this article and think about what a disaster like this could cost you.
Whistleblower: Fairview Health Services’ IT system keeps crashing
Fairview Health Services started small.
Founded in 1906 by a group of Minneapolis Lutherans, the hospital provided care to the city’s Norwegian immigrant community.
Over the past century, Fairview’s grown into a behemoth. Still headquartered in Minneapolis, the nonprofit healthcare organization today employs almost 25,000 staffers at various hospitals, dozens of clinics, 50-plus senior housing locations, and nearly 30 retail pharmacies.
That kind of expansion doesn’t come without growing pains. And not just when it comes to its clunky impending merger with University of Minnesota Physicians, or the game of musical chairs playing out at its president and CEO position.
The hospital system’s IT department is regularly straining to keep its systems online — and sometimes scrambling to get them working again at all.
For many years the nonprofit used global conglomerate Hitachi’s computer storage system. Hitachi served as an electronic warehouse for the volumes of medical records generated by 70,000 inpatients and 6.5 million outpatient visits each year.
The importance of a health care provider’s computer storage cannot be overstated.
It’s the foundation of the inverted IT triangle, with streams of data funneling downward through applications, to the server, to the network. The storage system is assigned with receiving, compressing, and saving all that information, like a patient’s medications history or the latest lab test results.
“Storage is critical,” says an IT professional familiar with Fairview’s system who spoke to City Pages on the condition of anonymity because he’s still employed in the field. “In compressing all that information up front, it’s working super hard and must be 100 percent active and performing functionally. Otherwise, you can have problems.”
In the fall of 2015 Fairview installed a new storage system. Hitachi’s successor, EMC, a company owned by the multinational corporation Dell, supposedly would be a state-of-the-art replacement. But the Dell EMC system is having stubborn problems that are affecting other crucial IT components.
According to internal Fairview documents, glitches related to the EMC storage system are limiting care givers’ access to Epic, a data system in use at Fairview and many other American hospitals. Epic’s applications are responsible for everything from registering a patient and scheduling blood work to fulfilling pharmacy orders.
In some instances, Fairview staff have intermittent access to the software. In others, chronic issues cause the entire system to be shut down.
And that, in turn, is creating issues for Fairview and its patient caretakers, according to documents obtained by City Pages, and interviews with current employees at the hospital system, who all agreed to speak only on the condition of anonymity for fear of professional repercussions.
One employee has reached out to former Minnesota Attorney General Mike Hatch.
“I can confirm I have met with one of the employees, the whistleblower, if you will, who is pursuing the whistleblower matter,” Hatch says.
Hatch added that he was not personally handling the employee’s case, and said he had “forwarded on the [employee’s] message to people in the state government.”
That employee, a veteran Fairview IT worker, says under its old storage system with Hitachi, the hospital chain had one across-the-board IT system outage in 12 years. Since switching to EMC in fall of 2015, it’s had three crashes in one year.
The staffer gives an example of a random hospital patient who checks in at a Fairview hospital. The patient’s name is introduced to the system, where health care professionals can access or add to his or her medical records. If there’s a hiccup somewhere within the larger IT system, Epic can often take the brunt of it. If Epic’s not available, Fairview staff are back to pen and paper.
“So in other words,” the staffer says, “you can’t pull people’s information who are at the hospitals or clinics. So there’s the potential for an impact for whatever they’re going to have done.”
The story of Fairview’s IT problems begins almost two years ago. With its existing Hitachi system needing an upgrade, Fairview was in the market for a new deal. Tasked with finding it was the nonprofit’s newly hired vice president of infrastructure Don Tierney.
This was no small decision. Nor would it come cheap. The system, for instance, would have to fluently interface with Fairview’s more than 1,500 computer applications. The storage hardware and accompanying software had a total price tag of roughly $3 million.
Various companies courted Fairview and Tierney: Hitachi, IBM, Computex and Pure Storage, and a company called EMC. The nonprofit’s IT staff favored sticking with Hitachi and springing for an upgrade.
Tierney awarded the contract to EMC.
“[Tierney] basically said to us, ‘This is what we’re going to get, and you guys don’t have a choice,'” says a Fairview employee. “I have to think they now, at least somewhat, regret that decision. Because the product that they bought wasn’t ready, wasn’t fully baked to handle what it was purchased to do.”
Among the incidents seen since the storage switch was a mid-April ordeal lasting parts of two days, in which “several of our technology systems, including Epic… were behaving inconsistently and a major outage was declared,” according to an April 22 email from Fairview Chief Information Officer Jacques Alistair, Tierney, and another Fairview vice president, Julie Flaschenriem.
The group email, addressed to the Physician and Ambulatory Informatics committee and Nursing leadership, among others, says “intermittent access problems” began “around 2:30 p.m.” It goes on to say that “[a]t 4:50 p.m. access to Epic was disabled for all users; for patient care, it was riskier to have inconsistent access versus no access to Epic.”
The problems began to get reconciled “at 6:30 p.m. and the last hospital finished their reconciliation processes around 11 p.m.,” the email continues.
In this episode, the “major outage” resulted in “access to Epic to freeze” — meaning doctors and nurses couldn’t open the software program they use almost constantly — according to an internal email, which also cites problems with “users’ access, inability to log in and system slowness.”
Tierney would admit as much months later in a Fairview document, which begins, “When systems — Epic or otherwise — are down, taking care of patients becomes more difficult.”
He continues: “IT fully recognizes just how disruptive outages are for everyone, especially to those providing patient care.”
An unreliable IT system raises the potential of compromised care, according to a former Fairview nurse, who worked for the hospital system for eight years starting in the mid-2000’s.
“In the [Epic] system, it does everything for us,” she says. “For the office visit, we enter in all the patient’s vitals, the history, the lab orders are ordered that way, any types of scans are ordered. It’s all electronic surgery scheduling.”
She gives the example of waiting for blood readings for enzymes on a patient who might have experienced a “cardiac event.” In that situation, there’s not a moment to spare.
“If you can’t get that reported to you right away,” the nurse says, “that the patient had a cardiac event — and the Epic is down, and you can’t see it and the person in the lab can’t see it — it wastes time. And it puts the life of the patient in danger.”
Adds a current Fairview employee, “However many years ago, everything was put down on paper. When there’s outages, every clinic, every hospital has downtime procedures when everything is written down by hand. So basically after the computer systems come back up, [staff has] has to go back and key in all that information. But if something gets missed, something gets thrown away, a paper gets lost, it’s kind of a bad deal.”
Hatch, who has reviewed some of the same internal documents obtained by City Pages, agrees.
“You’ve got a major hospital with 20,000 people working there,” he says. “You want to make sure everything is operating in the patients’ best interests. These communications and failures should raise concern.”
In recent months Fairview’s IT issues haven’t improved.
On September 1, “a major outage was declared” just after 9 a.m., an email written later that same day by Tierney acknowledges.
“I’d like to begin by recognizing and apologizing for the difficulties this — and all — system outages cause,” it says. “We know outages cause tremendous complications related to patient care and satisfaction, and for many of you, they make your jobs more difficult.
“Today’s event was a result of too much activity occurring on recently implemented storage system.”
Internal documents show the “event” started “around 8:30 a.m.”
Just after 9:00 a.m. that morning, Fairview IT cardiology manager Patty Vondlerstine wrote, “Users can’t access Epic,” tagging her email “High” importance.
The email chain in the ensuing hours instructs staff to contact Fairview “Operations” for any closing of departments such as “Clinics, OR’s, etc.” It also instructs Fairview’s pharmacies, “for patient safety, [that they] do not update medication records for patients who have moved location” since the outage began.
The issues lasted for hours. All Epic users weren’t granted full access to the system until 7:05 p.m. — more than 10 hours after issues were first reported — according to one of Tierney’s September 1 emails.
“Having to write everything down then input it into the system once it’s back up, I think, really opens you up for human error,” says the former Fairview nurse, who’s worked in the field for three decades. “You can’t order labs electronically so you have to pull a paper lab order sheet, write it down, send somebody to the lab to get this done. Then they’re writing this down. And you just hope everything will get re-entered the way it should be when it goes back up.”
The nurse calls her former employer’s IT problems “a huge deal” for those tasked with on-the-floor patient care.
“I can’t come up with a specific life-threatening situation off the top of my head,” she says, “but if you can’t verify who somebody is, their vitals, what medications they need, what the labs say, if somebody doesn’t get something they were supposed to or if they get something they weren’t supposed to, it sets you up for a huge liability and the possibility of a lawsuit.”
Fairview declined to get into specifics about its IT system and its outages during the past year. Camie Melton Hanily, director of communications and public affairs for the hospital system, sent the following statement in response to City Pages’ questions:
“Patient safety is always our top priority. Like other health care organizations, we have well established plans and processes for care continuity in instances when a particular tool or system is unavailable. It is not our policy to comment on specific patient or employee circumstances.”
For those of you that went to the cloud, or are thinking of moving to the cloud, can you answer with certainty who owns your data? This isn’t a new question or new controversy, as big data gets bigger, as more data is stored in the cloud, as more devices hit the market and as more hackers are getting into our banks and government servers do you own your data and if not who does and where is it?
Over the past few years there has been controversy over the “Cloud” and who owns what data where. For those willing to play “who’s data is it anyway”, the legal issues aren’t getting any clearer.
As a consumer user of the cloud -posting my so important pictures of my yellow lab Riley on Facebook
Riley, the best dog ever!
or using my Gmail account or the obsessive habit of using my Amazon prime account so I can feel like a kid at Christmas every day seeing a box on my doorstep, I fail to realize the pain of this issue: probably because it’s so convenient. I then suggested the cloud as a solution for one of my customers. That’s when it hit me.
As someone working in technology (ok I’m a sales geek) I need to really think about how real and complicated this issue is to better serve my customers by educating them in the pros and cons of using the cloud
For those thinking of going to the cloud, it seems like such and easy thing. So you call Mr. Cloud company and say “Mr. Cloud company, I want to put my data way up in the cloud so no one can get it.” Mr. Cloud says ok “we’ll store your data and all will be safe in the world forever and ever amen.” You sign the contract there, you’re in the cloud. You’re happy your data is safe, no one will ever get your data, you will have access to it at all times and you don’t have to hire and pay someone to support it.
What you don’t ask Mr. Cloud company is “what is the trail of your cloud”? Why would you ask that? What is the “trail of the cloud” Clouds don’t trail! Have you ever looked up and seen those long skinny clouds? Yeah, they trail.
In some cloud companies you give your data to a cloud provider who then outsources its work to another storage or process provider, who’s responsible if your information is lost or damaged? What if that outsourcing happens in another country? So if data is created in one country, but then stored in another the legal rules that apply become blurred. YIKES!!
Now you worry about your data. You call an attorney. What area of law is this? Cloud law isn’t a thing……yet. There are 3 main areas of law (and maybe more) that cover this: Copyright, Confidentiality and Contract. So do you need 3 attorneys? Also if your data is stored or outsourced in another country do their rules apply?
In speaking with a customer of mine from a University, he mentioned real concern about security in regard to the cloud. He mentioned his concern regarding student personal information as well as student loan information. Once student loan information is breached now we have a tax payer issue, and as he put it, “now we have a federal issue”.
There are guidelines from PTAC – the US Department of Education’s Privacy technical assistance center. But it gets a little “cloudy” regarding the cloud.
What it boils down to is Data Mining or Big Data. In the education world to use as an example; this is a huge no no as it violates the “no commercial use of student data” policy. According to Education Weekly when the litigation started in 2014, consent was not given to scan or index emails under the Google for education platform.
This issue isn’t any clearer in 2016 as UC Berkeley has this lawsuit pending for the same thing, called. “UC Berkeley students sue Google Alleging their emails were illegally scanned”.
I have only discussed a bit of the issue, but how about your industry? How about your data? Think about what you personally put out there? Your buying habits, your search habits. What about when you are in crisis? Is that something we want out in the “cloud”?
Should you decide to go to the cloud, read your contracts, ask some questions. Make sure the provider can specify who will be responsible for the data should it be lost or stolen. There should also be a provision in that contract as to who is responsible if the cloud company goes bankrupt, or is purchased by another cloud company.
If you need some help to get started in this process, help is here just give us a call.
Velocity Tech Solutions Partners with DataCore to Deliver Hyper-converged and Software-Defined Storage Solutions
World-Class Provider of Custom Servers joins DataCore as new Value-Added Reseller
ROSEVILLE, Minn. and FORT LAUDERDALE, Fla. – December 1, 2015 – DataCore Software, a leader in Software-Defined Storage and Hyper-converged Virtual SAN solutions, today announced that it has signed Velocity Tech Solutions as a new value-added reseller (VAR) serving the United States. Velocity Tech Solutions primarily resells server and storage hardware offerings from HP and Dell, including EqualLogic.
“We uncovered DataCore as part of a market research initiative we were conducting, and were drawn to the fact that DataCore is hardware-agnostic,” stated Kay Winchell, president, Velocity Tech Solutions. “It does not matter to DataCore what hardware is running, and what is even better – the hardware does not have to be brand new. DataCore helps to position our firm in the storage space, which was exactly what we wanted in becoming a DataCore VAR.”
Velocity owns thousands of servers which enable the firm to customize and deliver an order exactly the way that customer wants it and when they need it. Now, Velocity Tech Solutions’ engineers will be able to combine the benefits of Hyper-converged storage with central SAN functionality – supporting any storage, hypervisor, server, and topology.
The Answer to the Customer’s Storage Problem is DataCore
“Because DataCore offers a software-based approach to storage virtualization, the software lives beyond devices that ‘come and go’ – thereby extending the useful life of storage investments,” explained Devi Madhavan, vice president of sales channel enablement at DataCore Software. “DataCore enables customers to reduce storage costs by up to 75% in large part due to reducing operating expenses via market purchasing power and by removing disk vendor lock-in. This fact really resonated with Velocity Tech Solutions who valued that DataCore enables the user to only pay once for intelligent software and run it on the hardware of their choice.”
Customers use DataCore’s SANsymphony-V Software-Defined Storage solution to transform disparate storage resources into a unified and efficient infrastructure that is flexible, fast, reliable and scalable. The solution virtualizes and manages the capacity, performance and data protection capabilities of multiple instances of the same or different storage devices. DataCore Software also enables virtual disks or pooled storage resources to serve as a shared storage infrastructure for physical or virtual host systems.
The key benefits that prompted Velocity Tech Solutions to resell DataCore all concern the “empowerment’ DataCore gives to its users, including:
- Substantially reducing capital and operating expenses associated with storage
- Avoiding costly hardware lock-in and opening doors to more attractive alternatives from competing suppliers
- Extending the life of storage investments and enabling users to skip expensive refresh cycles (no more software throw-away)
No More Storage Vendor Lock-in
“Before, when we heard customers say things like ‘our maintenance support is running out on our storage platform,’ we did not have an answer to that,” stated Anne Tarantino, vice president of sales at Velocity Tech Solutions. “By not having a storage ‘answer’ – we could not do the whole project for our customers. Now, because DataCore gives users true empowerment, customers do not need to ‘rip-and-replace’ perfectly good equipment.”
Velocity Tech Solutions can now sell customers an entirely virtualized solution, whereas before their expertise stopped at server virtualization.
Added Tarantino, “It is great to be able to tell our customers that with DataCore they can scale to add-on whatever storage is needed. The added beauty of the DataCore solution is that once customers buy it, they own the solution.”
About Velocity Tech Solutions
Velocity Tech Solutions is a leader in delivering high quality, off-lease servers and solutions, customized to meet each business objective. VTS specializes in solutions that empower the customer to make decisions based on need and choices that fit the project’s budget. The company’s core values are: Excellence, Integrity, and Professionalism. We are customer-focused and our culture is second-to-none. VTS has the passion to build a total IT Solution and to help remove the pain associated with any IT project. The team lives by the mantra, “We Listen, We Support, and We Deliver!” For more, go to:
DataCore is the leading provider of Software-Defined Storage and Adaptive Parallel I/O Software – harnessing today’s powerful and cost-efficient server platforms to solve the IT industry’s biggest storage problem, the I/O bottleneck. The company’s comprehensive and flexible storage virtualization and hyper-converged virtual SAN solutions free users from the pain of labor-intensive storage management and provide customers true independence from storage solution vendors that cannot offer a hardware agnostic architecture. DataCore’s Software-Defined Storage platforms revolutionize storage infrastructure and serve as the cornerstone of the next-generation, software-defined data center – delivering greater value, performance, availability, and simplicity. Visit http://www.datacore.com or call (877) 780-5111 for more information.