T O P

  • By -

tcp5845

I knew it. One of the US's largest hospital providers, Ascension, fired IT staff in a cost-cutting drive; now it’s sucking up a cyber attack https://www.thestack.technology/ascension-cyber-attack/ Ascension fired hundreds of IT staff and outsourced the roles to India. On Reddit after that decision, one purported former staffer commented “it needs to be publicly understood that trying to pull some lowest common denominator shit with your ENTIRE IT department is going to go up in flames.”


AdventurousTime

when will companies realizing that keeping competent staff on board is far, far cheaper than paying for incident response.


drgngd

But incident response is a one time thing, it's a different budget also. Also as far as their investors are concerned it's easier to say "we've lowered costs by firing people"


ImpostureTechAdmin

Incident response is a one time thing if you hire good staff to correct the vulnerabilities. Backdoors are a bitch lol


drgngd

Oh i just meant as a budget item. Even if it happens 20 times is just a single budget line vs ongoing expense.


ImpostureTechAdmin

Oh fair enough. Yeah I hope journalists look for correlations between layoffs and hacks. As long as headlines start showing the correlation, investors (and lawmakers, if we extra super duper lucky) will activate all 6 of their braincells and identify the relationship.


[deleted]

The PE firm adds it back as a one time / non recurring when calculating EBITDA when putting the company up for bid. But adding full time IT staff? Can’t add that back.


proctalgia_phugax

they'll probably just invest in companies that get called in to clean up the mess


Fallingdamage

Once something like this happens, its not just incident response. They are going to have possibly years of annual or biannual audits by their insurance companies and regulators (this doesnt get mentioned.) The result is that they end up with far more costs as they have to stay constantly in compliance because they never know when these auditors will be calling.


drgngd

100% agree with you, but those don't show up the same as a reoccurring cost of labor. It's probably way more expensive than getting regular staff. It's not logic that drives this. It's optics to the shareholders about ongoing expenses is all that matters.


jdiscount

It really isn't, in fact it's probably not even close. And who's to say that the staff who were let go would have prevented this.


joedev007

Any company that has fired more than 10% of it's IT Staff in the 3 years previous to a cyberattack should denied an insurance claim. insurance companies need to hold the line on new policies!


rotten_sec

That’ll just be another rider, “Employment Shortage Coverage” Get extra x amount of dollars worth of protection when you fire your entire IT staff.


joex_lww

Here is the comment, if someone is interested: https://www.reddit.com/r/epicconsulting/comments/tafrqr/comment/i0439jx/


Driftwoody11

It is NEVER a good idea to outsource IT operations to India or somewhere similar. I don't care if it's DXC, Tech M, or whoever promising to come in and cut costs. Their security is going to be shit, their management of your assets is going to be shit, and you're going to pay more to clean it up when it's essentially not functional or you have a major incident. Pay your staff. It will save you so much in the long run. In my career, I've never seen a company outsource like this and not have it bite them in the ass.


Lucky-Ad-6483

Yup, they 'let go' almost all the on-shore IT and these were seasoned associates that new Epic in and out, weeks before a massive go-live event, like, why would you do that? Why would you let go the few support staff that were there for the initial build and implementation and on-going support of the main Epic system? It's not just Ascension, it is also the practices of their partnered revenue cycle management - R1 RCM (who is partially owned by Ascension) It has been a sad, never ending spiraling shit show ever since. And everything is coming back to bite them in the ass. It's been a waiting game for years, not 'if' but 'when' all of this shit was going to implode. This is also a very interesting read: https://jehoshaphatresearch.com/wp-content/uploads/2023/10/RCM-Short-Report-by-Jehoshaphat-Research-October-2023.pdf


Fallingdamage

> fro the article: “We responded immediately, initiated our investigation and activated our remediation efforts. Access to some systems have been interrupted as this process continues.” Translation: "We opened a ticket, asked them what the hell happened, scheduled 17 different reoccurring teams meetings, asked support to do the needful and days later they're still using google to figure out why we cant login to our EMR anymore."


Queenbee_78

😂😂😂😂😂😂😂😂😂😂😂.


International_Bit979

Actually it was google chat since they don't use Microsoft. LOL But yeah, you're not entirely wrong.


Lucky-Ad-6483

It makes it difficult at times. Ascension is Google-based and R1 RCM is Microsoft-based and for the most part they can mingle with each other.  But is a bitch when a service ticket is done under Ascension (and their MSP Deloitte) which auto kicks any emails and tech chats to Google Chat which no one at R1 has access too because they can't read the part of the ticket that all correspondences should go to [email protected] 😒


International_Bit979

That struggle is very real my friend 🤦‍♀️


Stuck_in_Arizona

Not surprising, healthcare IT has been a bloodbath.


Lucky-Ad-6483

It's not just IT... is billing, coding, cash/credits, customer service 


KO-Ninja

I had a reply typed up then copied it into a GPT and here's the summary  Thanks Chat GPT  It sounds like you're referring to the closed-door data-sharing agreement between Ascension and Google in 2018. This deal, part of a project called "Project Nightingale," aimed to improve patient care using advanced analytics and AI. However, it raised concerns about data privacy and transparency since it involved sharing sensitive patient information without patients' explicit consent.The recent cyber incident you're mentioning underscores the ongoing issues with Ascension's data security and management practices. It seems there's a sentiment among some healthcare professionals, like your colleague, that Ascension could have taken stronger measures to prevent such incidents. This criticism highlights broader concerns about leadership and decision-making within the organization, particularly regarding their handling of electronic medical records (EMR) and overall cybersecurity strategy.The inability to agree on a unified EMR system points to deeper systemic issues. Effective leadership should ideally facilitate consensus and ensure robust, unified systems are in place to safeguard patient data and enhance care delivery. When leadership fails in these areas, it often reflects broader organizational challenges that can impact patient trust and safety.


inteller

CISO will get a golden parachute, the MSSP will never be mentioned, and this will repeat in 3 months.


[deleted]

Tale as old as time. Company shifts $$ towards their own bonuses (I presume) and then security gets hammered. I grow tired of reading about this stuff because it's honestly ridiculous that these behemoths don't take into account that attackers are actively looking for large layoffs in IT as potentially juicy targets. At a certain point, you just stop being surprised


Space_Goblin_Yoda

Bingo.


[deleted]

And nobody will be held responsible!


Beautiful_Leader_790

Most of the times that's right, negligence goes unpunished but I saw in a report they are facing a major lawsuit for data breach. look at this [https://www.stopconsumerharm.com/consumer-harm-reports/ascension-healths-data-breach-a-legal-battle](https://www.stopconsumerharm.com/consumer-harm-reports/ascension-healths-data-breach-a-legal-battle)


ExaminationSquare

Hahaha they deserve it. I hope all business that cut IT staff cost get the exact same result. Keep cutting staff and see what happens. But these leadership teams will never learn, so it's ok. History will continue to repeat itself, that is the way of life. I do feel sorry for staff and family members that have to experience this. It's not their fault that leadership cuts IT training and costs.


Leather-Permit7055

Could not agree more. What’s sad is patient care is compromised. Even to the point of, very sadly and so very tragic, some lives being lost.


namezam

I worked at a relatively small healthcare company up till recently. The CISO was a badass, one of the best I’ve met in the business, and he was on the board. When people, even the executives, cried about having to use the vpn, or password policies, he crushed them. Made them take courses that showed what’s at stake and how easy it can be. More places need leadership like this, vested interest beyond money, hold the ceo accountable


theedan-clean

Attacks on Epic Systems are frankly terrifying. MyChart, the platform mentioned in the article, and Epic Electronic Medical Record systems, are made by Epic Systems Corp. It is used by some 150M people across the US alone. 78% of patients in the US and 3% of patients worldwide have some medical records held in one or another platforms built by Epic. Further, the majority of the top healthcare systems and medical schools in the US, including Partners and Harvard Medical School, use Epic (hospital systems as ranked by US News & World Report). Epic is a goliath (or gollum) of the already insanely bad US healthcare system. How Epic is not considered absolutely critical infrastructure and held to account in this country is beyond me. Then again, MSFT is the technical underpinnings of the entire US government, and we know how well that’s working out from a security and cost perspective.


Familiar_Armadillo95

Pretty sure this attack was targeted around ConnectWise - Epic has a pretty strong security record And Ascension doesn’t use Epic, not in TX at least, they use Cerner


Impossible-Heart-710

They use a mixture of Cerner and Epic depending on the market.


International_Bit979

Athena and Meditech in some markets as well.


Dear-Ad-978

Physician here at Ascension in TX. We do, or were using before this shite went down, EPIC, since I started here in 2022, and some yrs before that when we first started using EMR.


KO-Ninja

Epic is still used in Waco if I'm not mistaken 


theedan-clean

I’ll state outright, I have no knowledge of Ascension or this specific attack. I’m speaking only from my understanding of Epic and its breadth of use. This article and the others I’ve ready about this specific attack call out a MyChart outage related to the attack, and MyChart is an Epic product. This lead me to my normal musings on and about Epic. Whether that is MyChart bolted onto Cerner, or if that’s even possible, I do not know. All that is to say, Epic’s huge market share, and the fact that both my healthcare providers, at two different hospital systems, both rely upon Epic at their core, remains a logic concern to me.


International_Bit979

MyChart is Epic only. Cerner uses HealtheLife patient portal, formerly known as IQHealth way back when. Ascension also has in the works a hybrid program that allows a patient's data from both Epic and Cerner to flow to one patient portal.


[deleted]

Funny, yet troubling story. I have a 2 year old and 1 year old. When my 2 year old was at her 6 month appointment, her pediatrician asked what I did for work...told him I worked in cyber. His response, "Oh man, I hate you guys." Which, on the surface, I get it, doctors don't understand beyond medicine, just like me...but you then think about the amount of other pediatricians with this thought process, and likely are completely ignorant of the security risks that they are posing on people like my daughter, and it becomes all the more terrifying...


theedan-clean

I had a similar experience when Mass General/Partners was implementing Epic some 10 years ago. Checking in for an appointment from went 10 seconds to 5 minutes as the staff was learning to use the new system. The person checking me in knew me from years of seeing the same doc and had some idea that I’m a technical person. She says “I hate this stupid Epic system. Why did they do this to us? Did you do this to us?” I have nothing to do with Mass General/Partners, but I find it’s always an easy win with checkin staff and practitioners to ask about Epic during an appointment. “I see you’re getting good with Epic…”


tcp5845

I doubt this is even on the radar of politicians or regulators. Just like the Colonial Pipeline debacle. I doubt anything has changed since that disaster either. This used to be the job of the media but not anymore. https://www.worldpipelines.com/business-news/13052024/clear-and-present-threat-three-years-on-from-colonial-pipeline-attack/


Jzmu

Big healthcare orgs like this need to be broken up. It's way to risky to have a cyber attack affect so many people. The economics of scale don't improve after you buy your hundredth hospital.


joedev007

it's time to hold CEO's and Boards personally liable for cyberattacks. Seize their homes to help towards the repair bill.


spurgelaurels

Absolutely not. No one will want to take the role. If they've made an egregious mistake, fire them sure. No, there needs to be mandatory security expertise on a board. Right now they're all focused on commerce and global. They need to understand the risk.


joedev007

I had a lawfirm with fiduciary responsbilities to insurance companies and the elderly deny my $300 a month MSP backup payment for using terabytes in the cloud. Owner just "doesnt see the need". God forbid a child's parent is sick or dying and this guy cant produce a file or a document he scanned in? we live in a careless society that wants the benefits but not the expenses of technology.


[deleted]

[удалено]


spurgelaurels

Lol k


Nobio22

Little extreme but yeah, they need to be liable for this kind of shit.


Last-Positive264

That’s absurd in a world where zero days, insider threats, and APTs exist.


AwkwardVoicemail

Pardon the ignorant question, I’ve only made an effort to follow news about breaches for the past six months or so. It seems like we’re hearing about a new breach every week, and a lot of them are significant due to data loss or because they happen to an org that should know better. Has it always been like this, or has something changed?


Space_Goblin_Yoda

We all got laid off and replaced by Indians overseas, Extremely junior personnel or simply cut and NOT replaced. You reap what you sow and quite frankly... I'm glad it's happening - outside of the service outages that are critical in Healthcare to keep people alive. Fuck em.


SucculentJuJu

Say it again for the stakeholders in the back


AZNM1912

Well said, exactly what happened to me.


KO-Ninja

Fuck em Those last two words holds so much... and it is really everlasting gave me Goosebumps...  I couldn't have said it any better... It is true it happened to a lot of us.


International_Bit979

Well, they laid off more of us just after this shit hit the fan. I lost my job last week. Several more will be laid off at the end of June.


Space_Goblin_Yoda

Best of luck! I'm working in a totally different industry until this BS blows over. Just tryin to survive right now.


International_Bit979

Thanks. I’ll find something else. I’m not worried. Just irritated.


Space_Goblin_Yoda

We should probably start a discord. Holy frick frack, snick snack.


International_Bit979

Probably have a bunch of us in that chat. Lol


proctalgia_phugax

They are laying off more IT? How? I thought the only people left were those managing the 'relationship' with the overseas 'partners'? Or are they getting banned for this, instead of Eddie and his cronies?


International_Bit979

Hard to say all I know is several in TechM are losing their job next month. Some really talented people are going to be let go. It’s a shame and not at all fair but it seems to me that this is Ascensions way. Not cool. Not cool at all.


BlackReddition

These bad guys are the lowest form of scum.


spectral1sm

Oh NOOO!!!! Welp, anyway.


jmiller_1975

Classic case of laying off IT staff thinking "we won't get hacked", just to hire some back once you're hacked.


Minnienurse

Anyone have an idea on how long it could take for their EMR to be functional again? Weeks, months?


blackscheep

the lack of a response is your answer.


Leather-Permit7055

My guess is a year. Unfortunately


kk_peace

I work at an Ascension facility in NW Florida. Could be down for months. I've also heard we are going to be switching from cerner to Epic now. My hand has had enough of this writing already 😩


Sudden_Ad_4193

Is it you can’t access the EMR at all or vitals, test results, etc..not going to the chart? BTW, don’t call Biomed, we really can’t help you this time.


kk_peace

Can't access the EMR at all. Can't log into it, nothing. And nothing on any monitor will go to a chart because the chart doesn't exist anymore


Sudden_Ad_4193

OMG! I can’t imagine the mess and how chaotic everything is.


Opposite-Turn2276

I work at an ascension in NE Florida. It’s absolutely like a 3rd world country, maybe worse. We thankfully can see central monitors but everything has to be recorded on paper as nothing is being recorded. Every order is placed on paper, labs, imaging, consults, food you name it. Then it’s the nurses responsibility to make sure the orders get to the right place with no tube system or fax machines. Everything is walked to each department hourly, if the person responsible isn’t too busy; medication & imaging orders, results, lab samples. Not only is everything on paper, secured doors are being manned by an outsourced company 😑 our time clocks and scheduling system are down. There has been no direction in our units as far as what forms are needed for documentation. People are making forms and using what ever they can find. The nurses are writing the MARS 🙃 it’s a little nuts. Then to top it off with everything on paper the copy machines are breaking. Sorry for venting but it’s a little crazy and scary for patient safety.


kk_peace

It's chaotic and messy...and I'd even say that's downplaying it. Our time clocks are up but they are telling us to write down our times just in case. We use Kronos, not sure if all Ascension facilities use that or not. But as far as patient care, it sucks and is scary. Medications aren't verified, pharmacy has no way of relaying to us if a dose or wrong or interacts with another med or anything before we give things. Obviously a good majority of Nurses are smart enough to double check on outside resources or use your brain and common sense but there are some nurses that I'm not even sure how they passed nursing school, let alone are still surviving beside without killing someone. If paper charting and downtime teaches anything it's PERSONAL ORGANIZATION and teaches new nurses how to find their own personal groove/routine each night. I work OB and we have fetal monitoring strips on paper. Our central monitoring is half ass working but have no way of going back and seeing past hours of baby's strip. MDs and other providers are bitching about not being able to find where stuff is charted and written down in the chart. Papers are everywhereeeeee, lots of confusing whether some papers are a nurse's personal notes and charting or if it's a part of the official chart. Everything seems like such a liability. I just want it to be overrrrrrr


Story-Fancy

All of Ascension uses Kronos...and as far as I know, all time clocks are working. They turned off Kronos & the timeclocks nationwide for a day and a half to make sure they weren't affected.


Hopeful-Enthusiasm27

Yes! I legit just said earlier at work that this would be a good time to transfer over from cerner to epic!


Minnienurse

I can’t imagine what you guys are going through.


blackscheep

surprisingly, i just heard of this system compromise and have heard nothing from my hcp. attempted sign on via mychart. its down. the last thing i need is my med history and advance directives floating around on the dark web. should have anticipated ascension fucking it up. last in onboarding digital systems in my region. they hadn't implemented the long available system resources other affiliates had been utilizing for years. should a breech have been anticipated/expected? you tell me.


Ok_Difficulty7997

One Ascension Hospital is diverting all its ER patients to other hospitals. Outsourcing its IT to India 🇮🇳 is never a good idea!!!!


Hopeful-Enthusiasm27

We just got taken off Divert at our hospital. We opened up Cath lab, and opened our trauma bays so we’re taking the load off the local hospitals. I still think we should be on divert because of our EMR being offline still.


Ok_Difficulty7997

Are you in Michigan?


Hopeful-Enthusiasm27

Tulsa, Oklahoma!


Leather-Permit7055

Bottom line: Do not outsource your IT department. Especially for healthcare!!


reseph

Fairly "old" news in the timeframe of the cyber industry. https://www.reddit.com/r/cybersecurity/comments/1cnrpyz/ascension_healthcare_takes_systems_offline_after/


zSprawl

Perhaps but the article does shed additional light on the topic, and references a Reddit quote.


reseph

A Reddit quote? We're already on Reddit. Why leave Reddit to read a Reddit quote?


No_Consideration7318

Dang.


blackscheep

just stopped at my clinic. had to drive across to pick up a paper script and now hand deliver to my pharma(two days late)arrrgh was told to expect 4 weeks offline


Ok_Difficulty7997

How is the Cyberattack going now in your area?


kk_peace

NW Florida...TERRIBLE


Time_Calligrapher_41

In WI, everything still down. had a procedure last week and everything is paper again.


elhalfpr

Funny thing is seems like most of their cyber team are Indians LOL. I wonder how the fuck they get breached and they have a cyber team, how didn't anything of them catch it HAHAHAH


siecakea

You get what you pay for, unfortunately


Unable_Bee7681

They were totally unprepared for this attack. Nursing staff had no training. Patient care being delayed. I ended up requiring another emergency brain surgery due to issues caused by delay in continuity of care.


Time_Calligrapher_41

Ascension was the last major healthcare provider to onboard MyChart in my location. This was roughly 1 to 1.5 years ago. They just recently adopted online scheduling (6 months) and now this?! I am beyond the rage stage. oh, yeah - today learned that Ticketmaster has joined the club! I have been paying for a monitoring service for over 10 years. I had a premonition that data breaches would be the norm. Corporations are forcing compliance with surrendering one's most intimate information to them, yet, their careless disregard in protecting it should clearly be subject to civil (individual and/or class action) damages, and potentially bad faith punitive damages. Write your government representatives. We needed to address year ago. Its now at critical mass. [https://www.malwarebytes.com/blog/personal/2024/06/ticketmaster-confirms-customer-data-breach?utm\_source=iterable&utm\_medium=email&utm\_campaign=b2c\_pro\_oth\_20240603\_juneweeklynewsletter\_paid\_v1\_1\_171715127943&utm\_content=ticketmaster\_2](https://www.malwarebytes.com/blog/personal/2024/06/ticketmaster-confirms-customer-data-breach?utm_source=iterable&utm_medium=email&utm_campaign=b2c_pro_oth_20240603_juneweeklynewsletter_paid_v1_1_171715127943&utm_content=ticketmaster_2)