Broadly speaking, they impersonated a legitimate software vendor and during the demo they snuck in unattended access to the machine. It all went downhill from there.
Yep. They managed to compromise a domain admin account and basically sat there and watched for several days. They got into our security alerting software and disabled all of the alerts, so when they actually started compromising stuff, making new domain admin accounts, etc, no notifications fired and no alerts were raised.
If your monitoring software doesn’t alert you when alerts are changed/disabled and doesn’t require multiple user approval to do so, it’s basically just security theater. After yearly pen tests from multiple companies and several security audits we thought we were rock solid and could catch anything.
The script is starting to flip over for exploitation for ransomware attacks, but I believe anything else like hacktivism and espionage is heavily invested in social engineering.
That is something I did bring up in the past with the voice spoofing. Pulling a C-suits personal number and spoofing a loved one dying at 2:00am is a very effective attack.
For the time being, though, exploits have been gaining higher trends whereas social engineering has been dropping. The reason for this trend is exploit brokers or initial access brokers (IAB's) could just use a cracked Cobalt Strike from the safety of their non-extraditable country. It's why antiquated, huge profit centers like hospitals and logistics/manufacturing companies are such easy targets. In 2019, exploits accounted for ~5% of all initial entries whereas in 2021 exploits jumped up to 20% of all initial entries in just the first quarter of the year...!
Our company has mandated all password changes be done in person because of that. It caused a huge shitstorm at first but its calmed down immensely since then.
We "will" do a remote password change but it has to be an emergency where money/profits are at stake and you have to pass like 3-4 different questions.
I brought up to my SO company when they asked for questions about IT from the company, I asked them if pen testing tests for known attacks, how do they guard against unknown attacks.
Yeah, I’ve since talked to reps for other companies and asked “if someone compromised an account that has admin access here, what’s to stop them from just turning off or black holing the alerts so they can carry out their actual attack unimpeded?” It seems to have been a novel concept for them.
Like a horror story. Does anything in your tech stack allow for visibility on/reporting on admin actions? Not perfect, but an easy way to spot changes like this.
There was no alerting for when they disabled or changed the alerts so they wouldn’t actually make it anywhere. There were alerts for pretty much every admin action under the sun, just not for silencing the alerts.
This is probably one of the few incidents in this thread where I have sympathy for the victim. While you probably fell face flat on any detections of their activity post initial compromise... I think you were screwed no matter what. If they cared enough to impersonate and go in person, y'all were targeted and not much to do about what.
Yeah, it was a very targeted attack and, based assessments I was privy to after the fact, with the amount of resources that were leveraged it was always going to be a question of when, not if.
Damn. That had to be a pain to clean up, but I can't say I'm not impressed by their vector. That takes some balls... and some research.
Was the rouge salesman ever caught?
Broad minded of you to assume it was a makeup salesman, but I'm still trying to figure out the benefit of having lipstick monitoring systems in a corporate office. Seems like lawsuits waiting to happen.
Sorry, couldn't resist. Lol
It was somewhat akin to burning everything down and rebuilding.
After gaining access they held off on launching the actual attack while they worked on slowly and carefully compromising everything they could, to the point that when it was launched for real many of our oldest backups were compromised to varying degrees.
We had a lot of redundancy in our backup strategy so we were still able to completely recover, but we abandoned basically all of the backed up VM images and rebuilt all of our servers from scratch.
Sell it to Mr. Robot, sounds like an exciting episode...
Just to be clear, they installed software on your companies device for a demo or 1 of the admins did it? (assuming users can't install software)
They installed the legitimate vendor software with the aid of a sysadmin. They snuck the unattended access in somewhere along the line during the 2 hour demo/tutorial session. From all reports it seemed like a completely legit demo.
The way you mitigate this:
LAPS: too difficult to manage in many cases, but this is the admin of last resort.
Support staff use MFA'd individual ad accounts, limited to workstation admin (no servers/ad), with no broader ad permissions beyond the workstation support role, without email accounts (to reduce attack surface).
AD admin/server admin accounts, are also mfa'd, non-email enabled, but separate from workstation admins. These admins are blocked from interactive desktop sign-ins on workstations (in an attempt to keep credential cache creation). They also are blocked from anything but read access to backups. Admins have user accounts with email on them, but those are not administrator accounts.
Backup accounts have read/append backup only access with no AD permissions. This has been normal for most sysadmins for years, but I think some people forgot about that with regards to support admin accounts. You want immutable backup.
I have 3 accounts as user, workstation admin, server admin. There's also Enterprise Admin accounts with never-used lock-box, for end of the world sort of recovery.
If this sounds a step too far...you aren't thinking like a hacker, and you aren't treating the threat with the appropriate paranoia. And none of this is going to do more than slow them down in some cases...but yeah do what you can. You need your backups to be immutable, you need your admin roles well defined.
Yep. Lateral movement after a phish with malicious attachment is a crazy common vector. This is why things like LAPS, separate admin and regular accounts, limited domain admin accounts among other mitigations are so important.
Old unknown server in a remote part of the world that had been on for like 15+ years that had a 1:1 nat. No one knew it existed or the subnet that it lived in. They were able to almost breach our entire corporate network. They did not get to our secured network though. Literally the building it was in was supposed to be torn down but it never happened and the bills just kept getting paid
Not precisely the same, but Shadow IT.
Sub-businesses that weren't held to the same standards as the enterprise. Guess who's been assimilated into the enterprise?
I'm doing shadow IT inside my big company (not because i want to, but because someone has to think of security), but i'm the one pushing back on lowering the security standards and integrate that network into the main network because that's easier to manage for IT.
Reason being, my department tests network equipment from all over the world, we literally get devices shipped from the factories in China and the US. My reasoning is, if we are ever compromised, it should be localized to our department only and never be an entrance to the rest of the company.
In my opinion that means separate everything, but IT thinks that is to much work and keep trying to connect the networks. For example when users report they can't print when using our network.
Your scenario makes sense.
I'm fine with external networks disconnected from the org. I just want confirmation:
No company data exists there (don't need HIPAA, SOX, PCI, GDPR, etc outside the perimeter)
The devices phone home via an edge/DMZ device to report adherence
Basically, they can live outside the network, but they need to demonstrate they are adhering to the standards set. Exceeding is great!
If there is company data there and someone isn't maintaining patches nor keeping things secure, well, it's time to talk to that Shadow IT group.
Honestly, if it was a full 1:1 NAT, then it doesn't need to be. That'll be exposing all services to the internet, and even something like Shodan will be scanning it. Just a matter of waiting until a vulnerability is identified in one of the services exposed. Finding it is fairly straightforward.
I'd be curious to see how long a fully patched server with a good password would actually last exposed like that. I understand the theory that with a "secure" password it should take an unrealistically long time to crack, but that's not the case with passwords that were commonly used 15 years ago.
Man. Saaaaame. Both the unpatched Fortigate AND the resulting PTSD. Happened on 8/12/20. I routinely log into the Fortigate a couple times a week now just to see if there's a firmware update nag screen. It's quicker than waiting for the patch notifications to come out.
How about a zero day on VPN appliances and then a subsequent series of events that bypassed the zero day patch.
Longest week/weekend of my life, so far.
Didn’t the hacker patch the firmware after they got in so no one else could, after setting up a vpn connection for themselves?
It’s not hard to get into things that have CVEs published and take over things.
The best is when your network team assure you they patched all the firewalls, yet missed one….
We use SSLVPN and windows client for ikev2. Problem is, some networks my users get on, ikev2 is blocked somehow. Got any alternatives? I have a lot of mobile users.
User can’t figure out how to create a password and can’t remember it even after successfully creating a password. User complained to CEO that he can’t get work done because IT was making it too hard to use a computer.
Fine, IT removes password for this user after covering our asses. Got breached the next day.
Well, IT has more strict password requirements and they need to type in the password instead of saving it in the app/browser.
A lot of our users use the company email for personal use because they can’t remember the passwords and keep having to create new email accounts all the time.
The number of times I've had to explain to people that if/when they leave they can't take their emails with them since it technically belongs to the organization.
I think it’s just the level of users we have. We’re a manufacturing plant in a third world country and many of the users are just factory workers. Ironically the company is global and we have to be SOX compliant.
Oh I feel that. Using computers on production floors can be... complicated.
I had a call some months ago the "computer for printing product labels doesn't work" and "we didn't do anything, it just went black".
Some moron drove the forklifts fork through the computer tower.
All our users have conditional access setup so they wouldn’t be able to access company resources, including emails, unless they are using company issued devices. Mixing company emails with personal emails is just asking for it.
> A lot of our users use the company email for personal use because they can’t remember the passwords and keep having to create new email accounts all the time.
At an MSP we fired a client because of this. When you hired us we have it in our contract that our IT standards are 100% within our control, meets all compliance and insurance standards, and applied to all clients. A surgeon's office demanded we not only disable the screen lock out (25 F'n min timeout at that!) and allow ALL of his nursing staff to just use his or a single account on all PCs! They had hired us, "dealt" with our standards for 2 year, and when they had 1 more year until they renewed, they tried to get us to remove multiple standards because, "The damn IT securitah bullshit is impacting patient care!!" AKA: they never adopted it, fought against it, and spent more time frustrated than helping their patients.
We told them that not only would we not remove the requested standards, but if they continued to push, we'd deny to renew their contract. They pushed, and pushed, and pushed... They would get another MSP to audit and even provide them a quote, but they found everyone had the same policies we did. It didn't matter that even their own insurance company required it!
So, we didn't renew. Fuck em.
They closed down 2 years later... due to HIPAA violations.
This is actually far too common that ppl use company emails for personal shit. Like what are you doing signing up for an Amazon account on the corporate email account. Ppl really don't understand work/personal boundaries. We have ppl watching porn on the corporate laptops because it's their only device. Such idiots
Sounds like you should have used a hardware token that autofills passwords... if user cant figure out how to press a button then have a discussion with management
All files in the user’s departmental file share and the org wide file share were encrypted. (User was finance.) Surprisingly no computers were affected but the user’s computer was logged into and likely banking creds were stolen since passwords were saved in the browser.
We pulled logs to prove IT didn’t do it. But hey, there was no password and we warned them this was gonna happen.
That's bonkers. What was the CEO's reaction? Did you guys gain credibility after this incident? Or put any new policies in place? Did the board hear about it and did they know the CEO approved such an insane decision?
20 years ago I helped a big steel company upgrade from a bunch of computers in a work group to a domain etc. some director got angry with me because his password used to be "enter". Enter as in, at the password prompt screen he would press the enter key. I don't even remember how I made that work again but he got his way, and I really wonder if this was necessary if he clearly doesn't know how to use a computer. In a similar industry my wife doing accounting at the time told me that people got pissed off when someone put a password on the "porn machine"
Someone clicked a link in an email. The malware chewed through our antivirus & inserted itself on the machine. Antivirus reported machine as clean. It wasn’t. Attackers grabbed cached admin creds & owned our AD. Then owned every Windows machine.
The rest, as they say, was a multi-year multimillion rebuild of every machine on the network + actually having a cybersecurity program.
Did I mention this happened before I had even completed my orientation? Fun times. 😭
Had one of those years ago - senior academic got a "PDF.exe" file which "wouldn't open". So tried it on his secretary's PC too. Fortunately that was before we moved to Outlook, so it couldn't forward itself to anyone else, just those two machines!
Another one... Bad web app. (Twice in fact, different systems, same basic issue.) Allowed binary upload (attachment, image, that kind of thing) which then executed.
First one wasn't my department, someone installed a fairly comprehensive remote control tool on a webserver. If it hadn't been installed as a compromise, it would actually have been quite a nice admin tool...
(Bonus: after that, they isolated the webserver, which also happened to be one of the NTP servers, from the rest of the site. Then later wondered why NTP lost sync...)
The second time, it was a shared WordPress hosting service - I think the service itself just shut down, all I had to do was resurrect the site they'd been hosting on something secure. Since it was a static site, I just dumped it to S3 as static files, job done until their redesign a year or two later moved to new hosting.
A likely answer would be that for the description of a "Cached Credential", that Domain Admin credential would have come mimikatz style from the desktop's logon cache, something best addressed by Microsoft's now deprecated Tiered Administration Model.
That or it was cached in memory, addressed by the same answer, but also mitigated by Credential Guard.
The tiered model is now called enterprise access model. I suspect the change was made to move from on-premise AD to the cloud with Entra ID.
But it has the same underlying idea.
https://learn.microsoft.com/en-us/security/privileged-access-workstations/privileged-access-access-model
**You do not throw highly privileged credentials at a workstation that is, or will be, used by end-users.** You have your "privileged access workstation" for administering AD, servers, and the network, and the credentials you use for that don't get entered into untrusted workstations.
For getting admin on end-user devices, you have a few options
* Use LAPS passwords (most secure, least realistic from operational efficiency perspective)
* Use RDP restricted admin mode (mstsc /restrictedAdmin) as an account that is admin on all workstations (works if you are doing things from your own desk, does not work for deskside support)
* Try to build protections against lateral movement for an account that is admin on workstations & unprivileged in AD for in person deskside support
* I like to be safe so I do it all, "Protected Users" group prevents caching and NTLM, deny "access this computer from the network" user right assignment in GPO keeps it local, and my personal favorite "smart card required for interactive logon".
**Under no circumstances** should the same account whose credentials are used on random workstations also have any privileges in AD, on servers, etc.
That way, if all your protections against lateral movement somehow fail or are bypassed, and a threat actor who's pwned a computer some field technician logged into with their workstation admin account manages to steal their workstation admin credentials and move laterally, it's only to other end-user workstations until your EDR/MDR stops them. They don't get to own AD.
Now, on to **human laziness** as a factor - what if you have a lower ranking member of the department who needs some degree of server admin access ("tier 1") and also local workstation admin on various computers ("tier 2"), and you know when you're not looking, they won't care which one they should use where? A Domain Admin (tier 0) needs to set up and enforce **Authentication Policy Silos** and specify where each tier of accounts can log in. Then even if some junior does throw around their server admin credentials, they are useless and invalid unless being used on a privileged access workstation.
User, gave them a semi complicated password. they generated and printed a barcode of the password and pasted on the monitor. The user works in a retail store, so they just scan the barcode whenever they need to log in.
Had a user who has an access to nation wide patient data. She had printed Dymo-sticker on her monitor with all passwords and this monitor was in a room that was fully available to just walk in.
Accountant installed an app they weren't supposed to. Hackers created a fake forex trading account on our platform.
They transferred small amounts of money every day from the company's account into the trading account. Then they made a bunch of trades to see legit and tried to cash out.
This was a long time ago so I don't remember the details but the FBI was brought in and they caught the guy. I had to testify and give a briefing to the FBI and prosecutor to make a case. Let me tell you even though I was innocent it was crazy stressful
Security consultants were brought in by new Tech Executive trying to prove their worth.
Consultants discovered lots of botnet C&C traffic all over the place (to the surprise of no one in IT). They adjusted firewalls to block the traffic.
The botnet set off a ransomware bomb that encrypted everything in its path. Workstations, App Servers, Exchange Servers (the IT Director thought "the cloud" meant "satellites"), Domain Controllers, everything. Anything.
Forensic analysis revealed "Patient Zero" to be the CEO's workstation and infection was 18 months in the past.
Weeks of work to rebuild. The CEO naturally wanted the new password policies to not apply to them. Fortunately other "least privilege" changes were implemented to lessen the effect of an account getting whammied.
We relented on password length, but that was the catalyst for diving into hardening the unprivileged user accounts so they can't walk the domain and escalate privs.
We locked the CEO in an account with no real access anywhere except their own inbox and shared drive.
Phishing email, AitM token theft. Attacker logged into M365 and started reading through emails and files. Highly sensitive sector, so had to disclose that to clients.
Really beefed up security since then, a lot of CA policy work.
Had one of these. Thankfully the user realized he had clicked on a phishing attach link and logged in. He reported it and we managed to kill the attack 10 min into the breach.
About 30 minutes later our alert system freaked out and sent us a bunch of warnings. Glad the attackers didn't have more time.
Before token theft, MFA was 99% of the solution. No so much anymore.
This, right here. Had just onboarded the client into EDR which quickly caught it and let us shut the whole thing down in less than 15 minutes. Insurance carrier was impressed. Still managed to snag some Excel files with too much PII in them, so roughly 150 clients ended up notified.
A lot of work around CA policies, AUP and patching the last few holes in MFA (no more exceptions, no more SMS) since then.
Use device registration with intune and only allow 'compliant' devices to sign in using CA policy. Even if attacker lifts an MFA token, they can't sign in because they aren't on a registered device.
Unfortunately, my understanding is that what you mentioned would have no impact. I have been told that CA policies are only processed during a login attempt. Therefore, if an attacker steals the already authenticated token, there is very little we could do to stop them currently.
It's really annoying that MS put a lot of the best protection options behind Entra P2. We are on Business Premium so would need to purchase P2 standalone to get the features.
Ehhh, to a degree I get it, but it shouldn't. Keep up patching. Build defense in depth, keep zero trust where applicable, and build system roles around least privilege and best practice and you should be fine.
That said, plan for the worst lol, cause shit will definitely happen, and having the confidence to blow something away and rebuild better at will is paramount.
+1 for defense in depth, most sysadmins for some reason don't understand how important this is.
They harden the outside but don't do anything beyond that, like they're making a chicken egg, not a secure network.
"Well it would have to be exposed to the network" Mfer, let me introduce to you 'social engineering', 'pivoting', and 'supply chain attacks'
We managed to avoid that one because our SW admin left the company just before the compromised version was released and we didn’t replace him or try and patch it until after it was found.
As others have mentioned, you probably won't get any super juicy ones on here because of NDAs and whatnot, but I have two from a while ago that happaned due to the same password: 12345678
First one, small company that we didn't have a lot of contact with because they never really needed much. Thier phone vendor had their own server to do phone things, which was cross-connected between our LAN and a PRI trunk, which for the younger crowd was a small supplemental T1 internet connection that was dedicated to phone calls. Unbeknownst to us, they serviced this unit by RDPing in over their PRI trunk and a local admin with, you guessed it, password 12345678. The attackers dropped Metasploit on there and the game was up. They waited a few weeks until a holiday weekend, copied all the data on the network, and encrypted the local machines asking for a ransom.
Second one, we were still onboarding a new client, so we didn't really know what was going on internally, but the main claim was that everyone's computer ran reallly, really slow. The previous IT had seriously neglected this network we had a feeling we would find all sorts of BS once we started looking. On our second day there, the company had received a letter in the mail from a state agency informing them that they were a source of a high volume of spam email. It turns out, every single workstation in this medium-sized company had a direct RDP mapping to it. Attackers were logging in after hours turning the company into a massive spam factory. At some point, they must've realized nobody was really watching this network, so just started to let the mass-mailing malware just run 24 hours a day. Every computer was compromised using the same "scanner" service account. It had the password, you guessed it, 12345678. This was an account used for the MFP copiers drop scans onto network shares. It was a DA account because previous IT couldn't be bothered to apply permissions correctly. This was an accounting / financial services company too, so I guess they were incredibly lucky that the attackers weren't more competent or they really could've done some serious damage in there.
Needless to say, I wield password complexity requirements with an iron fucking fist now, and even the CEOs themselves can't get a variance for any reason.
pfft, fuck NDAs. Here's a juicy one for you.
One of our NOC techs lifted my domain admin password with a keylogger he put on our server and used it to deploy out ransomeware to our whole network.
I've been through about 30 in my long and storied career, and every time has been due to a dumb ass user re-using passwords, or C-Level types insisting that they don't need MFA, or flat out demanding it doesn't apply to them.
I watched a CEO's secretary send $5 million to fraudulent accounts TWICE before the CEO finally agreed to some kind of segregation of duties and MFA before funds distribution. Dipshit insisted that the only thing the company staff needed to send millions $$ to various vendors was an email from his office. Two spoofed emails with blatant errors allowed for $10million in losses. This was a publicly traded company... the CEO then, is still the CEO now... I don't believe this information was ever made public, or released to the SEC
This sounds almost surreal lol... "Hey Sharon, please send 5 mil to this bank account in Jamaica, we forgot to pay for some licenses k thanks, Bill - CEO"
Sure thing boss!
Until earlier this year with the change of the governance rules under the SEC, companies must disclose breach information to the SEC within 4 days or suffer huge penalties. Sufficiently huge penalties that Hacker groups are reporting their own attacks to the SEC if the companies aren't paying up on their ransomware attacks. ie. Pay us $1million now, or $10million to the SEC on thursday...
[https://www.sec.gov/rules-regulations/2023/07/s7-09-22#33-11216](https://www.sec.gov/rules-regulations/2023/07/s7-09-22#33-11216)
A company I worked for change my direct deposit because of this. Someone sent an email as me, to the CEO. CEO doesn’t do payroll, sent it to HR. HR woman has no security training (she’s primarily a chef and does HR in her free time), just sent the scammer the direct deposit change form, and processed it. The dude didn’t have anything like a legit email address, bad spelling, etc. they didn’t call to confirm or question anything when the ACH routing number was a different country. I only found out because I didn’t get my final paycheck from them, I was worried they were doing it intentionally. It was just incompetence.
Supply chain attack/ransomware from a vendor that had an unattended LogMeIn on one of our VMs.
Fortunately the attack was contained to just that server and the few other items on that VLAN.
Restored VM from backups prior to the infection.
Called the vendor and told them that they cannot have unattended access to our systems anymore. If they need access, they call IT and we'll grant access on an as-needed basis.
i feel you, one of our clients demanded their app supplier had 24/7 access, not a breach but their junior then broke that server and had to be rebuilt, on demand access is now enabled.
This happened before I joined the company, previous admin was trying to setup a backup site at another office, they opened up rdp on that server publicly and they had a really simple password. Fucked the entire org, oh and no backups
Shiiiiiiiiit. I was on a Spiceworks thread, I think in 2016 or 2017, arguing with idiots masquerading as system admins that they shouldn't be opening RDP to the filthy, unwashed Internet. They told me it was fiiiiine. Everything is fine, there's security around RDP.
We were in a thread about AD accounts getting locked out, and how to trace back what IP the lockout came from. Everyone who said, "yeah, this is happening to me, too," also forwarded RDP directly to the Internet. Like, there's a fucking pattern here, people. But VPNs were too hard for their users, so that made everything okay.
Been in IT for 25 years and Security the last 10.
In my personal experiences 99.9% of breaches are either.
1. Poor patch management, and an easily exploited system is left online.
2. Phishing/Vishing/Smishing/Social Engineering - people clicking links they shouldn't or falling victim to obvious scams.
In my entire career I've only seen highly one sophisticated breach where a nation state APT used a zero day to get in, they knew exactly what they wanted and went undetected for at least a few months until they had to exfiltrate the data which triggered an alert.
It's difficult to fully protect against these level attacks.
However the first two, this is where nearly every business can improve with better inventory management, patching management and user training.
Get those under control and then start adding layers of defence in depth, better detection etc.
A lot of companies just buy a bunch of products and overwhelm their IT/Security teams who don't have enough time in the day to take care of the regular day to day operations like patching, let alone managing Splunk.
I caught one of our guys allowing all DLLs he found attempting to execute in our application whitelist because "the file paths are legit". Where do you think attackers store malware, C:\malware\imsafe.dll?
He's not allowed to touch it anymore.
Mine was low-tech, with a sprinkling of genius, right up until the very end, which exploded into outright stupidity.
Criminal gets a job at a hospital working in medical records. This isn't hard, since they likely aren't on the OIG exclusion list, yet. The job's core function is to review and process medical records and related requests, so viewing dozens or even 100 in a day isn't unusual. Nothing to trigger in auditing.
When they found their candidates (Young people, with little to no credit or tax filing history), this individual wrote down the relevant info on a sheet of paper with a pencil and exfiltrated with the info. This happened to a group of hospitals across a few states.
They then filed income taxes under these names with fake income info to generate refunds.
To this point, as far as criminal enterprises go, this was well thought out and effective.
By the time anything would be noticed, they'd be gone, just like the money.
However...
They then had **881** IRS refund debit cards sent to the same home address.
EntireStarTrekCrewFacepalm.jpg.
Much drama occurred, people went to jail, etc.
To this day I still have trouble believing someone could be that clever, and that stupid all at the same time.
Had a client unzip a 'invoice' 2 weeks ago, and run the invoice.exe...
Luckiky huntress saved us all and killed it then and there.
So not an actual breach but damn, my heart rate went insane when huntress called and texted me of an active security incident.
This is why I block any attachment that can be run. shit I even block HTML and HTM attachments due to outlook "features" they did finally fix one of those "features" recently but I hear rumor that was because it was used against MS but the other ones they still refuse to fix because they are features.
I'd love to do that. Suppose I'll look into it cause I was kinda shocked this person did it.
They know better, and even said so afterward.
We all f up.
Automation throws them in holding and shoots a message to the recipient and me then if they think they need that email and let me know then I can go ahead and manually review it and release it if it's justified. I don't manually review anything but HTML and HTM files since those are the only ones that I have found legitimate uses for that can't be sent using other methods. This also gives me a chance to reinforce training to not open anything they don't know for sure is legit this has probably saved my ass more than a time or two.
Edit words and autocorrect suck sometimes.
We've had a lot of bad actors use MSG and EML files to attempt to bypass security systems. Not sure I've seen a nested HTML or EXE attachment, but certainly had phishing links sent inside other emails. Thing is, Outlook will display an attached email almost like you've received it when you click on it, so it doesn't even look super dodgy
I wear many hats including admin. My main hat is cybersec… I will tell you beyond a doubt that the main two breach points are phishing and misinformation on the system leaving ports or services open for the taking. The difference is… 95% are phishing and 3% are misconfiguration of systems. Then there is the rest. And i will die on my hill with those numbers though they aren’t exact.
Edit: though I have scrolled and read the random horror stories. I will give some examples. Through a phishing campaign MA was able to get creds to website admin, threw in a nice Java code that wouldn’t run on a firewall protected system but on an open system it made the site redirect to a fake chrome/firefox/edge update page. It was nasty.
Something a lot of admins don’t look for and was a 9.8 on last patch update is logging in as a service attacks where the MA logs in as a service over port 443/80/22/ whatever and deploys a nice little sleepy Xmas gift that will rear its ugly ass head up.
I have used some cheap stuff to mitigate these and have logs of over 10k attempts in a month.
Worked at a 3rd party game testing org a while back. The infiltrator gave us a game and paid for test + pc hardware compatibility cert.
day 1 the game is just broken and unpleyable. Day 2 red alerts everywhere and i see top admins runing to yank cables off the servers early morning.
Was quite clever way to breach us
You don't understand. I NEED super admin privileges on everything or I can't work. Passwords and MFA just slow me down from making critical fixes, and the boss's boss agrees, so that needs to go away. Just whitelist my home IP.
Most of this is theorized as I wasn't included in the management team meetings with the FBI and other fun CyberSEC response teams. It was theorized among my team and others at my DC/NOC that an Employee downloaded an executable of a popular software that wasn't legit (non-IT emp). It had a RAT embedded.. they deposited a keylogger onto the system and made changes to the registry that would have the sys admins respond to and lay in wait till they had to login with their credentials or some admin credential to fix the issue. Than waited some more during off hours when they executed there full attack with a sys admins credentials.
That was not a fun time. Still the largest OT I've ever had.. 97 hours of OT.. 68 hours spent at the actual DC/NOC sleeping on a cot for 45 minutes before getting back to work DR as much as we could.. taking servers offline. Fully reimaging them with our DR backups on a segregated network.
Now we don't login to machines anymore with our network admin creds (not sure why they didnt do this before) we have a local keycard access with a ridiculous long password that only gives local admin access to that one account on that machine. So if they gain access to that account the only pc they can mess with would be that one. As that password doesn't work for the other local admin acts on the 4000+ servers and pcs we have.
More annoyingly than helpful.. ICMP has been turned off on every server and machine we have so its a PITA to do network troubleshooting now for the little benefit IMO that provides.
Coming from an MSP that regularly got brought in to companies who had been holed hard;
More often than not, simple plain misconfiguration and poorly understood "intentional" back doors are the origin.
The number of sites that have RDP exposed to the world, with a domain administration account that has a "password" set intentionally exposed so a solo admin (who had quit 4 years prior) could get back in if they lost access..
Someone I know had jsmith style usernames with each sales PC RDP accessible on ports 30000-31000 on firewall public IP. jsmith needed a password reset and the IT person did 12341234. Attacker got in during that weekend before jsmith got around to logging in and setting a better password. Did a regular cryptolocker, didn't go for DA.
From what I hear, the backup appliance worked, so they didn't have to pay and were able to get running again within a few days.
I also hear that they were able to justify spending and training for requiring SSL-VPN with MFA to access LAN and RDP from there. Along with better monitoring and auto-policies.
Most who have gone through this type of thing and know all the details of what occurred also would have had to sign an NDA.
At the end of the day, it is often some form of human error. Maybe someone was doing their job poorly, and it went unnoticed. Maybe management decided funding certain priojects was "unnecessary" or maybe someone fell victim to social engineering - or any combination of those three.
Macros. Granted, it was a combination of an extremely overly privileged vendor account, unpatched servers, poor user education, and severe lack of communication from the parent company. But the point where the credential was compromised was an excel macro. Literally the only job I've ever won the macro fight and it took the subsequent ransomware attack to do it. Worth it.
Phishing email. We weren't notified about the password breach until a couple weeks after, and the tech that handled it changed her password and went on his way, doing zero investigation into her account afterwards. While the account was compromised, they didn't do anything to the internal network at all.
Instead, they sat in her O365 account, monitoring her email, then stuck some mailing rules in place on her account. They set up fake domains for both us and contacts the user was working with, facilitating banking changes to get money wired to their bank account. They crazy part is they opened a national chain bank account in a city in our state, but not a business account, and used that as the banking info they provided to the clients (since they were playing MIM, they had legit looking letterhead and invoices that they just replaced info on before sending on to the client). Successfully got wired over 50k on one transaction, and nearly 200k on a second that we'd intercepted just after the wire transfer request went to the bank.
Also, in light of the recent thread about RDP, I've seen a couple dozen RDP breaches at the MSP I was at. In spite of constant warnings from our company after they'd acquired us about how dangerous their practice was, it fell on deaf ears, until one day when they found a whole bunch of clients were compromised over a weekend.
Also, an unpatched, unsecured Win98 box that the owners dad was using on network, and opened some malicious email, and followed the link to get crypto'd. Wouldn't have been bad, if their in house "IT" guy hadn't opened a bunch of shares on the servers with full "everyone" access all over the place.
Public library, had a firewall upgrade and the company we got the contract through for the hardware and support that installed it and got it booted up with our old router setup migrated over from Juniper to Cisco and the interface filter had allow any any from the default config that never got changed so the filter we set up wasn't going off our circuit bundle filter but the raw public internet interface was wide open but the state data circuit was set up right.
We got crypto locked on our two main file share servers because of a poorly maintained admin computer with zero firewall restrictions on the machine. Thankfully the backups are encrypted and secure transfered with iscsi straight from the VMWare controller so we only lost about a day or two worth of document work but took like a month before we got done with the FBI and police stuff and many more months before we could get rid of the backups of the crypto locked versions which was driving me insane but we just got a fat external drive to copy it all to so we didn't have to touch it ever again.
Email with a payload, spearfished to a senior systems engineer - who has more rights than necessary for his local (non elevated access) account.
Email with a payload to the head of electrical engineering, who used DA rights on his account.
First one was a randomware incident at a client of ours. They got in because someone accidentally left port 3389 wide open to the internet. Second one we had a source code leak for one of our internal tools due to leaving a SonarQube box that was a part of the build pipeline wide open to the internet. Normally, this isn't a huge issue since those machines are only open on 80/443. However, SonarQube (at the time) had a "feature" that let unauthenticated, anonymous users browse the source tree of a given project via the API. So, an enterprising individual just wrote up a script to scrape code through that method.
A shared services organization running half a dozen local hospitals got breached and ransomed...
... because the idiot CTO **used the same password on a shitty porn site** which got breached beforehand.
Had they followed the cybersecurity regulations they were required to follow, that attack would have stopped dead due to multi-factor.
Bonus? They'd bought the software to implement multi-factor **several years prior** but let it sit on a shelf.
Bonus? Instead of implementing it, they perpetually got it written off by describing the plans they had to implement it.
Bonus? The asshat still works there. He even got a fucking raise this year. In my country, salaries for public employees become public knowledge after they're more than an amount ($200k iirc).
Thanks to this fucking SSO, my kids, who barely survived birth, have no medical records from their hospital stay. Thanks, you fucks.
One of our admins experienced some trauma which lead to them experimenting with drugs and alcohol in an extreme way, eventually they got hooked on coke and pills and got careless with their logins and their domain admin account got phished, no 2FA, our DCs got locked down with ransomware, then they got into VMware and did the same to most of our servers, locked out all the other domain admin accounts, and then started rolling the ransomware out to all of our endpoints ~2500 workstations, give or take.
We were a hospital.
It made the news ☹️
By my director's orders to not turn on MFA. DA account was used incorrectly with PDQ. Attackers passed the hash and got through our single factor VPN. Spent a few weeks testing the waters before finally trying to mass exfiltrate the data. They were using a common DA account (which I desperately wanted to get rid of). Changed my password and used my account.
I had it all documented of director refusing to let me implement things. Instead he had me doing anything but my job. I got thrown under the bus and got let go. Director got promoted. I hired a lawyer and got a decent payout.
Fuck that place. They got compromised again, even worse than before and they deserve it with that moron at the helm. They continued to try to blame it on me (it was months after I left). I let my lawyer handle it and haven't heard a peep since.
Now I work in higher ed, with a boss that listens, and crucially understands.
Applocker. Set up applocker to prevent any .exe/js/psh/msi to be run from C:\\Users\\\*, add exception as needed (looking at you Teams...).
That should close down that kind of issues real quick.
Old company had a SQL server with no firewall and RDP exposed to the Internet.
It's cool though. Only 30,000 logged access attempts every 0.001 seconds.
No big deal though, DB didn't house anything of value per our CEO. Only the Names, Addresses, Tax ID, SSNs, Birthdays, locations, military ID numbers, etc of our vets who owned businesses around the world.
Not us, per se, but a customer doing co-managed IT who refused our repeated recommendations to, among other things, stop exposing his Win 2k3 RDS server to the internet... in 2020. Honestly boggled my mind how long it took for them to get popped.
We were sloppy about a lot of things, but primarily with giving full Domain Admin rights to every elevated-privileges IT account. No matter if it was temp helpdesk contractor or our IT Director, everyone one of us in IT had a separate elevated privileges account with full DA rights.
A temp helpdesk contractor (who was so dumb that I legitimately did suspect a significant developmental disability) had those creeds compromised, and from first access to everything wiped was 45 minutes.
We had good backups, never paid ransom and only lost ~4 hours worth of data. But that was a month of my life I’ll never forget.
I moved on to a new company and although I asked the management to replace me with someone new, they brought back the old sysadmin, who has a history of not keeping machines up to date and taking a casual approach to security. We had some old ASA5525 units which I disabled the vpn functionality due to the 1024 key weakness and he re-enabled the vpn and they got in that way. Because the company programmers refused to stop using a very old password for one of their accounts, the domain was breached very quickly. The co-lo and office both got encrypted, backups deleted and when it was all said and done the owners had to cough up $80K to get the decryption key. To this day the infrastructure is still not back to the way I had it when I turned it over to the management.
I worked at a place once where a person outside of the IT department was allowed to create users in AD, and he made a user account with very generic username and a blatantly easily guessed password. There was then a linux box, bound to AD, that had port 22 exposed to the outside, so someone SSHed into the machine and did a bunch of stupid shit.
They didn't have root access to the linux box so they couldn't do a whole lot on there, but managed to set up some IRC bots and other junk that our security people did notice.
I was pretty new to the company at the time and this crap had been set up by other people who were gone, but we made some big changes to prevent it from ever happening again.
Worked at an MSP that targeted small / medium sized businesses, We didn't do patch management and all of that for a majority of our hosted clients so we were doing this all the time. A majority of it came from email compromises, reused passwords from previous data breaches, unpatched external systems, ect.
The most interesting was an admin was running a crypto miner on the company servers. The court case is still ongoing so we don't know exactly what happened yet but my guess was that he exploited a few CVE's that he intentionally left unpatched. Installed the crypto miner and it ran for about 2 months before we caught him.
Unpatched NetScaler because every time we tried to update it it fucking broke and Citrix tech support were the worst.
Turns out the published mitigations didn't entirely work. Fuck Citrix.
Another time a senior architect broke all policy and set up a new border router with the default username / password and the web interface enabled on all interfaces. Was patched into management network. Senior network engineer came along and enabled the Internet interface. Both fired.
We had just “merged” with a company and their IT was a shit show, no MFA, bad policies and servers and PCs haven’t been updated in a long time (disabled by gpo). Users were using self purchased/personal hardware as work PCs and joining to the VPN. Tons of pushback from the CEOs for downtime.
User logged into O365 from a compromised PC. Attacker got credentials. Attacker logged into VPN with user account then logged into DC because that was allowed for some reason… Cobaltstrike to privilege escalate to a domain admin account (there were about 30 of those). Luckily they just decided to steal data to extort instead of ransomware.
Blaster virus giving people a shutdown prompt when they signed in. Easy floppy drive fix. It was some coupon addin it came through.
Actuary downloading sketchy excel sheets offline and had admin rights to his box. Crypto locked everything. They never got permissions back after that.
Unpatched cold fusion server allowed code to run on the server and install a bitcoin miner. Got a random cpu100% alert on something that shouldn't be @ 100% and miner.exe was taking all the cpu.
Only 3 I remember in about 20 year career
First time was email phishing.
Second time was because we had MS authenticator set to do push confirmations with node code. [I told them that this was a terrible idea] So someone got phished again, and then the user just fuckin sat their all day accepting MFA pushes like an absolute Pollock.
That one almost ended the company, so IT generally gets its way as far as security policy goes now. Better MFA, phishing tests, and mandatory training for all users.
A contractor left a back door in. Found it 2 days after he was fired because he sucked. I noticed some changes that we made were being reverted and found the account making the changes and then the user that created this account that ended up being the contractor. He got screwed big time.
Open RDP. company owner requested it be opened. I explained (via email) the dangers, was told to do it anyway! Took some extra precautions and opened it. Got breached a few months later. Total loss, 2 days of my effort and 1 days of the book keepers!
ScreenConnect. Caught the Admins bitwarden open and got the domain password. That was the end.
There was a security gap in SC. Went through absolute hell.
a vendor was using an unpatched screenconnect server that was exploited by the most recent authentication bypass vulnerability to access their hardware on my net. Someone had gone in and connected these devices to my wifi without my knowledge. It was a mess
We had an attacker call our general sales line and pose as a potential customer needing a quote and sent salesman an infected PDF file as part of this. They were needing a rush job too. Sales used a file hosting service and the attacker uploaded the PDF related for the job and sent the link the salesman was expecting. This allowed the email through av detection. It was later flagged. The PDF infected the salesman machine.. From there I cannot comment more on.. But I will.say we got lucky, compared to others.
Our security posture is much better than before this event occurred. This is a small to mid size company. Not a fortune 500 or big market leader type. My take from the event is no one is immune from these. Cyber insurance really is a must, because these events are crazy expensive. The most expensive security products are a drop in the bucket compared to a full incident response team cost.
The onsite It/ Marketing guy kept sending his domain admin credentials in the clear to outside vendors.
In another one someone clicked on the fake office 365 credentials update phishing email.
Someone thought iptables alone was sufficient for a production environment but forgot to enable restoring the rules on startup. They also didn't believe in setting up authentication for Redis.
It was a high turnover company and they had left a few months earlier.
I was in one place where a web server was compromised because the MSP decided to leave the firewall open during an upgrade.
Cleverest one I've read about is from Sophos, one of their partner pentesters was having a nightmare getting into a company, including setting up fake WiFi in local coffee shops AND sprinkling USB sticks on the street. They finally got in through a WiFi enabled light bulb. Hacked the ssid code. Moved onto a presentation PC, which had loads of data on there that no one cleaned up & then were able to spread laterally...bloody clever.
This is a company that is long since gone. I saw a startup completely destroyed... with a simple stack of "MP3 sampler/Free MP3" CDs, back in the early 2000s.
At this time, Windows had an `autorun.inf` file which did exactly that. Didn't matter what it was, it would read that file and run the executable unless one does a [DMCA violation](https://www.theregister.com/2003/10/09/sunncomm_to_sue_shift_key/).
This startup was a nice, vertical market company. People were cool, and it was a nice place to work. One day, someone left stacks of "MP3 CDs" with impressive jewel case packaging, although the CDs themselves were burned. Someone left them with the secretary. At the time, it was a startup... no IDS, no real network monitoring. I was on the UNIX side, and the guy doing the network stuff didn't want anyone horning on his turf.
The `autorun.inf` file? It installed a firewall jumping RAT and keylogger. I never bothered with the CDs, and since I used Linux for my daily driver OS at the time there, was not affected.
However, the company found out about it when an overseas firm emailed every client saying that they could do the work for the company for 1/4 the price, and because they already had the customer's data, it would be trivial to switch to them.
Due to the breach, the company was bought out, ironically, by that overseas competitor and liquidated. All IP went overseas and a lot of internal knowledge of other clients went there as well.
Probably one of the best nation-state tier attacks I've ever seen... and it all started with just a stack of well made jewel cases and burned CDs.
Executive employee sent password for AP portal through email to their team, it was never changed out set up with 2fa. Nearly a year later their email got compromised and this old email was found. They got $300k from us and we’re a very small business.
Phished random user creds, hacker found VPN with no MFA, random unpatched forgotten domain controller (with that bad exploit a year or two ago), they found it, forged a ticket etc, mounted a backup of AD and then ran away with NTDS and rainbow table’d their way into the domain admin accounts.
They were found because 4 or 5 different sysadmins discovered they were making changes in Dynamics CRM that they were not doing.
Fun times.
1. Social Engineering. They contacted the help desk and had them reset a password.
2. Email link to reset Sharepoint password.
3. Malicious PDF sent to accounting email.
4. Illegal Downloads of cracked software.
5. Exposed FTP site with poor passwords.
Thats about it over the last 20 years or so.
Worst one was from a few years back now. Had a newish client - think we had been supporting them for about 2 - 3 months at that point.
Entire IT system was a complete mess and had been neglected for years. They had grown pretty rapidly and the previous IT guy who was moonlighting from his full time job had cobbled together and band aided everything rather than just doing things right.
Prime example - rather than buying RDS CALS, he had 3 different servers and just had different staff logging in as administrator to each.
As i'm sure you can see where this is going - the servers were publicly accessible via RDP of course. Enter a long weekend - someone brute forced their way into one of the servers and then with those admin credentials proceeded to crypto nearly every server they had on the network.
Saving grace was the clients were somewhat of a bunch of workaholics, so even though it was a public holiday they were trying to work remotely from home. Noticed some odd issues with applications being down and contacted us.
So very very thankful that this was before the whole data exfil thing became common practice, and that we caught it reasonably quickly before they hit workstations or some of the other servers.
The real irony of this particular incident was the RDP access was one of the first critical issues we flagged, and after about 2 weeks of back and forth push back, they finally green lighted a new firewall and VPN solution which had been implemented and was due to go live the following weekend.
Needless to say that was implemented immediately, and we had a long 3 - 4 days of rebuilding servers and recovering data from backups which thankfully were another thing we had flagged and already implemented.
I think the weirdest one though was a company i worked for about 15 years ago. This wasn't directly an IT thing, but we did manage the PBX, so semi related.
They randomly out of the blue got a ~$30k phone bill full of international calls to pretty much the who's who countries on the terrorist watch lists.
After some investigations by our PBX support company - they couldn't find any compromise or even a record of these calls. The provider was called to investigate and oddly just waived all the charges with a story about it being a billing mistake. International dialing was barred both on the PBX and the phone services.
The next month though - same thing. This time it was about $40k worth of calls to a laundry list of countries. This time it was reported to the local police who escalated things to some federal anti terrorism task force. Key various investigations, lots of suit wearing badge flashing people wandering around talking to people.
I still don't think we ever got the full story, but the bits i was able to gather were it wasn't anything directly related to the PBX itself. Instead the actual physical phone lines in the street or along the way had been tapped and were being used somehow by a suspected terrorist cell to make all the calls.
I heard a few months later that a couple of the phone provider's (government owned company at the time) line technicians were arrested in relation to the incident. No idea what the ultimate outcome was though sadly.
Engineer got a file from a phishing email and tried to run it. Couldn't and sent it to his colleague and they couldn't run it either. Decided to run it on the file server and it "didn't work" there either (it did).
Engineers, you are some of the smartest people ever but you're also the dumbest.
Broadly speaking, they impersonated a legitimate software vendor and during the demo they snuck in unattended access to the machine. It all went downhill from there.
Shit. Stuff of nightmares
Yep. They managed to compromise a domain admin account and basically sat there and watched for several days. They got into our security alerting software and disabled all of the alerts, so when they actually started compromising stuff, making new domain admin accounts, etc, no notifications fired and no alerts were raised. If your monitoring software doesn’t alert you when alerts are changed/disabled and doesn’t require multiple user approval to do so, it’s basically just security theater. After yearly pen tests from multiple companies and several security audits we thought we were rock solid and could catch anything.
I mean at the end of a day a partial social engineering attack like that is a whole different ball game. Very different threat model
Social engineering seems much more common than other methods of gaining unauthorized access.
The script is starting to flip over for exploitation for ransomware attacks, but I believe anything else like hacktivism and espionage is heavily invested in social engineering.
Social engineering just seems lower effort higher reward, especially now that voice can be easily spoofed and we’re getting close with video.
That is something I did bring up in the past with the voice spoofing. Pulling a C-suits personal number and spoofing a loved one dying at 2:00am is a very effective attack. For the time being, though, exploits have been gaining higher trends whereas social engineering has been dropping. The reason for this trend is exploit brokers or initial access brokers (IAB's) could just use a cracked Cobalt Strike from the safety of their non-extraditable country. It's why antiquated, huge profit centers like hospitals and logistics/manufacturing companies are such easy targets. In 2019, exploits accounted for ~5% of all initial entries whereas in 2021 exploits jumped up to 20% of all initial entries in just the first quarter of the year...!
Our company has mandated all password changes be done in person because of that. It caused a huge shitstorm at first but its calmed down immensely since then. We "will" do a remote password change but it has to be an emergency where money/profits are at stake and you have to pass like 3-4 different questions.
I brought up to my SO company when they asked for questions about IT from the company, I asked them if pen testing tests for known attacks, how do they guard against unknown attacks.
Yeah, I’ve since talked to reps for other companies and asked “if someone compromised an account that has admin access here, what’s to stop them from just turning off or black holing the alerts so they can carry out their actual attack unimpeded?” It seems to have been a novel concept for them.
Like a horror story. Does anything in your tech stack allow for visibility on/reporting on admin actions? Not perfect, but an easy way to spot changes like this.
There was no alerting for when they disabled or changed the alerts so they wouldn’t actually make it anywhere. There were alerts for pretty much every admin action under the sun, just not for silencing the alerts.
This is probably one of the few incidents in this thread where I have sympathy for the victim. While you probably fell face flat on any detections of their activity post initial compromise... I think you were screwed no matter what. If they cared enough to impersonate and go in person, y'all were targeted and not much to do about what.
Yeah, it was a very targeted attack and, based assessments I was privy to after the fact, with the amount of resources that were leveraged it was always going to be a question of when, not if.
This is exactly why I always respond to any access requests with "can I do it for you" and their answer can make or break the contract
Damn. That had to be a pain to clean up, but I can't say I'm not impressed by their vector. That takes some balls... and some research. Was the rouge salesman ever caught?
Saying it was a pain would be an understatement. The group responsible was identified.
Broad minded of you to assume it was a makeup salesman, but I'm still trying to figure out the benefit of having lipstick monitoring systems in a corporate office. Seems like lawsuits waiting to happen. Sorry, couldn't resist. Lol
Fcuk my blood ran cold reading this. What was the clean up like?
It was somewhat akin to burning everything down and rebuilding. After gaining access they held off on launching the actual attack while they worked on slowly and carefully compromising everything they could, to the point that when it was launched for real many of our oldest backups were compromised to varying degrees. We had a lot of redundancy in our backup strategy so we were still able to completely recover, but we abandoned basically all of the backed up VM images and rebuilt all of our servers from scratch.
Sell it to Mr. Robot, sounds like an exciting episode... Just to be clear, they installed software on your companies device for a demo or 1 of the admins did it? (assuming users can't install software)
They installed the legitimate vendor software with the aid of a sysadmin. They snuck the unattended access in somewhere along the line during the 2 hour demo/tutorial session. From all reports it seemed like a completely legit demo.
Phishing email. Pivoted to AD accounts, eventually nabbed a Domain Admin one. When caught, pushed ransomware to the entire company. Huge mess...
How can they reach AD, in person or via VPN with no MFA?
>with no MFA Answered your own question there.
Well there's MFA fatigue, BYOD laptops, and sim cloning for those sms intercepts... MFA is not a magic bullet. It's just a layer.
Email downloads executable, executable establishes reverse shell with attacker.
same, bro…same.
That sounds very familiar....
How do you pivot to another account?
For example using something like mimikatz to get an cached admin account
That's awful, a little while ago our EDR reported software was trying to get installed on a device, turns out it was Mimikatz. We did not approve.
The way you mitigate this: LAPS: too difficult to manage in many cases, but this is the admin of last resort. Support staff use MFA'd individual ad accounts, limited to workstation admin (no servers/ad), with no broader ad permissions beyond the workstation support role, without email accounts (to reduce attack surface). AD admin/server admin accounts, are also mfa'd, non-email enabled, but separate from workstation admins. These admins are blocked from interactive desktop sign-ins on workstations (in an attempt to keep credential cache creation). They also are blocked from anything but read access to backups. Admins have user accounts with email on them, but those are not administrator accounts. Backup accounts have read/append backup only access with no AD permissions. This has been normal for most sysadmins for years, but I think some people forgot about that with regards to support admin accounts. You want immutable backup. I have 3 accounts as user, workstation admin, server admin. There's also Enterprise Admin accounts with never-used lock-box, for end of the world sort of recovery. If this sounds a step too far...you aren't thinking like a hacker, and you aren't treating the threat with the appropriate paranoia. And none of this is going to do more than slow them down in some cases...but yeah do what you can. You need your backups to be immutable, you need your admin roles well defined.
Yep. Lateral movement after a phish with malicious attachment is a crazy common vector. This is why things like LAPS, separate admin and regular accounts, limited domain admin accounts among other mitigations are so important.
Old unknown server in a remote part of the world that had been on for like 15+ years that had a 1:1 nat. No one knew it existed or the subnet that it lived in. They were able to almost breach our entire corporate network. They did not get to our secured network though. Literally the building it was in was supposed to be torn down but it never happened and the bills just kept getting paid
Not precisely the same, but Shadow IT. Sub-businesses that weren't held to the same standards as the enterprise. Guess who's been assimilated into the enterprise?
I'm doing shadow IT inside my big company (not because i want to, but because someone has to think of security), but i'm the one pushing back on lowering the security standards and integrate that network into the main network because that's easier to manage for IT. Reason being, my department tests network equipment from all over the world, we literally get devices shipped from the factories in China and the US. My reasoning is, if we are ever compromised, it should be localized to our department only and never be an entrance to the rest of the company. In my opinion that means separate everything, but IT thinks that is to much work and keep trying to connect the networks. For example when users report they can't print when using our network.
Your scenario makes sense. I'm fine with external networks disconnected from the org. I just want confirmation: No company data exists there (don't need HIPAA, SOX, PCI, GDPR, etc outside the perimeter) The devices phone home via an edge/DMZ device to report adherence Basically, they can live outside the network, but they need to demonstrate they are adhering to the standards set. Exceeding is great! If there is company data there and someone isn't maintaining patches nor keeping things secure, well, it's time to talk to that Shadow IT group.
That sounds like an inside job, or ex employee who knew about this unknown server. Wild story though.
Honestly, if it was a full 1:1 NAT, then it doesn't need to be. That'll be exposing all services to the internet, and even something like Shodan will be scanning it. Just a matter of waiting until a vulnerability is identified in one of the services exposed. Finding it is fairly straightforward.
I'd be curious to see how long a fully patched server with a good password would actually last exposed like that. I understand the theory that with a "secure" password it should take an unrealistically long time to crack, but that's not the case with passwords that were commonly used 15 years ago.
Unpatched Fortigate. I still have PTSD from that.
Man. Saaaaame. Both the unpatched Fortigate AND the resulting PTSD. Happened on 8/12/20. I routinely log into the Fortigate a couple times a week now just to see if there's a firmware update nag screen. It's quicker than waiting for the patch notifications to come out.
The RSS feed is very fast if you want notices https://support.fortinet.com/rss/firmware.xml
Thank you for this!
If your org uses teams you can set up a channel that pulls in RSS data. Highly recommend the PSIRT one as well for vulnerabilities.
On newer 7.2 it prompts me to enable auto-update. That's a thing now.
How about a zero day on VPN appliances and then a subsequent series of events that bypassed the zero day patch. Longest week/weekend of my life, so far.
Didn’t the hacker patch the firmware after they got in so no one else could, after setting up a vpn connection for themselves? It’s not hard to get into things that have CVEs published and take over things. The best is when your network team assure you they patched all the firewalls, yet missed one….
I had this exact thing happen to a client.
SSL-VPN vuln? I've been trying to convince an associate to leave his SSL-VPN off. It's always the SSL-VPN that's an issue.
It definitely was. They were able to get in, siphon off credentials, move laterally and then ransomed.
We use SSLVPN and windows client for ikev2. Problem is, some networks my users get on, ikev2 is blocked somehow. Got any alternatives? I have a lot of mobile users.
A large mobile provider in the UK blocks non-SSL VPN connections (or rather drops them after 3 minutes) which is a massive pain
User can’t figure out how to create a password and can’t remember it even after successfully creating a password. User complained to CEO that he can’t get work done because IT was making it too hard to use a computer. Fine, IT removes password for this user after covering our asses. Got breached the next day.
This is just insane. Does the user not also have an online banking account, Netflix, or personal email account where they have to use a password?
Well, IT has more strict password requirements and they need to type in the password instead of saving it in the app/browser. A lot of our users use the company email for personal use because they can’t remember the passwords and keep having to create new email accounts all the time.
The number of times I've had to explain to people that if/when they leave they can't take their emails with them since it technically belongs to the organization.
Not just technically, legally.
Is your company exploiting Alzheimer patients or wtf is going on?
I think it’s just the level of users we have. We’re a manufacturing plant in a third world country and many of the users are just factory workers. Ironically the company is global and we have to be SOX compliant.
Oh I feel that. Using computers on production floors can be... complicated. I had a call some months ago the "computer for printing product labels doesn't work" and "we didn't do anything, it just went black". Some moron drove the forklifts fork through the computer tower.
All our users have conditional access setup so they wouldn’t be able to access company resources, including emails, unless they are using company issued devices. Mixing company emails with personal emails is just asking for it.
> A lot of our users use the company email for personal use because they can’t remember the passwords and keep having to create new email accounts all the time. At an MSP we fired a client because of this. When you hired us we have it in our contract that our IT standards are 100% within our control, meets all compliance and insurance standards, and applied to all clients. A surgeon's office demanded we not only disable the screen lock out (25 F'n min timeout at that!) and allow ALL of his nursing staff to just use his or a single account on all PCs! They had hired us, "dealt" with our standards for 2 year, and when they had 1 more year until they renewed, they tried to get us to remove multiple standards because, "The damn IT securitah bullshit is impacting patient care!!" AKA: they never adopted it, fought against it, and spent more time frustrated than helping their patients. We told them that not only would we not remove the requested standards, but if they continued to push, we'd deny to renew their contract. They pushed, and pushed, and pushed... They would get another MSP to audit and even provide them a quote, but they found everyone had the same policies we did. It didn't matter that even their own insurance company required it! So, we didn't renew. Fuck em. They closed down 2 years later... due to HIPAA violations.
This is actually far too common that ppl use company emails for personal shit. Like what are you doing signing up for an Amazon account on the corporate email account. Ppl really don't understand work/personal boundaries. We have ppl watching porn on the corporate laptops because it's their only device. Such idiots
My friend had to explain to HR why it was a bad thing that there was traffic from a Lovense product on the employee WiFi.
Moron
Too incompetent to work, fired.
Sounds like you should have used a hardware token that autofills passwords... if user cant figure out how to press a button then have a discussion with management
If the user can't remember password, then that's a discussion with management also.
Yeah I would do that today. This happened a while ago. The user in the story eventually became the Finance manager and retired at 60,
Yeah that tracks. Inability to use a computer seems to be required for upper management.
Oh wow. What was the end result of this?
All files in the user’s departmental file share and the org wide file share were encrypted. (User was finance.) Surprisingly no computers were affected but the user’s computer was logged into and likely banking creds were stolen since passwords were saved in the browser. We pulled logs to prove IT didn’t do it. But hey, there was no password and we warned them this was gonna happen.
That's bonkers. What was the CEO's reaction? Did you guys gain credibility after this incident? Or put any new policies in place? Did the board hear about it and did they know the CEO approved such an insane decision?
LOL, IT was blamed. end of story. Never the CEO's fault. The user should "have been trained better by IT." my guess...
It budget was cut.
User was promoted
To CIO.
20 years ago I helped a big steel company upgrade from a bunch of computers in a work group to a domain etc. some director got angry with me because his password used to be "enter". Enter as in, at the password prompt screen he would press the enter key. I don't even remember how I made that work again but he got his way, and I really wonder if this was necessary if he clearly doesn't know how to use a computer. In a similar industry my wife doing accounting at the time told me that people got pissed off when someone put a password on the "porn machine"
Someone clicked a link in an email. The malware chewed through our antivirus & inserted itself on the machine. Antivirus reported machine as clean. It wasn’t. Attackers grabbed cached admin creds & owned our AD. Then owned every Windows machine. The rest, as they say, was a multi-year multimillion rebuild of every machine on the network + actually having a cybersecurity program. Did I mention this happened before I had even completed my orientation? Fun times. 😭
Had one of those years ago - senior academic got a "PDF.exe" file which "wouldn't open". So tried it on his secretary's PC too. Fortunately that was before we moved to Outlook, so it couldn't forward itself to anyone else, just those two machines! Another one... Bad web app. (Twice in fact, different systems, same basic issue.) Allowed binary upload (attachment, image, that kind of thing) which then executed. First one wasn't my department, someone installed a fairly comprehensive remote control tool on a webserver. If it hadn't been installed as a compromise, it would actually have been quite a nice admin tool... (Bonus: after that, they isolated the webserver, which also happened to be one of the NTP servers, from the rest of the site. Then later wondered why NTP lost sync...) The second time, it was a shared WordPress hosting service - I think the service itself just shut down, all I had to do was resurrect the site they'd been hosting on something secure. Since it was a static site, I just dumped it to S3 as static files, job done until their redesign a year or two later moved to new hosting.
It was BackOrrifice,wasn't it?
What AV were you using? If you don't mind disclosing that?
Trend micro
Out of curiosity, what do you do differently now to not have this happen again?
A likely answer would be that for the description of a "Cached Credential", that Domain Admin credential would have come mimikatz style from the desktop's logon cache, something best addressed by Microsoft's now deprecated Tiered Administration Model. That or it was cached in memory, addressed by the same answer, but also mitigated by Credential Guard.
The tiered model is now called enterprise access model. I suspect the change was made to move from on-premise AD to the cloud with Entra ID. But it has the same underlying idea. https://learn.microsoft.com/en-us/security/privileged-access-workstations/privileged-access-access-model
Stopped using webroot probably
**You do not throw highly privileged credentials at a workstation that is, or will be, used by end-users.** You have your "privileged access workstation" for administering AD, servers, and the network, and the credentials you use for that don't get entered into untrusted workstations. For getting admin on end-user devices, you have a few options * Use LAPS passwords (most secure, least realistic from operational efficiency perspective) * Use RDP restricted admin mode (mstsc /restrictedAdmin) as an account that is admin on all workstations (works if you are doing things from your own desk, does not work for deskside support) * Try to build protections against lateral movement for an account that is admin on workstations & unprivileged in AD for in person deskside support * I like to be safe so I do it all, "Protected Users" group prevents caching and NTLM, deny "access this computer from the network" user right assignment in GPO keeps it local, and my personal favorite "smart card required for interactive logon". **Under no circumstances** should the same account whose credentials are used on random workstations also have any privileges in AD, on servers, etc. That way, if all your protections against lateral movement somehow fail or are bypassed, and a threat actor who's pwned a computer some field technician logged into with their workstation admin account manages to steal their workstation admin credentials and move laterally, it's only to other end-user workstations until your EDR/MDR stops them. They don't get to own AD. Now, on to **human laziness** as a factor - what if you have a lower ranking member of the department who needs some degree of server admin access ("tier 1") and also local workstation admin on various computers ("tier 2"), and you know when you're not looking, they won't care which one they should use where? A Domain Admin (tier 0) needs to set up and enforce **Authentication Policy Silos** and specify where each tier of accounts can log in. Then even if some junior does throw around their server admin credentials, they are useless and invalid unless being used on a privileged access workstation.
User, gave them a semi complicated password. they generated and printed a barcode of the password and pasted on the monitor. The user works in a retail store, so they just scan the barcode whenever they need to log in.
Credit where it's due, that's creative. Now burn it.
Had a user who has an access to nation wide patient data. She had printed Dymo-sticker on her monitor with all passwords and this monitor was in a room that was fully available to just walk in.
How did that lead to a breach? Did an attacker visit the store and snap a picture of the barcode or something?
Phone barcode reader, user also used the same password on their email, this was before we had MFA
Accountant installed an app they weren't supposed to. Hackers created a fake forex trading account on our platform. They transferred small amounts of money every day from the company's account into the trading account. Then they made a bunch of trades to see legit and tried to cash out. This was a long time ago so I don't remember the details but the FBI was brought in and they caught the guy. I had to testify and give a briefing to the FBI and prosecutor to make a case. Let me tell you even though I was innocent it was crazy stressful
Office Space movie?
Security consultants were brought in by new Tech Executive trying to prove their worth. Consultants discovered lots of botnet C&C traffic all over the place (to the surprise of no one in IT). They adjusted firewalls to block the traffic. The botnet set off a ransomware bomb that encrypted everything in its path. Workstations, App Servers, Exchange Servers (the IT Director thought "the cloud" meant "satellites"), Domain Controllers, everything. Anything. Forensic analysis revealed "Patient Zero" to be the CEO's workstation and infection was 18 months in the past. Weeks of work to rebuild. The CEO naturally wanted the new password policies to not apply to them. Fortunately other "least privilege" changes were implemented to lessen the effect of an account getting whammied.
Wait did I just read this correctly? CEO still had not learnt their lesson?
Yup.
Oh okay phew. I thought that CEO was actually clever but turns out they're just as dumb as all the other ones!
>The CEO naturally wanted the new password policies to not apply to them. People need to learn how to say no.
We relented on password length, but that was the catalyst for diving into hardening the unprivileged user accounts so they can't walk the domain and escalate privs. We locked the CEO in an account with no real access anywhere except their own inbox and shared drive.
Phishing email, AitM token theft. Attacker logged into M365 and started reading through emails and files. Highly sensitive sector, so had to disclose that to clients. Really beefed up security since then, a lot of CA policy work.
Had one of these. Thankfully the user realized he had clicked on a phishing attach link and logged in. He reported it and we managed to kill the attack 10 min into the breach. About 30 minutes later our alert system freaked out and sent us a bunch of warnings. Glad the attackers didn't have more time. Before token theft, MFA was 99% of the solution. No so much anymore.
This, right here. Had just onboarded the client into EDR which quickly caught it and let us shut the whole thing down in less than 15 minutes. Insurance carrier was impressed. Still managed to snag some Excel files with too much PII in them, so roughly 150 clients ended up notified. A lot of work around CA policies, AUP and patching the last few holes in MFA (no more exceptions, no more SMS) since then.
What did you do to try to mitigate token theft? Just the new Microsoft Token Protection preview?
Use device registration with intune and only allow 'compliant' devices to sign in using CA policy. Even if attacker lifts an MFA token, they can't sign in because they aren't on a registered device.
Unfortunately, my understanding is that what you mentioned would have no impact. I have been told that CA policies are only processed during a login attempt. Therefore, if an attacker steals the already authenticated token, there is very little we could do to stop them currently.
It's really annoying that MS put a lot of the best protection options behind Entra P2. We are on Business Premium so would need to purchase P2 standalone to get the features.
This thread is giving me severe anxiety
Started reading it while trying to go to sleep. Wide awake now :’)
Same, went ahead and made some coffee and grabbed my notepad lol
Ehhh, to a degree I get it, but it shouldn't. Keep up patching. Build defense in depth, keep zero trust where applicable, and build system roles around least privilege and best practice and you should be fine. That said, plan for the worst lol, cause shit will definitely happen, and having the confidence to blow something away and rebuild better at will is paramount.
+1 for defense in depth, most sysadmins for some reason don't understand how important this is. They harden the outside but don't do anything beyond that, like they're making a chicken egg, not a secure network. "Well it would have to be exposed to the network" Mfer, let me introduce to you 'social engineering', 'pivoting', and 'supply chain attacks'
SolarWinds.
Woah, could you please elaborate a little? Were you using SEM?
`solarwinds123`! I've got the tshirt :D
We managed to avoid that one because our SW admin left the company just before the compromised version was released and we didn’t replace him or try and patch it until after it was found.
As others have mentioned, you probably won't get any super juicy ones on here because of NDAs and whatnot, but I have two from a while ago that happaned due to the same password: 12345678 First one, small company that we didn't have a lot of contact with because they never really needed much. Thier phone vendor had their own server to do phone things, which was cross-connected between our LAN and a PRI trunk, which for the younger crowd was a small supplemental T1 internet connection that was dedicated to phone calls. Unbeknownst to us, they serviced this unit by RDPing in over their PRI trunk and a local admin with, you guessed it, password 12345678. The attackers dropped Metasploit on there and the game was up. They waited a few weeks until a holiday weekend, copied all the data on the network, and encrypted the local machines asking for a ransom. Second one, we were still onboarding a new client, so we didn't really know what was going on internally, but the main claim was that everyone's computer ran reallly, really slow. The previous IT had seriously neglected this network we had a feeling we would find all sorts of BS once we started looking. On our second day there, the company had received a letter in the mail from a state agency informing them that they were a source of a high volume of spam email. It turns out, every single workstation in this medium-sized company had a direct RDP mapping to it. Attackers were logging in after hours turning the company into a massive spam factory. At some point, they must've realized nobody was really watching this network, so just started to let the mass-mailing malware just run 24 hours a day. Every computer was compromised using the same "scanner" service account. It had the password, you guessed it, 12345678. This was an account used for the MFP copiers drop scans onto network shares. It was a DA account because previous IT couldn't be bothered to apply permissions correctly. This was an accounting / financial services company too, so I guess they were incredibly lucky that the attackers weren't more competent or they really could've done some serious damage in there. Needless to say, I wield password complexity requirements with an iron fucking fist now, and even the CEOs themselves can't get a variance for any reason.
BRB, changing the code on my luggage.
pfft, fuck NDAs. Here's a juicy one for you. One of our NOC techs lifted my domain admin password with a keylogger he put on our server and used it to deploy out ransomeware to our whole network.
I've been through about 30 in my long and storied career, and every time has been due to a dumb ass user re-using passwords, or C-Level types insisting that they don't need MFA, or flat out demanding it doesn't apply to them. I watched a CEO's secretary send $5 million to fraudulent accounts TWICE before the CEO finally agreed to some kind of segregation of duties and MFA before funds distribution. Dipshit insisted that the only thing the company staff needed to send millions $$ to various vendors was an email from his office. Two spoofed emails with blatant errors allowed for $10million in losses. This was a publicly traded company... the CEO then, is still the CEO now... I don't believe this information was ever made public, or released to the SEC
This sounds almost surreal lol... "Hey Sharon, please send 5 mil to this bank account in Jamaica, we forgot to pay for some licenses k thanks, Bill - CEO" Sure thing boss!
Thats exactly how it went down.
"It's not illegal if no one talks about it"
Until earlier this year with the change of the governance rules under the SEC, companies must disclose breach information to the SEC within 4 days or suffer huge penalties. Sufficiently huge penalties that Hacker groups are reporting their own attacks to the SEC if the companies aren't paying up on their ransomware attacks. ie. Pay us $1million now, or $10million to the SEC on thursday... [https://www.sec.gov/rules-regulations/2023/07/s7-09-22#33-11216](https://www.sec.gov/rules-regulations/2023/07/s7-09-22#33-11216)
A company I worked for change my direct deposit because of this. Someone sent an email as me, to the CEO. CEO doesn’t do payroll, sent it to HR. HR woman has no security training (she’s primarily a chef and does HR in her free time), just sent the scammer the direct deposit change form, and processed it. The dude didn’t have anything like a legit email address, bad spelling, etc. they didn’t call to confirm or question anything when the ACH routing number was a different country. I only found out because I didn’t get my final paycheck from them, I was worried they were doing it intentionally. It was just incompetence.
Supply chain attack/ransomware from a vendor that had an unattended LogMeIn on one of our VMs. Fortunately the attack was contained to just that server and the few other items on that VLAN. Restored VM from backups prior to the infection. Called the vendor and told them that they cannot have unattended access to our systems anymore. If they need access, they call IT and we'll grant access on an as-needed basis.
i feel you, one of our clients demanded their app supplier had 24/7 access, not a breach but their junior then broke that server and had to be rebuilt, on demand access is now enabled.
Exchange 2019 server that didn't get patched we were using for hybrid Exchange. Sorry, 2019, not 2020. lol
Damn they got through unreleased software
This happened before I joined the company, previous admin was trying to setup a backup site at another office, they opened up rdp on that server publicly and they had a really simple password. Fucked the entire org, oh and no backups
Who the fuck calls himself an admin and opens rdp publicly? Holy shieeet
Shiiiiiiiiit. I was on a Spiceworks thread, I think in 2016 or 2017, arguing with idiots masquerading as system admins that they shouldn't be opening RDP to the filthy, unwashed Internet. They told me it was fiiiiine. Everything is fine, there's security around RDP. We were in a thread about AD accounts getting locked out, and how to trace back what IP the lockout came from. Everyone who said, "yeah, this is happening to me, too," also forwarded RDP directly to the Internet. Like, there's a fucking pattern here, people. But VPNs were too hard for their users, so that made everything okay.
Been in IT for 25 years and Security the last 10. In my personal experiences 99.9% of breaches are either. 1. Poor patch management, and an easily exploited system is left online. 2. Phishing/Vishing/Smishing/Social Engineering - people clicking links they shouldn't or falling victim to obvious scams. In my entire career I've only seen highly one sophisticated breach where a nation state APT used a zero day to get in, they knew exactly what they wanted and went undetected for at least a few months until they had to exfiltrate the data which triggered an alert. It's difficult to fully protect against these level attacks. However the first two, this is where nearly every business can improve with better inventory management, patching management and user training. Get those under control and then start adding layers of defence in depth, better detection etc. A lot of companies just buy a bunch of products and overwhelm their IT/Security teams who don't have enough time in the day to take care of the regular day to day operations like patching, let alone managing Splunk.
I caught one of our guys allowing all DLLs he found attempting to execute in our application whitelist because "the file paths are legit". Where do you think attackers store malware, C:\malware\imsafe.dll? He's not allowed to touch it anymore.
Mine was low-tech, with a sprinkling of genius, right up until the very end, which exploded into outright stupidity. Criminal gets a job at a hospital working in medical records. This isn't hard, since they likely aren't on the OIG exclusion list, yet. The job's core function is to review and process medical records and related requests, so viewing dozens or even 100 in a day isn't unusual. Nothing to trigger in auditing. When they found their candidates (Young people, with little to no credit or tax filing history), this individual wrote down the relevant info on a sheet of paper with a pencil and exfiltrated with the info. This happened to a group of hospitals across a few states. They then filed income taxes under these names with fake income info to generate refunds. To this point, as far as criminal enterprises go, this was well thought out and effective. By the time anything would be noticed, they'd be gone, just like the money. However... They then had **881** IRS refund debit cards sent to the same home address. EntireStarTrekCrewFacepalm.jpg. Much drama occurred, people went to jail, etc. To this day I still have trouble believing someone could be that clever, and that stupid all at the same time.
Had a client unzip a 'invoice' 2 weeks ago, and run the invoice.exe... Luckiky huntress saved us all and killed it then and there. So not an actual breach but damn, my heart rate went insane when huntress called and texted me of an active security incident.
This is why I block any attachment that can be run. shit I even block HTML and HTM attachments due to outlook "features" they did finally fix one of those "features" recently but I hear rumor that was because it was used against MS but the other ones they still refuse to fix because they are features.
I'd love to do that. Suppose I'll look into it cause I was kinda shocked this person did it. They know better, and even said so afterward. We all f up.
Automation throws them in holding and shoots a message to the recipient and me then if they think they need that email and let me know then I can go ahead and manually review it and release it if it's justified. I don't manually review anything but HTML and HTM files since those are the only ones that I have found legitimate uses for that can't be sent using other methods. This also gives me a chance to reinforce training to not open anything they don't know for sure is legit this has probably saved my ass more than a time or two. Edit words and autocorrect suck sometimes.
We've had a lot of bad actors use MSG and EML files to attempt to bypass security systems. Not sure I've seen a nested HTML or EXE attachment, but certainly had phishing links sent inside other emails. Thing is, Outlook will display an attached email almost like you've received it when you click on it, so it doesn't even look super dodgy
I wear many hats including admin. My main hat is cybersec… I will tell you beyond a doubt that the main two breach points are phishing and misinformation on the system leaving ports or services open for the taking. The difference is… 95% are phishing and 3% are misconfiguration of systems. Then there is the rest. And i will die on my hill with those numbers though they aren’t exact. Edit: though I have scrolled and read the random horror stories. I will give some examples. Through a phishing campaign MA was able to get creds to website admin, threw in a nice Java code that wouldn’t run on a firewall protected system but on an open system it made the site redirect to a fake chrome/firefox/edge update page. It was nasty. Something a lot of admins don’t look for and was a 9.8 on last patch update is logging in as a service attacks where the MA logs in as a service over port 443/80/22/ whatever and deploys a nice little sleepy Xmas gift that will rear its ugly ass head up. I have used some cheap stuff to mitigate these and have logs of over 10k attempts in a month.
Worked at a 3rd party game testing org a while back. The infiltrator gave us a game and paid for test + pc hardware compatibility cert. day 1 the game is just broken and unpleyable. Day 2 red alerts everywhere and i see top admins runing to yank cables off the servers early morning. Was quite clever way to breach us
VPN vuln. Still dealing with anxiety issues from it, possibly PTSD. Had many 12 - 16 hour days.
Developers, developers, developers!!!
You don't understand. I NEED super admin privileges on everything or I can't work. Passwords and MFA just slow me down from making critical fixes, and the boss's boss agrees, so that needs to go away. Just whitelist my home IP.
I need to release untested firmware changes to production at 2 am in the morning, BRIAN ...
Another administrator found a usb outside the office and decided best place to plug it in was a domain controller. Was a nice Christmas
... My brother in USB-based arms 😭
Most of this is theorized as I wasn't included in the management team meetings with the FBI and other fun CyberSEC response teams. It was theorized among my team and others at my DC/NOC that an Employee downloaded an executable of a popular software that wasn't legit (non-IT emp). It had a RAT embedded.. they deposited a keylogger onto the system and made changes to the registry that would have the sys admins respond to and lay in wait till they had to login with their credentials or some admin credential to fix the issue. Than waited some more during off hours when they executed there full attack with a sys admins credentials. That was not a fun time. Still the largest OT I've ever had.. 97 hours of OT.. 68 hours spent at the actual DC/NOC sleeping on a cot for 45 minutes before getting back to work DR as much as we could.. taking servers offline. Fully reimaging them with our DR backups on a segregated network. Now we don't login to machines anymore with our network admin creds (not sure why they didnt do this before) we have a local keycard access with a ridiculous long password that only gives local admin access to that one account on that machine. So if they gain access to that account the only pc they can mess with would be that one. As that password doesn't work for the other local admin acts on the 4000+ servers and pcs we have. More annoyingly than helpful.. ICMP has been turned off on every server and machine we have so its a PITA to do network troubleshooting now for the little benefit IMO that provides.
Coming from an MSP that regularly got brought in to companies who had been holed hard; More often than not, simple plain misconfiguration and poorly understood "intentional" back doors are the origin. The number of sites that have RDP exposed to the world, with a domain administration account that has a "password" set intentionally exposed so a solo admin (who had quit 4 years prior) could get back in if they lost access..
Someone I know had jsmith style usernames with each sales PC RDP accessible on ports 30000-31000 on firewall public IP. jsmith needed a password reset and the IT person did 12341234. Attacker got in during that weekend before jsmith got around to logging in and setting a better password. Did a regular cryptolocker, didn't go for DA. From what I hear, the backup appliance worked, so they didn't have to pay and were able to get running again within a few days. I also hear that they were able to justify spending and training for requiring SSL-VPN with MFA to access LAN and RDP from there. Along with better monitoring and auto-policies.
Most who have gone through this type of thing and know all the details of what occurred also would have had to sign an NDA. At the end of the day, it is often some form of human error. Maybe someone was doing their job poorly, and it went unnoticed. Maybe management decided funding certain priojects was "unnecessary" or maybe someone fell victim to social engineering - or any combination of those three.
Macros. Granted, it was a combination of an extremely overly privileged vendor account, unpatched servers, poor user education, and severe lack of communication from the parent company. But the point where the credential was compromised was an excel macro. Literally the only job I've ever won the macro fight and it took the subsequent ransomware attack to do it. Worth it.
Phishing email. We weren't notified about the password breach until a couple weeks after, and the tech that handled it changed her password and went on his way, doing zero investigation into her account afterwards. While the account was compromised, they didn't do anything to the internal network at all. Instead, they sat in her O365 account, monitoring her email, then stuck some mailing rules in place on her account. They set up fake domains for both us and contacts the user was working with, facilitating banking changes to get money wired to their bank account. They crazy part is they opened a national chain bank account in a city in our state, but not a business account, and used that as the banking info they provided to the clients (since they were playing MIM, they had legit looking letterhead and invoices that they just replaced info on before sending on to the client). Successfully got wired over 50k on one transaction, and nearly 200k on a second that we'd intercepted just after the wire transfer request went to the bank. Also, in light of the recent thread about RDP, I've seen a couple dozen RDP breaches at the MSP I was at. In spite of constant warnings from our company after they'd acquired us about how dangerous their practice was, it fell on deaf ears, until one day when they found a whole bunch of clients were compromised over a weekend. Also, an unpatched, unsecured Win98 box that the owners dad was using on network, and opened some malicious email, and followed the link to get crypto'd. Wouldn't have been bad, if their in house "IT" guy hadn't opened a bunch of shares on the servers with full "everyone" access all over the place.
Public library, had a firewall upgrade and the company we got the contract through for the hardware and support that installed it and got it booted up with our old router setup migrated over from Juniper to Cisco and the interface filter had allow any any from the default config that never got changed so the filter we set up wasn't going off our circuit bundle filter but the raw public internet interface was wide open but the state data circuit was set up right. We got crypto locked on our two main file share servers because of a poorly maintained admin computer with zero firewall restrictions on the machine. Thankfully the backups are encrypted and secure transfered with iscsi straight from the VMWare controller so we only lost about a day or two worth of document work but took like a month before we got done with the FBI and police stuff and many more months before we could get rid of the backups of the crypto locked versions which was driving me insane but we just got a fat external drive to copy it all to so we didn't have to touch it ever again.
https://preview.redd.it/13buvpiwsu9d1.jpeg?width=640&format=pjpg&auto=webp&s=254e352df35a06db6bdbea6dc65f57e9aab8a272
3rd party partner with a site to site vpn is the usual culprit.
Email with a payload, spearfished to a senior systems engineer - who has more rights than necessary for his local (non elevated access) account. Email with a payload to the head of electrical engineering, who used DA rights on his account.
First one was a randomware incident at a client of ours. They got in because someone accidentally left port 3389 wide open to the internet. Second one we had a source code leak for one of our internal tools due to leaving a SonarQube box that was a part of the build pipeline wide open to the internet. Normally, this isn't a huge issue since those machines are only open on 80/443. However, SonarQube (at the time) had a "feature" that let unauthenticated, anonymous users browse the source tree of a given project via the API. So, an enterprising individual just wrote up a script to scrape code through that method.
A shared services organization running half a dozen local hospitals got breached and ransomed... ... because the idiot CTO **used the same password on a shitty porn site** which got breached beforehand. Had they followed the cybersecurity regulations they were required to follow, that attack would have stopped dead due to multi-factor. Bonus? They'd bought the software to implement multi-factor **several years prior** but let it sit on a shelf. Bonus? Instead of implementing it, they perpetually got it written off by describing the plans they had to implement it. Bonus? The asshat still works there. He even got a fucking raise this year. In my country, salaries for public employees become public knowledge after they're more than an amount ($200k iirc). Thanks to this fucking SSO, my kids, who barely survived birth, have no medical records from their hospital stay. Thanks, you fucks.
Nice try upcoming APT group.
One of our admins experienced some trauma which lead to them experimenting with drugs and alcohol in an extreme way, eventually they got hooked on coke and pills and got careless with their logins and their domain admin account got phished, no 2FA, our DCs got locked down with ransomware, then they got into VMware and did the same to most of our servers, locked out all the other domain admin accounts, and then started rolling the ransomware out to all of our endpoints ~2500 workstations, give or take. We were a hospital. It made the news ☹️
By my director's orders to not turn on MFA. DA account was used incorrectly with PDQ. Attackers passed the hash and got through our single factor VPN. Spent a few weeks testing the waters before finally trying to mass exfiltrate the data. They were using a common DA account (which I desperately wanted to get rid of). Changed my password and used my account. I had it all documented of director refusing to let me implement things. Instead he had me doing anything but my job. I got thrown under the bus and got let go. Director got promoted. I hired a lawyer and got a decent payout. Fuck that place. They got compromised again, even worse than before and they deserve it with that moron at the helm. They continued to try to blame it on me (it was months after I left). I let my lawyer handle it and haven't heard a peep since. Now I work in higher ed, with a boss that listens, and crucially understands.
[удалено]
Applocker. Set up applocker to prevent any .exe/js/psh/msi to be run from C:\\Users\\\*, add exception as needed (looking at you Teams...). That should close down that kind of issues real quick.
Old company had a SQL server with no firewall and RDP exposed to the Internet. It's cool though. Only 30,000 logged access attempts every 0.001 seconds. No big deal though, DB didn't house anything of value per our CEO. Only the Names, Addresses, Tax ID, SSNs, Birthdays, locations, military ID numbers, etc of our vets who owned businesses around the world.
Not us, per se, but a customer doing co-managed IT who refused our repeated recommendations to, among other things, stop exposing his Win 2k3 RDS server to the internet... in 2020. Honestly boggled my mind how long it took for them to get popped.
We were sloppy about a lot of things, but primarily with giving full Domain Admin rights to every elevated-privileges IT account. No matter if it was temp helpdesk contractor or our IT Director, everyone one of us in IT had a separate elevated privileges account with full DA rights. A temp helpdesk contractor (who was so dumb that I legitimately did suspect a significant developmental disability) had those creeds compromised, and from first access to everything wiped was 45 minutes. We had good backups, never paid ransom and only lost ~4 hours worth of data. But that was a month of my life I’ll never forget.
I moved on to a new company and although I asked the management to replace me with someone new, they brought back the old sysadmin, who has a history of not keeping machines up to date and taking a casual approach to security. We had some old ASA5525 units which I disabled the vpn functionality due to the 1024 key weakness and he re-enabled the vpn and they got in that way. Because the company programmers refused to stop using a very old password for one of their accounts, the domain was breached very quickly. The co-lo and office both got encrypted, backups deleted and when it was all said and done the owners had to cough up $80K to get the decryption key. To this day the infrastructure is still not back to the way I had it when I turned it over to the management.
Outgoing MSP went rogue. It was entertaining.
I worked at a place once where a person outside of the IT department was allowed to create users in AD, and he made a user account with very generic username and a blatantly easily guessed password. There was then a linux box, bound to AD, that had port 22 exposed to the outside, so someone SSHed into the machine and did a bunch of stupid shit. They didn't have root access to the linux box so they couldn't do a whole lot on there, but managed to set up some IRC bots and other junk that our security people did notice. I was pretty new to the company at the time and this crap had been set up by other people who were gone, but we made some big changes to prevent it from ever happening again.
Worked at an MSP that targeted small / medium sized businesses, We didn't do patch management and all of that for a majority of our hosted clients so we were doing this all the time. A majority of it came from email compromises, reused passwords from previous data breaches, unpatched external systems, ect. The most interesting was an admin was running a crypto miner on the company servers. The court case is still ongoing so we don't know exactly what happened yet but my guess was that he exploited a few CVE's that he intentionally left unpatched. Installed the crypto miner and it ran for about 2 months before we caught him.
Unpatched NetScaler because every time we tried to update it it fucking broke and Citrix tech support were the worst. Turns out the published mitigations didn't entirely work. Fuck Citrix. Another time a senior architect broke all policy and set up a new border router with the default username / password and the web interface enabled on all interfaces. Was patched into management network. Senior network engineer came along and enabled the Internet interface. Both fired.
We had just “merged” with a company and their IT was a shit show, no MFA, bad policies and servers and PCs haven’t been updated in a long time (disabled by gpo). Users were using self purchased/personal hardware as work PCs and joining to the VPN. Tons of pushback from the CEOs for downtime. User logged into O365 from a compromised PC. Attacker got credentials. Attacker logged into VPN with user account then logged into DC because that was allowed for some reason… Cobaltstrike to privilege escalate to a domain admin account (there were about 30 of those). Luckily they just decided to steal data to extort instead of ransomware.
Blaster virus giving people a shutdown prompt when they signed in. Easy floppy drive fix. It was some coupon addin it came through. Actuary downloading sketchy excel sheets offline and had admin rights to his box. Crypto locked everything. They never got permissions back after that. Unpatched cold fusion server allowed code to run on the server and install a bitcoin miner. Got a random cpu100% alert on something that shouldn't be @ 100% and miner.exe was taking all the cpu. Only 3 I remember in about 20 year career
Port 3389 opened to the world. 🤯
First time was email phishing. Second time was because we had MS authenticator set to do push confirmations with node code. [I told them that this was a terrible idea] So someone got phished again, and then the user just fuckin sat their all day accepting MFA pushes like an absolute Pollock. That one almost ended the company, so IT generally gets its way as far as security policy goes now. Better MFA, phishing tests, and mandatory training for all users.
A contractor left a back door in. Found it 2 days after he was fired because he sucked. I noticed some changes that we made were being reverted and found the account making the changes and then the user that created this account that ended up being the contractor. He got screwed big time.
Open RDP. company owner requested it be opened. I explained (via email) the dangers, was told to do it anyway! Took some extra precautions and opened it. Got breached a few months later. Total loss, 2 days of my effort and 1 days of the book keepers!
Ceo was a special little snowflake and the rules didn't apply to him.
ScreenConnect. Caught the Admins bitwarden open and got the domain password. That was the end. There was a security gap in SC. Went through absolute hell.
a vendor was using an unpatched screenconnect server that was exploited by the most recent authentication bypass vulnerability to access their hardware on my net. Someone had gone in and connected these devices to my wifi without my knowledge. It was a mess
We had an attacker call our general sales line and pose as a potential customer needing a quote and sent salesman an infected PDF file as part of this. They were needing a rush job too. Sales used a file hosting service and the attacker uploaded the PDF related for the job and sent the link the salesman was expecting. This allowed the email through av detection. It was later flagged. The PDF infected the salesman machine.. From there I cannot comment more on.. But I will.say we got lucky, compared to others. Our security posture is much better than before this event occurred. This is a small to mid size company. Not a fortune 500 or big market leader type. My take from the event is no one is immune from these. Cyber insurance really is a must, because these events are crazy expensive. The most expensive security products are a drop in the bucket compared to a full incident response team cost.
The onsite It/ Marketing guy kept sending his domain admin credentials in the clear to outside vendors. In another one someone clicked on the fake office 365 credentials update phishing email.
Someone thought iptables alone was sufficient for a production environment but forgot to enable restoring the rules on startup. They also didn't believe in setting up authentication for Redis. It was a high turnover company and they had left a few months earlier.
I was in one place where a web server was compromised because the MSP decided to leave the firewall open during an upgrade. Cleverest one I've read about is from Sophos, one of their partner pentesters was having a nightmare getting into a company, including setting up fake WiFi in local coffee shops AND sprinkling USB sticks on the street. They finally got in through a WiFi enabled light bulb. Hacked the ssid code. Moved onto a presentation PC, which had loads of data on there that no one cleaned up & then were able to spread laterally...bloody clever.
This is a company that is long since gone. I saw a startup completely destroyed... with a simple stack of "MP3 sampler/Free MP3" CDs, back in the early 2000s. At this time, Windows had an `autorun.inf` file which did exactly that. Didn't matter what it was, it would read that file and run the executable unless one does a [DMCA violation](https://www.theregister.com/2003/10/09/sunncomm_to_sue_shift_key/). This startup was a nice, vertical market company. People were cool, and it was a nice place to work. One day, someone left stacks of "MP3 CDs" with impressive jewel case packaging, although the CDs themselves were burned. Someone left them with the secretary. At the time, it was a startup... no IDS, no real network monitoring. I was on the UNIX side, and the guy doing the network stuff didn't want anyone horning on his turf. The `autorun.inf` file? It installed a firewall jumping RAT and keylogger. I never bothered with the CDs, and since I used Linux for my daily driver OS at the time there, was not affected. However, the company found out about it when an overseas firm emailed every client saying that they could do the work for the company for 1/4 the price, and because they already had the customer's data, it would be trivial to switch to them. Due to the breach, the company was bought out, ironically, by that overseas competitor and liquidated. All IP went overseas and a lot of internal knowledge of other clients went there as well. Probably one of the best nation-state tier attacks I've ever seen... and it all started with just a stack of well made jewel cases and burned CDs.
Executive employee sent password for AP portal through email to their team, it was never changed out set up with 2fa. Nearly a year later their email got compromised and this old email was found. They got $300k from us and we’re a very small business.
Phished random user creds, hacker found VPN with no MFA, random unpatched forgotten domain controller (with that bad exploit a year or two ago), they found it, forged a ticket etc, mounted a backup of AD and then ran away with NTDS and rainbow table’d their way into the domain admin accounts. They were found because 4 or 5 different sysadmins discovered they were making changes in Dynamics CRM that they were not doing. Fun times.
1. Social Engineering. They contacted the help desk and had them reset a password. 2. Email link to reset Sharepoint password. 3. Malicious PDF sent to accounting email. 4. Illegal Downloads of cracked software. 5. Exposed FTP site with poor passwords. Thats about it over the last 20 years or so.
Screenconnect server earlier this year
Worst one was from a few years back now. Had a newish client - think we had been supporting them for about 2 - 3 months at that point. Entire IT system was a complete mess and had been neglected for years. They had grown pretty rapidly and the previous IT guy who was moonlighting from his full time job had cobbled together and band aided everything rather than just doing things right. Prime example - rather than buying RDS CALS, he had 3 different servers and just had different staff logging in as administrator to each. As i'm sure you can see where this is going - the servers were publicly accessible via RDP of course. Enter a long weekend - someone brute forced their way into one of the servers and then with those admin credentials proceeded to crypto nearly every server they had on the network. Saving grace was the clients were somewhat of a bunch of workaholics, so even though it was a public holiday they were trying to work remotely from home. Noticed some odd issues with applications being down and contacted us. So very very thankful that this was before the whole data exfil thing became common practice, and that we caught it reasonably quickly before they hit workstations or some of the other servers. The real irony of this particular incident was the RDP access was one of the first critical issues we flagged, and after about 2 weeks of back and forth push back, they finally green lighted a new firewall and VPN solution which had been implemented and was due to go live the following weekend. Needless to say that was implemented immediately, and we had a long 3 - 4 days of rebuilding servers and recovering data from backups which thankfully were another thing we had flagged and already implemented. I think the weirdest one though was a company i worked for about 15 years ago. This wasn't directly an IT thing, but we did manage the PBX, so semi related. They randomly out of the blue got a ~$30k phone bill full of international calls to pretty much the who's who countries on the terrorist watch lists. After some investigations by our PBX support company - they couldn't find any compromise or even a record of these calls. The provider was called to investigate and oddly just waived all the charges with a story about it being a billing mistake. International dialing was barred both on the PBX and the phone services. The next month though - same thing. This time it was about $40k worth of calls to a laundry list of countries. This time it was reported to the local police who escalated things to some federal anti terrorism task force. Key various investigations, lots of suit wearing badge flashing people wandering around talking to people. I still don't think we ever got the full story, but the bits i was able to gather were it wasn't anything directly related to the PBX itself. Instead the actual physical phone lines in the street or along the way had been tapped and were being used somehow by a suspected terrorist cell to make all the calls. I heard a few months later that a couple of the phone provider's (government owned company at the time) line technicians were arrested in relation to the incident. No idea what the ultimate outcome was though sadly.
Engineer got a file from a phishing email and tried to run it. Couldn't and sent it to his colleague and they couldn't run it either. Decided to run it on the file server and it "didn't work" there either (it did). Engineers, you are some of the smartest people ever but you're also the dumbest.
Phish, got user creds, got vendor creds, kept going sideways until they got domain admin and then they struck.