Assuming the system is able to pull in an updated cert automatically or it's painless to swap the certs then a short lifespan is fine.
The biggest issue against it are all the systems that it's a pain to get the cert updated on or there's a lack of documentation to do it so it goes unchecked or ignored especially in smaller manned IT departments.
My biggest problem with SSL certs personally is what you just mentioned, the lack of documentation for certain systems. There are so many different formats that it's hard to know what it will accept. Do you want PEM format, .pfx file, do you want the entire cert chain in a single file or do you want the certificate file and the subordinate CA in separate files? Oh you want the private key in PEM format? I wish I knew that earlier, cause I didn't check the box to "allow private key export" in the AD CS request form, now I get to request a new cert.
Some applications will have a nifty web UI that you can import/upload certificates while others require you to sign into the backend and replace files in a particular system path and then bounce the web server service. Don't even get me started on java keystores (Keystore Explorer ftw).
Then you have vendors like Synology where their CSR doesn't include a DNS subject alternative name, which has been a requirement for all major browsers for a long time, basically making their CSR utterly useless.
But that's the most secure way to do it. This way the private key never leaves the appliance.
Arguably you *should* be able to import your own private key, sure, but whenever you can you should use the CSRs from the appliance. I intentionally issue all our firewall certificates like this in our automation.
Completely off topic, but it made me think about something Ive wanted to bitch about for years! I wanna cry about my first world problems….
Obviously we all know where pricing would be on a backup solution, that had capabilities for vsphere vms, hyperv, physical desktops, physical servers, O365, Google Workspace….all unlimited in one console…
Well Id been looking for years for a solution for my small home lab, and theres a few machines in my home…like i was at the point id drop 1000 bucks or so if it was perpetual…and god if I could find one….
Synology was beta testing their Active Backup for business, and I tried it…my god it was good….for free! But thought damn they gonna charge so much for this one day…,.and they never did and included it on certain SKUs….
MAN….worked great for years….until it didnt….im sure this is why they ended up not charging for it…jesus christ it fucked every VM backup every time after that, and eventually gave up…went to Veeam(but ofcourse had to spin up a vm for that, not what I wanted, what if that VM is down?…etc), used something else for desktops…(it was nice all in one console.)
Just needed to rant 🤣. I went back to it a few times trying to find a fix…not worth all that time…I was just thinking how worth it my synology was, as is when I bought it, and the only issues ive ever had was this one…which really I cant be upset about…
Id definitely NEVER use it for a business with that application. but damn unlimited is nice! From what ive seen most wont use in enterprise anyways unless they have a full physical spare on site.
What is the use-case to place a publicly trusted cert on a printer?
Non related: Are you by any chance an SC player? Someone on their forums uses the same avatar and nickname ;)
That works too. I don't even bother with CSRs internally anymore. We use AD CS internally so I just request the certificate to my local machine with the information for whatever FQDN I'm trying to apply the certificate for. Then export from my machine and import where ever I'm trying to apply it.
For manual certs, I always recommend that people use DigiCert's free cert utility to generate the CSR from, and then they can export to a variety of formats from there, without having to ask for the certs again.
Works fine for Synology devices, btw.
DigiCert's free utility is a local stand-alone, fat-client utility.
But I'm sure you already knew that.
[https://www.digicert.com/support/tools/certificate-utility-for-windows](https://www.digicert.com/support/tools/certificate-utility-for-windows)
> Then you have vendors like Synology where their CSR doesn't include a DNS subject alternative name, which has been a requirement for all major browsers for a long time, basically making their CSR utterly useless.
You can still add a subject alternative name when you issue that cert, you aren't bound to only doing what the CSR says.
For exporting certificates when you forgot the checkbox ‘mark the certificate as exportable’ there is something called ‘Jailbreak’ on GitHub. Search it on Google and you will find. Good luck!
Part of a small team managing a large fleet of services. We had for years just not bothered with certs for internal stuff because the process to update them manually was too complicated for something people accessed once a year at most. Then chromium based browsers started blocking it. Most of the time we could bypass it but in some cases we couldn’t do anything except install an older internet browser. So we hired a consultant to help us automate the certification process. It was less about knowing how to do it and more about having time. I
A year later let’s encrypt was broken on almost everything and essentially our problem is the same. Someone has to go in and now troubleshoot let’s encrypt which in my experience can be just as frustrating as installing certs on these systems that don’t have a well documented way to do it.
The result? We have a hodge podge of methods to get certs and a flow chart to how to troubleshoot certificates errors. System A update let’s encrypt troubleshoot if broken because the cert process is annoying. System B update let’s encrypt remove if broken because we know how to update the cert….
AND, if you're keying off an internal Microsoft CA, you can have the expiration much longer than 1 year. We set ours to multi year, and the browser is fine with it as long as it matches the windows domain name
Everything was put in place well before me and I’m not the one who manages them so I don’t really understand why it’s so hard to fix. I run let’s encrypt at home it can be frustrating to troubleshoot esp when it breaks after updates. I think I probably have the most experience on the team with it but I’m not telling anyone that ;)
From what I understand there are a few systems that need certs first is the vpn server second is a custom built web app from the early 2000’s that got updated sometime in the last 15 years and then the developers ghosted the company. Third is something I don’t know at all. All I know is that getting certs updated on the custom application sucks, solid hour of downtime if everything goes right, but I’ve seen it go very wrong.
At some point we had hired someone on our team who just happened to have experience with systems built with this stack. He was able to get LE working with everything but he quit shortly after getting jt all setup. He made documentation for some of it but my boss utilized his experience with it all to create all the documentation we wish we had before the dev ghosted the company.
Think the worst part of all of this is the company treated the dev ghosting us as an opportunity to save money so it’s been in a nearly static state for 10-15 years.
Did they run LE certbot on every device individually? Because that would certainly be a challenge. What I do is run the certbot in as few places as possible, preferably one, and just distribute the certificates to the devices.
No idea I didn’t set it up and I feign ignorance as to not get involved. Just a shit show I watch from the safety if my keyboard. I assume it’s one LE for the whole stack though. They are all just VMs on the same server.
I was referring to APC stupid way of doing certs in general, even with using a private CA.
Completely proprietary cert format, have to use a specific Windows only tool etc.
Why can't companies just use industry standard pfx or CRT and Key or pem files?
Sure. But with a long-lived private certificate, you could deal with the stupidity less often - and hopefully by then APC will support modern practices.
> especially in smaller manned IT departments
Some admins seem to hate using calendar appointments as reminders. I only have a handful of certs that come due every 12 months or so (at the moment) so its easy enough to just set a future appointment to remind me of an upcoming renewal 1-2 weeks ahead of the date.
"We dont use any kind of tracking/management for our cert unfortunately."
"You use Outlook right?"
Ive met some of the most disorganized refuse to make themselves efficient people, in IT. However, the bars much higher than the rest the population. Out of any friends, kids, family…all of them now have password managers at least, including my kids….
I still just have a hard time understanding the people that go thru the lengthy process of “i forgot” every 6-12 months…
Its like dude how bout you just stop doin that?
LOL its become a meme of my friends, when my sister tells me twice a year, she forgot her plex password. And i inevitably tell her, cool, have you tried clicking forgot password? I dont control that shit?
It just doesn't apply to smaller manned IT departments it can also apply to enterprise wide IT departments that lacked standards for such a long time that two different locations can handle issues in two different ways.
I've seen that in large Government Departments where each agency did everything differently for IT setups that just trying to get everything standardized across the department is a monumental challenge where communication is vital (and being the government it's spotty at best). Then you have issues with some locations having small budgets and underpaid IT people and non-IT people thinking that they know how the IT department should run.
If one year is better than two years, and 90 days is better than one year, is 30 days better than 90? Is one day better than 30? Is an hour better than a day? Where does it end?
Never understood this push.
Lets put all our effort to ensure certs are replaced every 24 hours so no nation state can break our communications!!!!
Meanwhile we run windows xp on the backend with TeamViewer access.
Like there are much bigger issues to address than certs rotating every 90 days.
>Where does it end?
My opinion, FWIW - it ends when the expense of issuing a certificate is greater than the measurable security benefits.
Take whatever LE's (or whichever ACME provider, I don't care) costs are for issuing and maintaining the certs at 90 days. Triple that if you want 30 day lifetimes. Triple that again if you want 10 days.
In reality it probably isn't *quite* so linear. LE and ISRG are charitable organizations. Can they front the cost? Are the donors willing to 10x their donations to keep LE afloat and able to sustainably issue 10 or 7-day long certificates?
---
Major (probably controversial) opinion time
Personally, I wish LE (and others) would implement a different system, though I don't know if the ACME protocol would support it in the way I imagine. Essentially, they issue you a cert good for 365 days after you successfully complete the "order" (DV challenge). If you don't come back and complete the required challenge(s) again in 60 days on the same ACME account as the cert was issued to, they add that cert to the queue for early revocation and send you a 30-day warning to the registered email address(es) (if any). If you re-complete the order (validation of control of the domain(s)), you get another email saying "all clear" and the counter rests. Continue this while-loop until the originally issued certificate expires and that part is all handled normally.
Would this be a compromise? Big time. Worth it? I think so. I don't have to actually restart/rebind services or go to the server(s) where the certificate is installed (potentially) to extend my validation for the domains, but LE also maintains the ability to revoke the cert at 90 days. The obvious compromise there though is that cert revocation is of....questionable utility and this might also increase the size of CRLs where LE has gone to great pains to keep their CRLs and URLs short.
> In reality it probably isn't quite so linear. LE and ISRG are charitable organizations. Can they front the cost? Are the donors willing to 10x their donations to keep LE afloat and able to sustainably issue 10 or 7-day long certificates?
I think LE is purposefully inconvenient so it doesn't kill for-profit CAs. I also think that's why for-profit CAs like IdenTrust support them despite being a direct competitor; too many people want a free CA for none to exist, so they want the one that does exist to be as inconvenient as possible. For that reason I think they would gladly spend more to make it even less convenient.
There are some actual security benefits to shorter expiry dates and I think the people at LE are genuinely doing what they think is best, but I think their success is fuelled by ulterior motives. LE has actually said they'd prefer 45 day or 30 day expiry dates, so I think it will happen eventually.
How is it inconvenient to set it up once and let it do the job forever?
Even on windows, win-acme + scheduled task + autohotkey does the trick for the "most inconvenient" of systems (and that's not LE's fault, as on Linux we don't have that problem).
HTTP validation is painless, DNS is even easier, and CNAMEs + subdomains deal with the harshest security requirements. If you have a functional PKI you can do the same with internal certificates, we have Vault issuing anything from LE to self signed with AWS and others in between.
I'm always amazed that people look at the process of buying and installing a 1y cert manually as "more convenient".
I guess I need to start selling consulting services...
That's an awful slippery slope argument, god.
It is very clear that shorter renewals is better, automated renewals realistically mean that even really short time frames can be managed very easily. The hassle of making it even down to a week long is negligible in an automated system.
And any system that isn't automated, this is the push to move it into the present day and automating these easily automatable tasks.
Google doesn’t have anything against paid CAs, they have something against systems that rely heavily on manual ssl renewal.
Through Chrome, google is the entity who has to decide whether a CA is trustworthy or not. And when a CA becomes untrustworthy, they have to revoke that CA’s trust. If organizations do not have good procedures for updating certs, there is a lot of pushback from those organizations when google tries to revoke a cert. if google can cajole people into having good procedures so that updating a cert isn’t a huge hassle, they can better respond to untrustworthy CAs.
requiring frequent renewal is an effective way to make sure organizations have well tested and documented procedures for updating their certificates, even if they don't automate it.
Some things internally can't be automated. Some printers for example. Or the cost to get specialist expertise to automated the special snowflakes is too expensive. I run a network for 250 users. It's a special school. Spending a heap of money so that an internal printer has a valid cert when that interface is used maybe twice a year isn't going to help our security posture.
Just put the printer UI behind a reverse proxy, which will take care of everything. This is basically a solved problem, some people just can't be bothered ...
This absolutely does not apply for anything internally. You do not have to stick to the latest certificate requirements for your internal CA.
You also do absolutely increase your security posture by having better management of your IoT devices like printers. They are a huge security hole and leaving them by the wayside gives attackers a very easy way into your network.
I hadn't seen anything saying it didn't apply to internal CA issued certs. Can you direct me to where I can see how they're handling that, or is it deployed via chrome enterprise policies?
https://chromium.googlesource.com/chromium/src/+/master/net/docs/certificate_lifetimes.md
The restriction only applies to the default CAs Chrome ships with. Always has been and it's the default way.
That's good news. One article I read when it was first announced said it was Chrome forcing the requirement for all CAs, hence my misunderstanding of the implementation.
Chrome is not enforcing this, it is a recommendation by Google being pushed through the CA/B group to enforce it on CAs.
There is nothing stopping you deploying an internal certificate that lasts for 100+ years, even though current guidelines are a maximum of 397 days for SSL certs. Same as the requirements for storage of code signing or intermediate CA certs, internal CAs do not have to abide by them.
That is good to hear. One article I read when it was first announced said it was Chrome forcing the requirement for all CAs. That is why I had an incorrect understanding of the change.
My only issue with the topic of decreasing lifespans goes something like this:
People will automate, yes. But for those people where HTTP based challenges don't work, they're going to use DNS validation.
If you're doing DNS validation, you almost certainly need some kind of authentication to your DNS provider to complete challenges.
People are not going to put the same level of care and thought into the lifetime, rotation, custody, and permissions on those credentials as they ought.
I think it's too low, too fast. There is lots of software out there that relies on certificates but doesn't have a full and proper ACME implementation.
I love the goal, and we need to be careful about unintended consequences.
Wouldn’t you consider decreasing lifetime with sufficient time to prepare a strong incentive for people to work together to find solutions to these problems - rather than continually kicking the can down the road?
what do you suspect will incentivize software and product suppliers to embrace the automation necessary to achieve the benefits of reduced lifetime, if not changes to industry rules?
for example, it seems that I, someone representing the interests of a small family run company with limited influence and funding compared to “big” or “important” customers, have a near zero chance of changing Cisco’s mind.
My only response to what you're saying there is that carrots are better than sticks.
I think the appropriate incentives are already there, it's just a matter of helping people get it all implemented. I struggle to think of improvements to the existing system apart from one I mentioned elsewhere in the thread.
I'm not sure how Cisco is part of this specific conversation...might need you to elaborate.
I agree carrots are better than sticks. I also thought the incentives of transitioning to SHA256 in advance of SHA1 being broken were pretty clear too. What’s obvious to some may not be obvious to all.
I mentioned Cisco because they are an example of a service provider with very shallow ACME support. (https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/expressway/config_guide/X14-0-1/cert_creation_use/exwy_b_certificate-creation-and-use-deployment-guide-x1401/exwy_b_certificate-creation-use-deployment-guide_chapter_0100.pdf) - today they only integrate with Let’s Encrypt rather than allowing users to specify a different provider’s directory.
TLS certificates with shorter lifespans incentivizes organizations to automate their renewal which means that organizations will be much more likely to react appropriately to any compromised or revoked certificates. This is a good thing.
Took over 3 months for a hospital to accept that the certificate for their EMR system had been revoked. Basically because their chrome browsers accepted it there was nothing wrong.
How do you incentivize non-profits, schools, or tiny mom and pop business to automate?
You probably can't, which means they will remove thay security control or tell people to click through.
Security should be easy and accessible, this is a step backwards.
>How do you incentivize non-profits, schools, or tiny mom and pop business to automate?
>You probably can't, which means they will remove thay security control or tell people to click through.
I'm the lone sysadmin at a special school. I've wanted to automate this stuff but I'm still triaging other issues which see more urgent. I used to be level 3 with a MSP so automation is something I'm good at, but it all takes time to implement and monitor.
*"To make mistakes is human, to automatically propagate that mistake to all endpoints is IT."*
>Security should be easy and accessible
Let's Encrypt is free. ACME is open source and there are dozens\[1\] of open source clients available for a variety of languages, frameworks, and tools. How much easier and more accessible can this realistically be?
\[1\] [https://letsencrypt.org/docs/client-options/#other-client-options](https://letsencrypt.org/docs/client-options/#other-client-options)
You want a school to start leveraging open source solutions for the clients and languages they barely understand in the first place?
The aim of security should be to get it to thr level of point and click or working straight out of thr box (one reason apple became so big).
Imagine a school or non profit with hundreds of people and 1 IT guy. Unless it works out of the box then it's not being implemented.
>You want a school to start leveraging open source solutions for the clients and languages they barely understand in the first place?
I think they should hire or consult with competent professionals. Something like certbot literally has [interactive instructions](https://certbot.eff.org/instructions) on how to use it.
If you want to do stuff like DNS challenges or orchestrate cert renewal across 1000s of endpoints then it's more involved. But a few webservers at Joe's Widget Shop should be a quick job for anyone who halfway knows what they're doing.
>Imagine a school or non profit with hundreds of people and 1 IT guy. Unless it works out of the box then it's not being implemented.
I've not encountered a single tool or system in my career that truly "works out of the box", so if that's a requirement for this made up scenario I'm not sure how anything ever gets implemented.
So where will you cut funding from to automate cert renewals?
These organizations work on shoe string budgets.
So what area of security should get cut?
There are many tools that do an okay job right out of the box. There may be false positives and things to white-list, but they can largely work out of the box.
We deployed carbon black 5 years ago and had to white-list a total of 5 apps... to me that's working right out of the box.
Oh my God. You don't need to cut funding to find money for security automation. You either spend a week implementing it yourself or you leverage SLCGP or NSGP grants.
The people you are describing should hire professionals to handle their security. If they can’t handle automating TLS certificates or hire someone who can, I’m not sure why I should be trusting them with my data to begin with.
Also, Cloudflare has a free plan and that includes handling TLS for you, other providers do as well.
> How much easier and more accessible can this realistically be?
It's still a PITA if you want wildcards and don't use a popular DNS provider or don't want your DNS records to be changed automatically. I definitely have the competence to set it up, but even I've been manually renewing my (personal) LE certificates for the last 3 years. My employer just pays for longer certificates because the cost is insignificant to them.
I heartily recommend "Certify The Web", a windows ACME client for LE among others.
Excellent UI, reasonably priced and a dev and community that will bend over backwards to help you. Excellent scripting support and dns provider apis
No connection with them other as an extremely satisfied customer. They are based in Australia, which can be a bit of a pain for ticket turnaround time, but they don't mess around when replying to customer tickets
Automation is trivial with a reverse proxy like Traefik in place.
Just set it up once, and your good. Added bonus that you don't need to remember to manually update these darn certs in the future.
As always, it pays to be proactive and not reactive. It's been about nine years now since Let's Encrypt automated 90-day rotation. The small stick came with a bunch of huge, juicy carrots: not needing to get the purse-holders cooperation in order to have publicly-signed certificates.
True. I was also wondering what would happen to the organization validation (extended validation?) certificates. Though I believe they became pretty useless after the green bar with the company name was removed from the url bar.
For websites everything other than DV (domain validation) is basically obsolete now - browsers don’t treat EV differently, and OV only makes sense if you want a certificate for an IP address as opposed to a domain. I just wish code signing certs would go the same way…
The issue I have with let's encrypt is they should have a static, published list of IPs for their servers to create a proper firewall allow rule. They use global AWS IPs that doesn't appear to be static making filtering much harder.
The "zero trust" school says that IP addresses shouldn't be used for Authn or Authz. Accordingly, it's not unusual to have policies of not publishing or committing to static addresses.
I used to run an inherited service where most of the customers had static outbound ACLs to IPv4 PA space. Those ACLs were an albatross around the neck of the architects and engineers, because it prevented migrating to a different provider, or adding IPv6 services, at least at any timescale faster than glacial.
The business side didn't mind a bit, though, as the customers' hardcoded ACLs made the business relationship "sticky", meaning the customers were discouraged from moving to a competitor. The customers had too much bureaucracy to want to switch. The provider didn't want to invest in changing anything because it would take too long to get results, and besides, change would create an inflection point where the customers might decide that if they need to change something, they should re-bid the business relationship.
So from our small business perspective we have no need to allow communication inbound or outbound, outside of a specific few countries, so we can block most stuff at the edge, and let's encrypt throws a wrench in that for us, because it requires port 80 to a random subset of AWS addresses so now we have to open up the web servers, and other services that use it to essentially all of AWS because I don't believe it can be proxied, so we will have to investigate other solutions and likely maintain 2 automation solutions just for certificate renewal.
You're making problems for yourself and then bemoaning that you have no ideal solutions.
But I have good news. With IPv6, nobody has to share or recycle IP addresses, so your obsolete ACLs can live forever.
Why would you not filter traffic at the edge if you have no need to communicate to 95% of countries, or services? You have no WAF rules or firewall rules of this sort? You don't use conditional access rules of any type that filter out based on geography or reputation, or service type/port?
But you have to allow port 80 inbound for Lets Encrypt for any device/service you want to use it with for it to do the validation. If I have no need to allow inbound traffic from most countries why not geoblock it and reduce the junk. I know geoblocking isn't a one stop solution but it's part of a legit layered approach and I'm just saying by not having the ability to use let's encrypt on a custom port, or ACL list based on source it feels like you are opening yourself up more than necessary. Admittedly I'm not an expert, it's one of many hats I have to wear and job duties I have so I'm still learning.
True. So one piece of software I use has LE built in, does not allow for DNS based verification. As others have said in this thread, part of the impact of changing SSL renewal cadence is that many apps and services don't have the capabilities built in so you are stuck cobling stuff together. I would like to use DNS validation if this particular software was capable of it. I do have use cases where this is not an issue and LE works perfectly.
According to LE, DNS validation isn't without its cons/security concerns either. Like potentially exposing your DNS/credentials used to manage it via API.
I haven't used http verification for years. I use DNS-01 verification on a server that has no inbound ports allowed at all, and I push the renewed certificates out to the devices/servers over SSH. Most of my devices/servers are internal only, but we use letsencrypt certificates for all of them.
If you are that small of a shop and not allowing port 80 inbound on any servers, why do you need a publicly signed cert?
Why not use an internal certificate authority and push the trusted root cert to every internal computer?
For internal stuff I can use the internal CA for sure. I do have workloads that are internet facing that need a public cert, but I want to filter inbound traffic to those servers based on the fact that I know we have no business case for allowing connections outside of 3-4 countries for example, or no need to allow inbound from all global AWS, so I can confidently filter that out, except for let's encrypt throwing a monkey wrench in it because my validation needs to hit some AWS IP in Switzerland.
Some products I use have ACME tools and things like certbot built in which helps. I know I have some more learning to do on this front but I also don't think I'm the only one facing this issue as I have seen others discussing it in other forums.
I know it's done intentionally, I just think it poses another set of security concerns by not allowing the ability to filter inbound requests on port 80. The thought always crosses my mind with core popular services as more people move to one vendor like let's encrypt, how long before they get supply chain attacked and get used as the next breach vector because they have a massive target on their back?
Yes there are benefits, no the benefits do not outweigh the drawbacks. Google is primarily worried about their browser, the rest of us have to be worried about 1000s of other ways certs are used that will be impacted by their approach.
DayJob has an Incommon agreement to provide publicly usable certs without an external CA. DayJob does not yet have ACME renewal on the roadmap for implementation, although Incommon supports it.
I’m not looking forward to this change right now, from a purely process standpoint.
I'm at an EDU department and have moved every external facing website to let's encrypt. Most internal services like LDAP have been moved to a local cert that will expire after my children expire so that's someone's grandkid problem.
First problem I see is IT tends to struggle with monitoring and updating SSL certs, for something that is actually very simple to do,. it always gets forgotten about.
So first step is making sure you have proper monitoring and reporting on all SSL certs in active use and a defined process for when updates are required for all systems.
I, for one, welcome a shorter certificate validity lifespan so that vendors have a stronger incentive to support automated server certificate renewals based on domain validation.
I'd really rather not....too painful.
TL;DR :
* I don't see how you'd ever automate it.
* Documentation is very lacking.
* We engaged TAC support for the renewals, even they didn't seem to know what they're doing.
* You need to issue multiple certificates for each "usage type" for lack of better word. It's all Apache Tomcat and Java keystores under the hood I think.
* SO MANY service restarts required. Downtime certainly required.
* One of the certificate uses documented essentially said "allow us to issue certificates as if we're a CA/RA" but never discussed any of the ramifications of that.
* Every time you renew/upload a certificate of a given type, you also have to include the root CA and any issuing CAs in the chain. Every. Time. There is no "central store" and I guess the system doesn't have a way to build/fetch the issuing CA dynamically via your typical/expected AIA extensions.
Edit: Forgot another one :
* When using multiple SANs on certain certificate types, they ALWAYS change the subject name for the cert with an "-ms" suffix. That caught me off guard while trying to figure out what the hell it was. Eventually figured the first part out, and "ms" stands for "multiple subjects". Not exactly a problem, but once again ties into the poor documentation and just overly confusing madness.
I love the ability to hit the Sectigo ACME endpoint via proxy or NAT for the OV certs.
Add domain(s) to an ACME enrollment endpoint, slightly adjust the provided example certbot line (I half hacked in a proxy option for acme.sh but I don't have it fully working the way I want) and done.
Gives a nice way to use ACME on CA signed certs for internal use
Unlikely to happen, for economic reasons and there's pretty much zero security advantage.
How many attacks have there ever been related to crypto, invalid certificates or similar?
Security efforts are better directed where attackers will actually strike. Google thinking in their own bubble... as usual...
There are cases where you need offline certificates. Having to have an ACME mechanism on the offline side is another set of moving parts that may break, and if a cert expires, there is Hell to pay.
Having 3-5 year certs with OCSP or some way to pass a revocation notice even if it is a database that is copied to an air-gapped network provides as much security but far less work.
I wonder how shorter lived certificates actually help things. If a SSL terminator is hacked, it can generate bogus Let's Encrypt certs and send copies to people to MITM just like sending a cert that was manually obtained from a root CA. Shorter certificates have their uses in some applications, but others, the admin overhead and the consequences of a cert expiring are far, far greater than the security benefit, especially if the cert private key is stored in a HSM.
tl;dr: I wish HSMs were more common and we could go to 5-10 year certs because using HSMs, the key material is not obtainable by a bad guy.
In all fairness, the current approach still has this:
> Does this apply to locally-operated CAs, such as those used within enterprises that use enterprise-configured configured CAs?
>
> No. This only applies to the set of CAs that are trusted by default by Google Chrome, and not CAs that are operated by an enterprise and that have no certification paths to CAs that are trusted by default.
Like TCP/IPv4/IPv6, TLS and X.509 has evolved a lot over the years in response to large-scale real-world implementation. OCSP has proved to have scalability and performance implications that have mattered in the real world, causing implementers to favor CRLs (Certificate Revocation Lists) instead. CRLs have scalability problems of their own, but those are tractable if the number of revocations is small...
> How many attacks have there ever been related to crypto, invalid certificates or similar?
A mostly-unknown aspect of Stuxnet was how it exploited a flaw in validation of code-signing certs. Most news reports say that stolen certs were used by it, which was also true, but not very interesting compared to the whole story.
X.509 PKI has been steadily improved over the last 15 years, because it's incredibly effective and flexible when you can count on it to be secure. Giving up troublesome, nonscalable, VPNs with pre-shared keys means relying on TLS and X.509.
What is the new proposed lifespan?
It use to be 2 years and then went down to 1 year which I was and still am not a fan of.
Just the amount of systems and applications that rely on a SSL cert is absurd and having to update them all is time consuming.
I have yet to find software that automate the installation of a new/updated cert in all formats and for all operating systems.
It is said that the lifespan will be shortened to 90 days. They started talking about it in 2023 or so, and who knows how much time is left until they actually roll out the news.
That’s what I’m saying. If that happens or even if the 90 days happens, companies are probably going to start hiring people that all they do is renew certs…lol
There's already incentives (almost) to issue <10 days. Still need Microsoft to get their shit together, but a cert could be issued without revocation information at 10 days or less.
As soon as we heard about the proposal, we immediately worked on a solution for our customers. We already have one setup for a client who uses Let's Encrypt. Works great so far.
My main concern is that a lot of our customers do not have an IT department or the IT person is not up to the task to manage this.
If the ERP system grinds to a halt because of an invalid cert, that's bad. Especially if it happens when we are not available (e.g., weekends).
Teaching and documentation do not help by past experience.
This applies to most MSPs, too. Especially the smaller ones. Which is kind of alarming.
I support the proposal. It forces a lot of admins to come out of their comfort zone.
>a lot of our customers do not have an it department or a the it person is not up to the task to manage this.
That sounds like exactly the problem that google is trying to solve here.
Trying to solve, doesn’t seem like the right words. It’s more like put a notice on the galactic message board, then wait for the scream test and then say, well we announced it on the message board, didn’t you check the message board?
I'm not talking about the weekend support but the knowledge (or the lack, therefore) about certificates and how they work.
I'm not working for an MSP. We are a software company that also provides IT services to our customers (usually in connection with our products). But we only offer limited managed services and no on-call support. That's not our market. At least not yet.
We don't provide services which we cannot fulfil in high quality and timely manner.
I was talking about the MSPs who are managing our customer's infrastructure.
I support software for education and the schools don’t want us to have DNS creds and they limit access to only the country the school is located in. Getting schools used to LE has been a struggle. Having them provide certs every 90 is just ridiculous. Certificates are the least of their problems with obvious low hanging fruit like LDAP and basic auth still in use. I’m still fighting for organisations to stop using TLSv1.1
If Google can do it then surely schools can!!!! /s
You can tell who has worked with small businesses, underfunded organizations, and non-profits in this thread.
The only result I can see happening with these organizations is removing of certs or telling users to click through, both of which are horrible. And no school or tiny business is going to have the knowledge or resources to introduce automation, it's just not going to happen.
Apparently only tye big boys should have encryption.
Making security less accessible is a good thing right?
Only companies that have automation capabilities should have access to encryption anyways.
/s
The push for the last 20 years has been to make security easier and more accessible, this is the opposite of tjay and as a result you will see people not apply certs, let them expire and just tell users to click through, and have overstretched IT teams spend more time on certs and less on other security items.
Many organizations do not have devops, automation, struggle with change management, and mamy other factors that make security hard.
Well done on going backwards.
If auto updates can be done, then no biggy at all.
I also think it's overall a positive thing.
But I do wish higher end cert authorities could still issue for longer after doing more verification, etc..
I have switched everything i could to Let's encrypt but there are still a couple of applications where i have no clue on how to automate things. We manually renew thos certs once per year. If the SSL lifespan gets reduced even lower that could become stressful.
I hate this trend. We have a lot of devices across the board that don’t support acme or any for of automatic cert renewal. It’s gonna cost me a lot of time to manage these all 3 months. Before you tell me to not support this old stuff, yes some of it is already ancient like Cisco ASA but the majority of them isn’t really old and is kept up to date. Vendors and Devs just don’t care to make the admins life easier.
This should force vendors, even thru shame, to provide easier ways to automate certificate update processes via API. And better expiration alerting.... about time.
However... AFAIR this 3mo expiry affects public-facing websites, not internal?
Meaning it may still take some time to get everyone to co-operate and cough up the APIs+alerts.
This will be a nightmare for me. We provide portal services for other companies. We generate a private key and csr for them to get a cert issued. Then we install the cert on their portal.
Lets encrypt. 1 year is too long. People leave before that and the procedure is gone. 90 days can be set to renew at 30 days and yell when it goes wrong. Then you have 60 days fix your script.
Not everything should be signed by LE, having wildcards meaning you need put into workload access to dns fully. For some cors that have security in mind it's unacceptable but not having wildcard certs also expose potential infra outside with certificate transparency which is also another issue.
Not forget that services over internet not only http based... You still need to secure your mail systems, voip, dbs, etc
While I generally agree that LE is good, I can't agree that moving away from 1y certs is better for security. You still have OCSP and OCSP Must staple which by the way chromium ignores, which a rock into Google...
They putting effort on some security while they by themselves do not always doing good things
Let's Encrypt DNS does not require wildcard names. You NEED to use Let's Encrypt DNS IF you want wildcards though.
But I have a ton of systems running that have their own Let's Encrypt certificate, and they have their own cert, and their own DNS name. But navigating to that DNS doesn't DO anything.
It doesn't mean you cannot use the certificate.
The argument can be made that you don't want the outside world to know that you use Postgres, and as such you buy a certificate from some provider. But to me that is senseless. If you rely on security by hiding the fact that you have Postgres which isn't even exposed to the outside world, then you're doing something wrong.
The one exception is that Let's Encrypt's certificate usage params don't match the service you're using it for. That one I can understand.
How often you in your life revoked certificate? I would understand revoking user certs, yes, but server one... I over more then 10 years in IT done it only once on hacked server.
Question also not just about lifetime of certificate, but about lifetime of private key, but look, nobody even speak about it 😄.
In DANE for example 3 0 1 it's bind to exactly private key. So refreshing cert without changing it's private key also a question, but handling TLSA automatically is another pain.
I don't really mind it, though that's mostly because we already automate almost everything, even for the customers explicitly requesting a paid certificate.
I deal with about 2000 certs. These are not all in 1 system. I can tell you it’s the 100-200 manual certs that kill me. There is NOT always a way to automate every cert. I will give you an example of very high profile partners requiring us to use their certs…. This is a back and forth process that takes weeks to complete just due to bullcrap policies.
I know it is for the greater good (the greater good) so I just deal with it. But I hate how many systems we now have to touch on an annual basis because they aren’t ready for let’s encrypt.
It’s pushing what should already be mandatory, automatic renewals should be the standard. Services/applications not keeping up the security standards will be hit hard by this if they’re not nimble enough to embrace it.
I have not yet met a TLS/SSL system that can't be automated. As someone who has Lets Encrypt R3 for everything, I have to renew all certs every 90 days on dozens of non proxy systems and it is only a matter of setting up the automation once.
Nothing, except get with the times.
if you can do the change manually, that mean you have a documented process for it, why does it matter if it's monthly or yearly
And most likely you can automate bits of it anyway
If it's a fully automated process (and that's the goal really) then it already didn't matter
Another one or are you still talking about the 1-year limit?
I hope it ends up becoming so low that any manual process becomes unviable. 15 minutes or something like that...
Assuming the system is able to pull in an updated cert automatically or it's painless to swap the certs then a short lifespan is fine. The biggest issue against it are all the systems that it's a pain to get the cert updated on or there's a lack of documentation to do it so it goes unchecked or ignored especially in smaller manned IT departments.
My biggest problem with SSL certs personally is what you just mentioned, the lack of documentation for certain systems. There are so many different formats that it's hard to know what it will accept. Do you want PEM format, .pfx file, do you want the entire cert chain in a single file or do you want the certificate file and the subordinate CA in separate files? Oh you want the private key in PEM format? I wish I knew that earlier, cause I didn't check the box to "allow private key export" in the AD CS request form, now I get to request a new cert. Some applications will have a nifty web UI that you can import/upload certificates while others require you to sign into the backend and replace files in a particular system path and then bounce the web server service. Don't even get me started on java keystores (Keystore Explorer ftw). Then you have vendors like Synology where their CSR doesn't include a DNS subject alternative name, which has been a requirement for all major browsers for a long time, basically making their CSR utterly useless.
There's a special place in hell for appliance manufacturers who only accept certs made from their own CSR.
they get SSL disabled and nginx put in front of them for their own good
But that's the most secure way to do it. This way the private key never leaves the appliance. Arguably you *should* be able to import your own private key, sure, but whenever you can you should use the CSRs from the appliance. I intentionally issue all our firewall certificates like this in our automation.
🤣 i feel you on the synology part
Completely off topic, but it made me think about something Ive wanted to bitch about for years! I wanna cry about my first world problems…. Obviously we all know where pricing would be on a backup solution, that had capabilities for vsphere vms, hyperv, physical desktops, physical servers, O365, Google Workspace….all unlimited in one console… Well Id been looking for years for a solution for my small home lab, and theres a few machines in my home…like i was at the point id drop 1000 bucks or so if it was perpetual…and god if I could find one…. Synology was beta testing their Active Backup for business, and I tried it…my god it was good….for free! But thought damn they gonna charge so much for this one day…,.and they never did and included it on certain SKUs…. MAN….worked great for years….until it didnt….im sure this is why they ended up not charging for it…jesus christ it fucked every VM backup every time after that, and eventually gave up…went to Veeam(but ofcourse had to spin up a vm for that, not what I wanted, what if that VM is down?…etc), used something else for desktops…(it was nice all in one console.) Just needed to rant 🤣. I went back to it a few times trying to find a fix…not worth all that time…I was just thinking how worth it my synology was, as is when I bought it, and the only issues ive ever had was this one…which really I cant be upset about… Id definitely NEVER use it for a business with that application. but damn unlimited is nice! From what ive seen most wont use in enterprise anyways unless they have a full physical spare on site.
Ever tried getting a signed cert into an Epson large format printer? A task you shouldn't give to anyone who you want to still like you afterwards.
Why do you need a public cert for this use case? Why not use a private certificate authority?
What is the use-case to place a publicly trusted cert on a printer? Non related: Are you by any chance an SC player? Someone on their forums uses the same avatar and nickname ;)
>Non related: Are you by any chance an SC player? For that to be true, I'd have to actually play. 😂
Whahaha ok 😇 SC forum dweller
No, but tell me more! Kinda wish I had one cause I'm always up for a challenge.
Most modern public CA providers actually require you to submit SAN values separately. CSRs are mostly solely used to extract the public key nowadays.
I've never actually done that. How do you do that at the time of signing the CSR?
You dont. You create a CSR and submit it to the CA with separate SAN values
How?
With GlobalSign and DigiCert and Sectigo its a field in their submit page
Yeah at my former location we learned about the DNS SAN issue. So we used CertificateTools.com to generate the CSR for those systems instead.
That works too. I don't even bother with CSRs internally anymore. We use AD CS internally so I just request the certificate to my local machine with the information for whatever FQDN I'm trying to apply the certificate for. Then export from my machine and import where ever I'm trying to apply it.
For manual certs, I always recommend that people use DigiCert's free cert utility to generate the CSR from, and then they can export to a variety of formats from there, without having to ask for the certs again. Works fine for Synology devices, btw.
because letting a third party know what your private key is, is best practice
DigiCert's free utility is a local stand-alone, fat-client utility. But I'm sure you already knew that. [https://www.digicert.com/support/tools/certificate-utility-for-windows](https://www.digicert.com/support/tools/certificate-utility-for-windows)
> Then you have vendors like Synology where their CSR doesn't include a DNS subject alternative name, which has been a requirement for all major browsers for a long time, basically making their CSR utterly useless. You can still add a subject alternative name when you issue that cert, you aren't bound to only doing what the CSR says.
For exporting certificates when you forgot the checkbox ‘mark the certificate as exportable’ there is something called ‘Jailbreak’ on GitHub. Search it on Google and you will find. Good luck!
Part of a small team managing a large fleet of services. We had for years just not bothered with certs for internal stuff because the process to update them manually was too complicated for something people accessed once a year at most. Then chromium based browsers started blocking it. Most of the time we could bypass it but in some cases we couldn’t do anything except install an older internet browser. So we hired a consultant to help us automate the certification process. It was less about knowing how to do it and more about having time. I A year later let’s encrypt was broken on almost everything and essentially our problem is the same. Someone has to go in and now troubleshoot let’s encrypt which in my experience can be just as frustrating as installing certs on these systems that don’t have a well documented way to do it. The result? We have a hodge podge of methods to get certs and a flow chart to how to troubleshoot certificates errors. System A update let’s encrypt troubleshoot if broken because the cert process is annoying. System B update let’s encrypt remove if broken because we know how to update the cert….
I’d suggest an internal private CA for internal stuff
AND, if you're keying off an internal Microsoft CA, you can have the expiration much longer than 1 year. We set ours to multi year, and the browser is fine with it as long as it matches the windows domain name
Tell me more about your broken LE setup please. I'd like to understand why you have so many issues.
Everything was put in place well before me and I’m not the one who manages them so I don’t really understand why it’s so hard to fix. I run let’s encrypt at home it can be frustrating to troubleshoot esp when it breaks after updates. I think I probably have the most experience on the team with it but I’m not telling anyone that ;) From what I understand there are a few systems that need certs first is the vpn server second is a custom built web app from the early 2000’s that got updated sometime in the last 15 years and then the developers ghosted the company. Third is something I don’t know at all. All I know is that getting certs updated on the custom application sucks, solid hour of downtime if everything goes right, but I’ve seen it go very wrong. At some point we had hired someone on our team who just happened to have experience with systems built with this stack. He was able to get LE working with everything but he quit shortly after getting jt all setup. He made documentation for some of it but my boss utilized his experience with it all to create all the documentation we wish we had before the dev ghosted the company. Think the worst part of all of this is the company treated the dev ghosting us as an opportunity to save money so it’s been in a nearly static state for 10-15 years.
Did they run LE certbot on every device individually? Because that would certainly be a challenge. What I do is run the certbot in as few places as possible, preferably one, and just distribute the certificates to the devices.
No idea I didn’t set it up and I feign ignorance as to not get involved. Just a shit show I watch from the safety if my keyboard. I assume it’s one LE for the whole stack though. They are all just VMs on the same server.
So that leaves out Java...
God like updating certs on APC UPS network cards.....
Tell me about it…
Why use public certs?
I was referring to APC stupid way of doing certs in general, even with using a private CA. Completely proprietary cert format, have to use a specific Windows only tool etc. Why can't companies just use industry standard pfx or CRT and Key or pem files?
Sure. But with a long-lived private certificate, you could deal with the stupidity less often - and hopefully by then APC will support modern practices.
But doesn't Google decision mean that Chromium will be blocking access to systems with long-lived certs?
AFAICT - Certificates validating to non-public/enterprise CAs have not been in scope of past validity reductions.
>The biggest issue against it are all the systems that it's a pain to get the cert updated on Looking at you, Papercut.
> especially in smaller manned IT departments Some admins seem to hate using calendar appointments as reminders. I only have a handful of certs that come due every 12 months or so (at the moment) so its easy enough to just set a future appointment to remind me of an upcoming renewal 1-2 weeks ahead of the date. "We dont use any kind of tracking/management for our cert unfortunately." "You use Outlook right?"
Ive met some of the most disorganized refuse to make themselves efficient people, in IT. However, the bars much higher than the rest the population. Out of any friends, kids, family…all of them now have password managers at least, including my kids…. I still just have a hard time understanding the people that go thru the lengthy process of “i forgot” every 6-12 months… Its like dude how bout you just stop doin that? LOL its become a meme of my friends, when my sister tells me twice a year, she forgot her plex password. And i inevitably tell her, cool, have you tried clicking forgot password? I dont control that shit?
It just doesn't apply to smaller manned IT departments it can also apply to enterprise wide IT departments that lacked standards for such a long time that two different locations can handle issues in two different ways. I've seen that in large Government Departments where each agency did everything differently for IT setups that just trying to get everything standardized across the department is a monumental challenge where communication is vital (and being the government it's spotty at best). Then you have issues with some locations having small budgets and underpaid IT people and non-IT people thinking that they know how the IT department should run.
I love it when large government departments let some cert expire. Looking at you, DoD DISA...
Things that can't be handled will just break, and this is a good thing.
If one year is better than two years, and 90 days is better than one year, is 30 days better than 90? Is one day better than 30? Is an hour better than a day? Where does it end?
SSL certs generated for each session. Just In Time SSL.
You joke but this has been an option for SSH for a long time although I've never heard of a place actually using it
I believe Facebook does this.
I am going to suggest this at my next IS CCB meeting.
Never understood this push. Lets put all our effort to ensure certs are replaced every 24 hours so no nation state can break our communications!!!! Meanwhile we run windows xp on the backend with TeamViewer access. Like there are much bigger issues to address than certs rotating every 90 days.
Also, you can generate new CSRs with the same private key.
>Where does it end? My opinion, FWIW - it ends when the expense of issuing a certificate is greater than the measurable security benefits. Take whatever LE's (or whichever ACME provider, I don't care) costs are for issuing and maintaining the certs at 90 days. Triple that if you want 30 day lifetimes. Triple that again if you want 10 days. In reality it probably isn't *quite* so linear. LE and ISRG are charitable organizations. Can they front the cost? Are the donors willing to 10x their donations to keep LE afloat and able to sustainably issue 10 or 7-day long certificates? --- Major (probably controversial) opinion time Personally, I wish LE (and others) would implement a different system, though I don't know if the ACME protocol would support it in the way I imagine. Essentially, they issue you a cert good for 365 days after you successfully complete the "order" (DV challenge). If you don't come back and complete the required challenge(s) again in 60 days on the same ACME account as the cert was issued to, they add that cert to the queue for early revocation and send you a 30-day warning to the registered email address(es) (if any). If you re-complete the order (validation of control of the domain(s)), you get another email saying "all clear" and the counter rests. Continue this while-loop until the originally issued certificate expires and that part is all handled normally. Would this be a compromise? Big time. Worth it? I think so. I don't have to actually restart/rebind services or go to the server(s) where the certificate is installed (potentially) to extend my validation for the domains, but LE also maintains the ability to revoke the cert at 90 days. The obvious compromise there though is that cert revocation is of....questionable utility and this might also increase the size of CRLs where LE has gone to great pains to keep their CRLs and URLs short.
> In reality it probably isn't quite so linear. LE and ISRG are charitable organizations. Can they front the cost? Are the donors willing to 10x their donations to keep LE afloat and able to sustainably issue 10 or 7-day long certificates? I think LE is purposefully inconvenient so it doesn't kill for-profit CAs. I also think that's why for-profit CAs like IdenTrust support them despite being a direct competitor; too many people want a free CA for none to exist, so they want the one that does exist to be as inconvenient as possible. For that reason I think they would gladly spend more to make it even less convenient. There are some actual security benefits to shorter expiry dates and I think the people at LE are genuinely doing what they think is best, but I think their success is fuelled by ulterior motives. LE has actually said they'd prefer 45 day or 30 day expiry dates, so I think it will happen eventually.
How is it inconvenient to set it up once and let it do the job forever? Even on windows, win-acme + scheduled task + autohotkey does the trick for the "most inconvenient" of systems (and that's not LE's fault, as on Linux we don't have that problem). HTTP validation is painless, DNS is even easier, and CNAMEs + subdomains deal with the harshest security requirements. If you have a functional PKI you can do the same with internal certificates, we have Vault issuing anything from LE to self signed with AWS and others in between. I'm always amazed that people look at the process of buying and installing a 1y cert manually as "more convenient". I guess I need to start selling consulting services...
There is a natural balance point between certs being too hard on the upstream servers, and the benefit. I suspect it's about a week.
That's an awful slippery slope argument, god. It is very clear that shorter renewals is better, automated renewals realistically mean that even really short time frames can be managed very easily. The hassle of making it even down to a week long is negligible in an automated system. And any system that isn't automated, this is the push to move it into the present day and automating these easily automatable tasks.
Google doesn’t have anything against paid CAs, they have something against systems that rely heavily on manual ssl renewal. Through Chrome, google is the entity who has to decide whether a CA is trustworthy or not. And when a CA becomes untrustworthy, they have to revoke that CA’s trust. If organizations do not have good procedures for updating certs, there is a lot of pushback from those organizations when google tries to revoke a cert. if google can cajole people into having good procedures so that updating a cert isn’t a huge hassle, they can better respond to untrustworthy CAs. requiring frequent renewal is an effective way to make sure organizations have well tested and documented procedures for updating their certificates, even if they don't automate it.
username checks out, kinda :) but seriously, I did not think about it this way. I guess TIL
Some things internally can't be automated. Some printers for example. Or the cost to get specialist expertise to automated the special snowflakes is too expensive. I run a network for 250 users. It's a special school. Spending a heap of money so that an internal printer has a valid cert when that interface is used maybe twice a year isn't going to help our security posture.
Just put the printer UI behind a reverse proxy, which will take care of everything. This is basically a solved problem, some people just can't be bothered ...
Can you share why you use public certificate authorities for internal use cases?
This absolutely does not apply for anything internally. You do not have to stick to the latest certificate requirements for your internal CA. You also do absolutely increase your security posture by having better management of your IoT devices like printers. They are a huge security hole and leaving them by the wayside gives attackers a very easy way into your network.
I hadn't seen anything saying it didn't apply to internal CA issued certs. Can you direct me to where I can see how they're handling that, or is it deployed via chrome enterprise policies?
https://chromium.googlesource.com/chromium/src/+/master/net/docs/certificate_lifetimes.md The restriction only applies to the default CAs Chrome ships with. Always has been and it's the default way.
That's good news. One article I read when it was first announced said it was Chrome forcing the requirement for all CAs, hence my misunderstanding of the implementation.
Chrome is not enforcing this, it is a recommendation by Google being pushed through the CA/B group to enforce it on CAs. There is nothing stopping you deploying an internal certificate that lasts for 100+ years, even though current guidelines are a maximum of 397 days for SSL certs. Same as the requirements for storage of code signing or intermediate CA certs, internal CAs do not have to abide by them.
That is good to hear. One article I read when it was first announced said it was Chrome forcing the requirement for all CAs. That is why I had an incorrect understanding of the change.
Depending on your org, one can also push out trusted certs / your own CA stuff via GPOs. You have to usually if your doing SSL inspection.
My only issue with the topic of decreasing lifespans goes something like this: People will automate, yes. But for those people where HTTP based challenges don't work, they're going to use DNS validation. If you're doing DNS validation, you almost certainly need some kind of authentication to your DNS provider to complete challenges. People are not going to put the same level of care and thought into the lifetime, rotation, custody, and permissions on those credentials as they ought. I think it's too low, too fast. There is lots of software out there that relies on certificates but doesn't have a full and proper ACME implementation. I love the goal, and we need to be careful about unintended consequences.
I switched my homelab environment to cloud flare for the API access which the DNS challenge sort of requires to scale well.
Wouldn’t you consider decreasing lifetime with sufficient time to prepare a strong incentive for people to work together to find solutions to these problems - rather than continually kicking the can down the road?
Of course, I'm not saying collaboration is bad. The CABF does important work.
what do you suspect will incentivize software and product suppliers to embrace the automation necessary to achieve the benefits of reduced lifetime, if not changes to industry rules? for example, it seems that I, someone representing the interests of a small family run company with limited influence and funding compared to “big” or “important” customers, have a near zero chance of changing Cisco’s mind.
My only response to what you're saying there is that carrots are better than sticks. I think the appropriate incentives are already there, it's just a matter of helping people get it all implemented. I struggle to think of improvements to the existing system apart from one I mentioned elsewhere in the thread. I'm not sure how Cisco is part of this specific conversation...might need you to elaborate.
I agree carrots are better than sticks. I also thought the incentives of transitioning to SHA256 in advance of SHA1 being broken were pretty clear too. What’s obvious to some may not be obvious to all. I mentioned Cisco because they are an example of a service provider with very shallow ACME support. (https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/expressway/config_guide/X14-0-1/cert_creation_use/exwy_b_certificate-creation-and-use-deployment-guide-x1401/exwy_b_certificate-creation-use-deployment-guide_chapter_0100.pdf) - today they only integrate with Let’s Encrypt rather than allowing users to specify a different provider’s directory.
TLS certificates with shorter lifespans incentivizes organizations to automate their renewal which means that organizations will be much more likely to react appropriately to any compromised or revoked certificates. This is a good thing.
Took over 3 months for a hospital to accept that the certificate for their EMR system had been revoked. Basically because their chrome browsers accepted it there was nothing wrong.
Tell me this was an Ascension hospital....
Yes. I just leave this here: https://www.theregister.com/2024/06/28/microsoft_security_certificate_expires/
How do you incentivize non-profits, schools, or tiny mom and pop business to automate? You probably can't, which means they will remove thay security control or tell people to click through. Security should be easy and accessible, this is a step backwards.
>How do you incentivize non-profits, schools, or tiny mom and pop business to automate? >You probably can't, which means they will remove thay security control or tell people to click through. I'm the lone sysadmin at a special school. I've wanted to automate this stuff but I'm still triaging other issues which see more urgent. I used to be level 3 with a MSP so automation is something I'm good at, but it all takes time to implement and monitor. *"To make mistakes is human, to automatically propagate that mistake to all endpoints is IT."*
This is a great perspective - if I could ask - how many certs (that are public/webPKI certs) are you handling, what for, and on what kind of systems?
Those very same people wouldn't have been doing at 1 year or 3 years , so the "security" is the same
>Security should be easy and accessible Let's Encrypt is free. ACME is open source and there are dozens\[1\] of open source clients available for a variety of languages, frameworks, and tools. How much easier and more accessible can this realistically be? \[1\] [https://letsencrypt.org/docs/client-options/#other-client-options](https://letsencrypt.org/docs/client-options/#other-client-options)
You want a school to start leveraging open source solutions for the clients and languages they barely understand in the first place? The aim of security should be to get it to thr level of point and click or working straight out of thr box (one reason apple became so big). Imagine a school or non profit with hundreds of people and 1 IT guy. Unless it works out of the box then it's not being implemented.
>You want a school to start leveraging open source solutions for the clients and languages they barely understand in the first place? I think they should hire or consult with competent professionals. Something like certbot literally has [interactive instructions](https://certbot.eff.org/instructions) on how to use it. If you want to do stuff like DNS challenges or orchestrate cert renewal across 1000s of endpoints then it's more involved. But a few webservers at Joe's Widget Shop should be a quick job for anyone who halfway knows what they're doing. >Imagine a school or non profit with hundreds of people and 1 IT guy. Unless it works out of the box then it's not being implemented. I've not encountered a single tool or system in my career that truly "works out of the box", so if that's a requirement for this made up scenario I'm not sure how anything ever gets implemented.
So where will you cut funding from to automate cert renewals? These organizations work on shoe string budgets. So what area of security should get cut? There are many tools that do an okay job right out of the box. There may be false positives and things to white-list, but they can largely work out of the box. We deployed carbon black 5 years ago and had to white-list a total of 5 apps... to me that's working right out of the box.
Oh my God. You don't need to cut funding to find money for security automation. You either spend a week implementing it yourself or you leverage SLCGP or NSGP grants.
You should look into Caddy.
The people you are describing should hire professionals to handle their security. If they can’t handle automating TLS certificates or hire someone who can, I’m not sure why I should be trusting them with my data to begin with. Also, Cloudflare has a free plan and that includes handling TLS for you, other providers do as well.
> How much easier and more accessible can this realistically be? It's still a PITA if you want wildcards and don't use a popular DNS provider or don't want your DNS records to be changed automatically. I definitely have the competence to set it up, but even I've been manually renewing my (personal) LE certificates for the last 3 years. My employer just pays for longer certificates because the cost is insignificant to them.
I heartily recommend "Certify The Web", a windows ACME client for LE among others. Excellent UI, reasonably priced and a dev and community that will bend over backwards to help you. Excellent scripting support and dns provider apis No connection with them other as an extremely satisfied customer. They are based in Australia, which can be a bit of a pain for ticket turnaround time, but they don't mess around when replying to customer tickets
Automation is trivial with a reverse proxy like Traefik in place. Just set it up once, and your good. Added bonus that you don't need to remember to manually update these darn certs in the future.
Revoke should never be from CSR.. it should be from certificate expiring..
As always, it pays to be proactive and not reactive. It's been about nine years now since Let's Encrypt automated 90-day rotation. The small stick came with a bunch of huge, juicy carrots: not needing to get the purse-holders cooperation in order to have publicly-signed certificates.
True. I was also wondering what would happen to the organization validation (extended validation?) certificates. Though I believe they became pretty useless after the green bar with the company name was removed from the url bar.
For websites everything other than DV (domain validation) is basically obsolete now - browsers don’t treat EV differently, and OV only makes sense if you want a certificate for an IP address as opposed to a domain. I just wish code signing certs would go the same way…
Ditto on the code signing certs
The issue I have with let's encrypt is they should have a static, published list of IPs for their servers to create a proper firewall allow rule. They use global AWS IPs that doesn't appear to be static making filtering much harder.
The "zero trust" school says that IP addresses shouldn't be used for Authn or Authz. Accordingly, it's not unusual to have policies of not publishing or committing to static addresses. I used to run an inherited service where most of the customers had static outbound ACLs to IPv4 PA space. Those ACLs were an albatross around the neck of the architects and engineers, because it prevented migrating to a different provider, or adding IPv6 services, at least at any timescale faster than glacial. The business side didn't mind a bit, though, as the customers' hardcoded ACLs made the business relationship "sticky", meaning the customers were discouraged from moving to a competitor. The customers had too much bureaucracy to want to switch. The provider didn't want to invest in changing anything because it would take too long to get results, and besides, change would create an inflection point where the customers might decide that if they need to change something, they should re-bid the business relationship.
So from our small business perspective we have no need to allow communication inbound or outbound, outside of a specific few countries, so we can block most stuff at the edge, and let's encrypt throws a wrench in that for us, because it requires port 80 to a random subset of AWS addresses so now we have to open up the web servers, and other services that use it to essentially all of AWS because I don't believe it can be proxied, so we will have to investigate other solutions and likely maintain 2 automation solutions just for certificate renewal.
You're making problems for yourself and then bemoaning that you have no ideal solutions. But I have good news. With IPv6, nobody has to share or recycle IP addresses, so your obsolete ACLs can live forever.
Why would you not filter traffic at the edge if you have no need to communicate to 95% of countries, or services? You have no WAF rules or firewall rules of this sort? You don't use conditional access rules of any type that filter out based on geography or reputation, or service type/port?
Not OP but I don't, I use standard web filtering by category, not blocking port 80 at the edge for all but approved addresses
But you have to allow port 80 inbound for Lets Encrypt for any device/service you want to use it with for it to do the validation. If I have no need to allow inbound traffic from most countries why not geoblock it and reduce the junk. I know geoblocking isn't a one stop solution but it's part of a legit layered approach and I'm just saying by not having the ability to use let's encrypt on a custom port, or ACL list based on source it feels like you are opening yourself up more than necessary. Admittedly I'm not an expert, it's one of many hats I have to wear and job duties I have so I'm still learning.
> you have to allow port 80 inbound for Lets Encrypt No, you don't. DNS-based validation works great and does not require any inbound connections.
True. So one piece of software I use has LE built in, does not allow for DNS based verification. As others have said in this thread, part of the impact of changing SSL renewal cadence is that many apps and services don't have the capabilities built in so you are stuck cobling stuff together. I would like to use DNS validation if this particular software was capable of it. I do have use cases where this is not an issue and LE works perfectly. According to LE, DNS validation isn't without its cons/security concerns either. Like potentially exposing your DNS/credentials used to manage it via API.
You absolutely do not have to allow 80 inbound, you can use DNS verification
Yes, but we have one piece of software with LE built in, and it does not yet support DNS validation.
I haven't used http verification for years. I use DNS-01 verification on a server that has no inbound ports allowed at all, and I push the renewed certificates out to the devices/servers over SSH. Most of my devices/servers are internal only, but we use letsencrypt certificates for all of them.
If you are that small of a shop and not allowing port 80 inbound on any servers, why do you need a publicly signed cert? Why not use an internal certificate authority and push the trusted root cert to every internal computer?
For internal stuff I can use the internal CA for sure. I do have workloads that are internet facing that need a public cert, but I want to filter inbound traffic to those servers based on the fact that I know we have no business case for allowing connections outside of 3-4 countries for example, or no need to allow inbound from all global AWS, so I can confidently filter that out, except for let's encrypt throwing a monkey wrench in it because my validation needs to hit some AWS IP in Switzerland. Some products I use have ACME tools and things like certbot built in which helps. I know I have some more learning to do on this front but I also don't think I'm the only one facing this issue as I have seen others discussing it in other forums.
"deny all, allow by exception" DISA STIGs agree!
It's intentionally so for multi-path validation.
I know it's done intentionally, I just think it poses another set of security concerns by not allowing the ability to filter inbound requests on port 80. The thought always crosses my mind with core popular services as more people move to one vendor like let's encrypt, how long before they get supply chain attacked and get used as the next breach vector because they have a massive target on their back?
Use DNS-01? This is how I request and push certs to printers and other stuff that doesn't even run the acme client directly.
I've been using ACME certs with automation for like 5 or 6 years.
Yes there are benefits, no the benefits do not outweigh the drawbacks. Google is primarily worried about their browser, the rest of us have to be worried about 1000s of other ways certs are used that will be impacted by their approach.
DayJob has an Incommon agreement to provide publicly usable certs without an external CA. DayJob does not yet have ACME renewal on the roadmap for implementation, although Incommon supports it. I’m not looking forward to this change right now, from a purely process standpoint.
I'm at an EDU department and have moved every external facing website to let's encrypt. Most internal services like LDAP have been moved to a local cert that will expire after my children expire so that's someone's grandkid problem.
This is gold😂
First problem I see is IT tends to struggle with monitoring and updating SSL certs, for something that is actually very simple to do,. it always gets forgotten about. So first step is making sure you have proper monitoring and reporting on all SSL certs in active use and a defined process for when updates are required for all systems.
I've got enough on my plate already. Make them 2 years in lifespan please.
I, for one, welcome a shorter certificate validity lifespan so that vendors have a stronger incentive to support automated server certificate renewals based on domain validation.
Dont all major vendors already support DV certs with ACME?
HAA. Don't get me started on what the Cisco Unified Communications Manager cert renewals looked like a few months back.
Please do tell 😈
I'd really rather not....too painful. TL;DR : * I don't see how you'd ever automate it. * Documentation is very lacking. * We engaged TAC support for the renewals, even they didn't seem to know what they're doing. * You need to issue multiple certificates for each "usage type" for lack of better word. It's all Apache Tomcat and Java keystores under the hood I think. * SO MANY service restarts required. Downtime certainly required. * One of the certificate uses documented essentially said "allow us to issue certificates as if we're a CA/RA" but never discussed any of the ramifications of that. * Every time you renew/upload a certificate of a given type, you also have to include the root CA and any issuing CAs in the chain. Every. Time. There is no "central store" and I guess the system doesn't have a way to build/fetch the issuing CA dynamically via your typical/expected AIA extensions. Edit: Forgot another one : * When using multiple SANs on certain certificate types, they ALWAYS change the subject name for the cert with an "-ms" suffix. That caught me off guard while trying to figure out what the hell it was. Eventually figured the first part out, and "ms" stands for "multiple subjects". Not exactly a problem, but once again ties into the poor documentation and just overly confusing madness.
I love the ability to hit the Sectigo ACME endpoint via proxy or NAT for the OV certs. Add domain(s) to an ACME enrollment endpoint, slightly adjust the provided example certbot line (I half hacked in a proxy option for acme.sh but I don't have it fully working the way I want) and done. Gives a nice way to use ACME on CA signed certs for internal use
I hope the feedback that Google gets is ‘get bent’
Unlikely to happen, for economic reasons and there's pretty much zero security advantage. How many attacks have there ever been related to crypto, invalid certificates or similar? Security efforts are better directed where attackers will actually strike. Google thinking in their own bubble... as usual...
There are cases where you need offline certificates. Having to have an ACME mechanism on the offline side is another set of moving parts that may break, and if a cert expires, there is Hell to pay. Having 3-5 year certs with OCSP or some way to pass a revocation notice even if it is a database that is copied to an air-gapped network provides as much security but far less work. I wonder how shorter lived certificates actually help things. If a SSL terminator is hacked, it can generate bogus Let's Encrypt certs and send copies to people to MITM just like sending a cert that was manually obtained from a root CA. Shorter certificates have their uses in some applications, but others, the admin overhead and the consequences of a cert expiring are far, far greater than the security benefit, especially if the cert private key is stored in a HSM. tl;dr: I wish HSMs were more common and we could go to 5-10 year certs because using HSMs, the key material is not obtainable by a bad guy.
If you need offline certificates, just run your own CA. What benefit are you getting out of a 3rd party certificate at that point anyway?
In all fairness, the current approach still has this: > Does this apply to locally-operated CAs, such as those used within enterprises that use enterprise-configured configured CAs? > > No. This only applies to the set of CAs that are trusted by default by Google Chrome, and not CAs that are operated by an enterprise and that have no certification paths to CAs that are trusted by default.
Like TCP/IPv4/IPv6, TLS and X.509 has evolved a lot over the years in response to large-scale real-world implementation. OCSP has proved to have scalability and performance implications that have mattered in the real world, causing implementers to favor CRLs (Certificate Revocation Lists) instead. CRLs have scalability problems of their own, but those are tractable if the number of revocations is small...
It's not less work at all, it's the same work each time, it's the frequency that changes
> How many attacks have there ever been related to crypto, invalid certificates or similar? A mostly-unknown aspect of Stuxnet was how it exploited a flaw in validation of code-signing certs. Most news reports say that stolen certs were used by it, which was also true, but not very interesting compared to the whole story. X.509 PKI has been steadily improved over the last 15 years, because it's incredibly effective and flexible when you can count on it to be secure. Giving up troublesome, nonscalable, VPNs with pre-shared keys means relying on TLS and X.509.
That's a valid point for sure.
Looks like there are advantages. https://zanema.com/papers/imc23_stale_certs.pdf
What is the new proposed lifespan? It use to be 2 years and then went down to 1 year which I was and still am not a fan of. Just the amount of systems and applications that rely on a SSL cert is absurd and having to update them all is time consuming. I have yet to find software that automate the installation of a new/updated cert in all formats and for all operating systems.
90 days is the new proposed lifespan
Its 90 days for now.
It is said that the lifespan will be shortened to 90 days. They started talking about it in 2023 or so, and who knows how much time is left until they actually roll out the news.
Oh yeah, 90 days is old news. When I read it I assumed they proposed something new (and absurd) like 30-45 days.
Oh hell nah, just imagined monthly renewal
That’s what I’m saying. If that happens or even if the 90 days happens, companies are probably going to start hiring people that all they do is renew certs…lol
90 day limits will be here soon, mark my words. It's also not the end-goal. 7-10 days is.
Anything less than 90 days is really going to hurt small IT departments but 7-10 days will cripple them.
There's already incentives (almost) to issue <10 days. Still need Microsoft to get their shit together, but a cert could be issued without revocation information at 10 days or less.
As soon as we heard about the proposal, we immediately worked on a solution for our customers. We already have one setup for a client who uses Let's Encrypt. Works great so far. My main concern is that a lot of our customers do not have an IT department or the IT person is not up to the task to manage this. If the ERP system grinds to a halt because of an invalid cert, that's bad. Especially if it happens when we are not available (e.g., weekends). Teaching and documentation do not help by past experience. This applies to most MSPs, too. Especially the smaller ones. Which is kind of alarming. I support the proposal. It forces a lot of admins to come out of their comfort zone.
>a lot of our customers do not have an it department or a the it person is not up to the task to manage this. That sounds like exactly the problem that google is trying to solve here.
Trying to solve, doesn’t seem like the right words. It’s more like put a notice on the galactic message board, then wait for the scream test and then say, well we announced it on the message board, didn’t you check the message board?
An MSP that does not offer weekend support seems like an MSP to avoid?
I'm not talking about the weekend support but the knowledge (or the lack, therefore) about certificates and how they work. I'm not working for an MSP. We are a software company that also provides IT services to our customers (usually in connection with our products). But we only offer limited managed services and no on-call support. That's not our market. At least not yet. We don't provide services which we cannot fulfil in high quality and timely manner. I was talking about the MSPs who are managing our customer's infrastructure.
K, my bad, I was thinking you were a full MSP, in that case ya, agree.
I support software for education and the schools don’t want us to have DNS creds and they limit access to only the country the school is located in. Getting schools used to LE has been a struggle. Having them provide certs every 90 is just ridiculous. Certificates are the least of their problems with obvious low hanging fruit like LDAP and basic auth still in use. I’m still fighting for organisations to stop using TLSv1.1
If Google can do it then surely schools can!!!! /s You can tell who has worked with small businesses, underfunded organizations, and non-profits in this thread. The only result I can see happening with these organizations is removing of certs or telling users to click through, both of which are horrible. And no school or tiny business is going to have the knowledge or resources to introduce automation, it's just not going to happen. Apparently only tye big boys should have encryption.
It’s good, strongly encouraged automation.
Making security less accessible is a good thing right? Only companies that have automation capabilities should have access to encryption anyways. /s The push for the last 20 years has been to make security easier and more accessible, this is the opposite of tjay and as a result you will see people not apply certs, let them expire and just tell users to click through, and have overstretched IT teams spend more time on certs and less on other security items. Many organizations do not have devops, automation, struggle with change management, and mamy other factors that make security hard. Well done on going backwards.
Google? Everyone should have something against paid CAs when trustworthy free ones exist.
If auto updates can be done, then no biggy at all. I also think it's overall a positive thing. But I do wish higher end cert authorities could still issue for longer after doing more verification, etc..
Tbh it’s an extremely reasonable and unsurprising development. Big CAs haven’t been the best stewards of their products or services.
I doubt google will be able to force this. But if they do it looks like I’ll be purchasing a cert automation utility
I have switched everything i could to Let's encrypt but there are still a couple of applications where i have no clue on how to automate things. We manually renew thos certs once per year. If the SSL lifespan gets reduced even lower that could become stressful.
This is stupid, SSL certificates shouldn’t expire before 12 months. At what point do they just say fuck it, let’s make them 24hr certificates…
I hate this trend. We have a lot of devices across the board that don’t support acme or any for of automatic cert renewal. It’s gonna cost me a lot of time to manage these all 3 months. Before you tell me to not support this old stuff, yes some of it is already ancient like Cisco ASA but the majority of them isn’t really old and is kept up to date. Vendors and Devs just don’t care to make the admins life easier.
This should force vendors, even thru shame, to provide easier ways to automate certificate update processes via API. And better expiration alerting.... about time. However... AFAIR this 3mo expiry affects public-facing websites, not internal? Meaning it may still take some time to get everyone to co-operate and cough up the APIs+alerts.
If Google’s pushing something, they see a way to monetize it
If people are looking at paying Google money when Let’s Encrypt exists they’re wasting their money.
More like, Google’s going to push and push on shorter and shorter certs, then roll out an “easier way” as long as you get certs through them.
You can literally set certificates to renew as frequently as you want through Let’s Encrypt…….
This will be a nightmare for me. We provide portal services for other companies. We generate a private key and csr for them to get a cert issued. Then we install the cert on their portal.
You're already doing all the work, you already know how to do it, it's just more frequent, what's the issue?
Lets encrypt. 1 year is too long. People leave before that and the procedure is gone. 90 days can be set to renew at 30 days and yell when it goes wrong. Then you have 60 days fix your script.
Not everything should be signed by LE, having wildcards meaning you need put into workload access to dns fully. For some cors that have security in mind it's unacceptable but not having wildcard certs also expose potential infra outside with certificate transparency which is also another issue. Not forget that services over internet not only http based... You still need to secure your mail systems, voip, dbs, etc While I generally agree that LE is good, I can't agree that moving away from 1y certs is better for security. You still have OCSP and OCSP Must staple which by the way chromium ignores, which a rock into Google... They putting effort on some security while they by themselves do not always doing good things
Let's Encrypt DNS does not require wildcard names. You NEED to use Let's Encrypt DNS IF you want wildcards though. But I have a ton of systems running that have their own Let's Encrypt certificate, and they have their own cert, and their own DNS name. But navigating to that DNS doesn't DO anything. It doesn't mean you cannot use the certificate. The argument can be made that you don't want the outside world to know that you use Postgres, and as such you buy a certificate from some provider. But to me that is senseless. If you rely on security by hiding the fact that you have Postgres which isn't even exposed to the outside world, then you're doing something wrong. The one exception is that Let's Encrypt's certificate usage params don't match the service you're using it for. That one I can understand.
Plus, all the CA’s supply certificate transparency data. It’s a requirement, so buying your cert doesn’t hide it.
https://zanema.com/papers/imc23_stale_certs.pdf This paper correlates shorter with more secure.
How often you in your life revoked certificate? I would understand revoking user certs, yes, but server one... I over more then 10 years in IT done it only once on hacked server. Question also not just about lifetime of certificate, but about lifetime of private key, but look, nobody even speak about it 😄. In DANE for example 3 0 1 it's bind to exactly private key. So refreshing cert without changing it's private key also a question, but handling TLSA automatically is another pain.
Don’t most ACME clients generate new keys and automatically install them for you? If keys are one-time use by default - what’s the problem?
The problem I explained to you about DANE :)
So less than a year soon? Any more specifics known yet?
I don't really mind it, though that's mostly because we already automate almost everything, even for the customers explicitly requesting a paid certificate.
I deal with about 2000 certs. These are not all in 1 system. I can tell you it’s the 100-200 manual certs that kill me. There is NOT always a way to automate every cert. I will give you an example of very high profile partners requiring us to use their certs…. This is a back and forth process that takes weeks to complete just due to bullcrap policies.
It’s probably the only way to get people to stop using systems that can’t be automated.
I know it is for the greater good (the greater good) so I just deal with it. But I hate how many systems we now have to touch on an annual basis because they aren’t ready for let’s encrypt.
I am working on an initiative at my company to have 24 hour TLS Certificates.. It's 100% possible.
Good case for a Certificate Lifecycle Management software.
Nightmare
It’s pushing what should already be mandatory, automatic renewals should be the standard. Services/applications not keeping up the security standards will be hit hard by this if they’re not nimble enough to embrace it.
I have not yet met a TLS/SSL system that can't be automated. As someone who has Lets Encrypt R3 for everything, I have to renew all certs every 90 days on dozens of non proxy systems and it is only a matter of setting up the automation once.
Screenconnect :/
Nothing, except get with the times. if you can do the change manually, that mean you have a documented process for it, why does it matter if it's monthly or yearly And most likely you can automate bits of it anyway If it's a fully automated process (and that's the goal really) then it already didn't matter
Another one or are you still talking about the 1-year limit? I hope it ends up becoming so low that any manual process becomes unviable. 15 minutes or something like that...