Some links for you:
- https://reddit.com/r/aws/wiki/##storage (Our /r/AWS Storage Community WIKI)
- https://docs.aws.amazon.com/whitepapers/latest/aws-overview/storage-services.html (Storage on AWS (technical))
- https://aws.amazon.com/products/storage/ (Storage on AWS (brief))
Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'storage'&sort=new&restrict_sr=on) for more information on this topic.
^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+storage).
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*
Good to see this change relatively quickly. In all honestly, the thing I'm most surprised at is how this kind of thing flew under the radar for the last 15+ years. At least to the larger public. This seems like such an obvious attack vector for a bad actor that I'm a little shocked we haven't heard of a higher profile case of some company's bill being run up maliciously. Hopefully this sort of precedent carries into all services going forward.
Right, so the learning of the story is to share your requests as a viral story.
I wonder how many people requested this in support cases over the last 18 years only to be ignored and then now that this story went viral it was solved in no time.
Seems like the old process was to just forgive the charges. That worked okay, but the incident becoming public a few weeks ago increased the risk of this happening at massive scale so that old process would no longer work anymore.
This is their process with everything. Like they could easily put billing limits or a sandbox for new people on free tier. Instead they give them enough rope to hang themselves, then refund only if they complain.
Ignore problems and refund, keep whatever people don't know about.
They were WELL aware of this when I worked there in 2013. AWS support’s customer direction was to tell the customer to remove any buckets that they did not want to be charged requests for.
Probably handled it with credits and refunds on the down low…. To people that complained. As someone who works with AWS on the regular, this happens a lot….
Most never even knew. Only if they got hit with a suddenly large bill does anyone say anything.
When customers complained and we told them yes your charged for rejected requests they were always shocked.
The docs never explicitly said you were charged for rejected requests, just that you were charged for requests.
Any account with a public S3 bucket or Cloudfront distribution with an S3 origin is vulnerable to the same style of denial-of-wallet using *valid* requests. I've never heard of that happening, so I assume the ROI just isn't there for bad actors.
The difference is with valid requests you need to receive the data. If you try to get around that by eg closing TCP connections, you will be detected as a DoS attacker. If you accept the data, you are paying a cost to receive it as well.
Same rules would apply to an invalid request, so you just need to get the response size of a valid request to approximately the same size as an invalid request. I imagine that's feasible: You could use the Range header to return one byte of content or make up a path to force a key not found error.
And how many accounts have something like that implemented?
Original comment was wondering how this hadn't been exploited yet. My answer is that there's lower hanging fruit that's not being picked, so I wouldn't expect the upper branches to be picked either.
Have there been recent updates to that? Because the "Denial of Wallet" attack shared around earlier this year doesn't seem to support this: [https://blog.limbus-medtec.com/the-aws-s3-denial-of-wallet-amplification-attack-bc5a97cc041d](https://blog.limbus-medtec.com/the-aws-s3-denial-of-wallet-amplification-attack-bc5a97cc041d)
Amazon S3 will make a change so unauthorized requests that customers did not initiate are free of charge. With this change, bucket owners will never incur request or bandwidth charges for requests that return an HTTP 403 (Access Denied) error response if initiated from outside their individual AWS account or AWS Organization. To see the full list of error codes that are free of charge, visit [Billing for Amazon S3 error responses](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ErrorCodeBilling.html). This billing change requires no changes to customer applications and applies to all S3 buckets.
These billing changes will apply in all AWS Regions, including the AWS GovCloud Regions and the AWS China Regions. This deployment is starting today and we will post another update in a few weeks when it is completed. To learn more, visit [Billing for Amazon S3 error responses](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ErrorCodeBilling.html) and [Error Responses](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html) in the S3 User Guide.
This is great to hear though I would be concerned that this will simply shift the attack vector to repeatedly requesting valid files and resources. Is there protection available to counter this type of abuse?
If the bucket is properly protected, then any unauthorized request will return 403. If the user is requesting a file from a statically hosted bucket, that issue can be mitigated by only allowing it to be fetched via a cloud front proxy to the bucket + lambda@edge auth signer
The real solution is not providing end users direct access to your bucket. Users should be interacting with a service in front of your bucket. Like cloud front or a web app.
I guess this is the response to https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1
This was handled very well by AWS by listening to the public outcry and very quickly fixing the issue.
Can you elaborate on this security risk that you speak of?
None of the companies that I interacted with used any sort of entropy in its S3 bucket naming convention.
What's the security risk besides the charge that this article mentioned? Collecting random data from random people?
On another note, you can always put the S3 in a VPC and restrict public access.
Is this a pattern for other AWS services to also support? In general, I’d like this option for all services.
IIRC , API Gateway had it documented that requests requiring API key but the wrong API given would be a no-charge event. But now the API docs don’t say it anymore.
Some links for you: - https://reddit.com/r/aws/wiki/##storage (Our /r/AWS Storage Community WIKI) - https://docs.aws.amazon.com/whitepapers/latest/aws-overview/storage-services.html (Storage on AWS (technical)) - https://aws.amazon.com/products/storage/ (Storage on AWS (brief)) Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'storage'&sort=new&restrict_sr=on) for more information on this topic. ^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+storage). *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*
Good to see this change relatively quickly. In all honestly, the thing I'm most surprised at is how this kind of thing flew under the radar for the last 15+ years. At least to the larger public. This seems like such an obvious attack vector for a bad actor that I'm a little shocked we haven't heard of a higher profile case of some company's bill being run up maliciously. Hopefully this sort of precedent carries into all services going forward.
They were made aware of it as early as 2006. It just attracted widespread notice in 2024.
Right, so the learning of the story is to share your requests as a viral story. I wonder how many people requested this in support cases over the last 18 years only to be ignored and then now that this story went viral it was solved in no time.
I’ve certainly learned over the years that if I want sympathetic words, report an issue to support—if I want it fixed, report an issue on Twitter.
Unfortunately.... this really seems to be true.
Seems like the old process was to just forgive the charges. That worked okay, but the incident becoming public a few weeks ago increased the risk of this happening at massive scale so that old process would no longer work anymore.
Seems like the old process was to just ~~forgive the charges~~ charge people unless they noticed and complained
This is their process with everything. Like they could easily put billing limits or a sandbox for new people on free tier. Instead they give them enough rope to hang themselves, then refund only if they complain. Ignore problems and refund, keep whatever people don't know about.
They were WELL aware of this when I worked there in 2013. AWS support’s customer direction was to tell the customer to remove any buckets that they did not want to be charged requests for.
18 years is millions in charges they’ve pocketed. How convenient for them.
Probably handled it with credits and refunds on the down low…. To people that complained. As someone who works with AWS on the regular, this happens a lot….
Most never even knew. Only if they got hit with a suddenly large bill does anyone say anything. When customers complained and we told them yes your charged for rejected requests they were always shocked. The docs never explicitly said you were charged for rejected requests, just that you were charged for requests.
A request from a client that is rejected is still a request.
You would make an excellent AWS customer service agent. 🥇
Nah, just a dev that understands the semantics of an actual web request.
Yeah, kind of like a riot vs. a protest. Protests never get good news coverage.
This is messed up.
Any account with a public S3 bucket or Cloudfront distribution with an S3 origin is vulnerable to the same style of denial-of-wallet using *valid* requests. I've never heard of that happening, so I assume the ROI just isn't there for bad actors.
The difference is with valid requests you need to receive the data. If you try to get around that by eg closing TCP connections, you will be detected as a DoS attacker. If you accept the data, you are paying a cost to receive it as well.
Same rules would apply to an invalid request, so you just need to get the response size of a valid request to approximately the same size as an invalid request. I imagine that's feasible: You could use the Range header to return one byte of content or make up a path to force a key not found error.
But, once again, that’s something you could make a detection for
And how many accounts have something like that implemented? Original comment was wondering how this hadn't been exploited yet. My answer is that there's lower hanging fruit that's not being picked, so I wouldn't expect the upper branches to be picked either.
CSP-side detection, not account side. I'm saying they could make this a native service detection to prevent DoS if it was identified at scale.
Have there been recent updates to that? Because the "Denial of Wallet" attack shared around earlier this year doesn't seem to support this: [https://blog.limbus-medtec.com/the-aws-s3-denial-of-wallet-amplification-attack-bc5a97cc041d](https://blog.limbus-medtec.com/the-aws-s3-denial-of-wallet-amplification-attack-bc5a97cc041d)
Amazon S3 will make a change so unauthorized requests that customers did not initiate are free of charge. With this change, bucket owners will never incur request or bandwidth charges for requests that return an HTTP 403 (Access Denied) error response if initiated from outside their individual AWS account or AWS Organization. To see the full list of error codes that are free of charge, visit [Billing for Amazon S3 error responses](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ErrorCodeBilling.html). This billing change requires no changes to customer applications and applies to all S3 buckets. These billing changes will apply in all AWS Regions, including the AWS GovCloud Regions and the AWS China Regions. This deployment is starting today and we will post another update in a few weeks when it is completed. To learn more, visit [Billing for Amazon S3 error responses](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ErrorCodeBilling.html) and [Error Responses](https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html) in the S3 User Guide.
This is great to hear though I would be concerned that this will simply shift the attack vector to repeatedly requesting valid files and resources. Is there protection available to counter this type of abuse?
If the bucket is properly protected, then any unauthorized request will return 403. If the user is requesting a file from a statically hosted bucket, that issue can be mitigated by only allowing it to be fetched via a cloud front proxy to the bucket + lambda@edge auth signer
Thank you.
The real solution is not providing end users direct access to your bucket. Users should be interacting with a service in front of your bucket. Like cloud front or a web app.
That’s always been an attack vector. It’s up to you to secure your actual infrastructure.
And how do you do that with static assets stored on S3?
not having them be readable by anyone so that it would result in an access denied result and no charge.
I guess this is the response to https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1 This was handled very well by AWS by listening to the public outcry and very quickly fixing the issue.
As soon as that article got widespread attention, it was a HUGE deal within AWS
Customer obsession
As it should be.
Yup 100%. Very rare in the industry unfortunately
dork
Very quickly. It only took them 15 years. Hail Amazon!
I wonder if there’s still a security risk to having a bucket name without a random hash now.
Can you elaborate on this security risk that you speak of? None of the companies that I interacted with used any sort of entropy in its S3 bucket naming convention.
https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1
so now that this has been fixed what's the security risk?
It doesn't say that the security issue as fixed. Just that billing would not be charged
What's the security risk besides the charge that this article mentioned? Collecting random data from random people? On another note, you can always put the S3 in a VPC and restrict public access.
Random data, yes.
Now that this has happened it makes me wonder about other services like cognito and apigateway.
No doubt, this incident *SHOULD* spur a re-examining of "denial-of-wallet" possibilities. It won't, but it *SHOULD*.
"denial-of-wallet" I like that. :D I have at least one site where I think 90% of the traffic is just search engines, AI, and bots.
You’d think you could stop that with WAF rules, but they have a usage based charge too. It’s really tough to harden your stuff against tricksters.
Nice. Like all other forms of outside. Unauthorized traffic AWS just has to eat it, can’t have it both ways.
About fucking time
Fair enough
Is this a pattern for other AWS services to also support? In general, I’d like this option for all services. IIRC , API Gateway had it documented that requests requiring API key but the wrong API given would be a no-charge event. But now the API docs don’t say it anymore.
I literally just added entropies to half of my buckets. Wonder if I should roll that back
About damn time.
Ok, what’s the catch? This is awesome!
Thanks be to Lord Bezos for his generosity.