Needless to say AWS should really reconsider their approach here... š¤¦āāļø https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1
Imagine you create an empty, private AWS S3 bucket in a region of your preference. What will your AWS bill be the next morning? A few weeks ago, I began working on the PoC of a document indexing system for my client. I created a single S3 bucket in the eu-west-1 region and uploaded some files there for testing. Two days later, I checked my AWS billing page, primarily to make sure that what I was doing was well within the free-tier limits. Apparently, it wasnāt. My bill was over 1300$, with the billing console showing nearly 100,000,000 S3 PUT requests executed within just one day! Where were these requests coming from? By default, AWS doesnāt log requests executed against your S3 buckets. However, such logs can be enabled using AWS CloudTrail or S3 Server Access Logging. After enabling CloudTrail logs, I immediately observed thousands of write requests originating from multiple accounts. But why would some third party accounts bombard my S3 bucket with unauthorised requests? Was it some kind of DDoS-like attack against my account? Against AWS? As it turns out, one of the popular open-source tools had a default configuration to store their backups in S3. And, as a placeholder for a bucket name, they usedā¦ the same name that I used for my bucket. This meant that every deployment of this tool with default configuration values attempted to store its backups in my S3 bucket! Note: Unfortunately, I canāt disclose the name of the tool Iām referring to, as doing so would put the impacted companies at risk (as explained further). So, a horde of unauthorized third parties is attempting to store their data in my private S3 bucket. But why should I be the one paying for this mistake? Hereās why: S3 charges you for unauthorized incoming requests This was confirmed in my exchange with AWS support. As they wrote: Yes, S3 charges for unauthorized requests (4xx) as well[1]. Thatās expected behavior. So, if I were to open my terminal now and type: aws s3 cp ./file.txt s3://your-bucket-name/random_key I would receive an AccessDenied error, but you would be the one to pay for that request. Another question was bugging me: why was over half of my bill coming from the us-east-1 region? I didnāt have a single bucket there! The answer to that is that the S3 requests without a specified region default to us-east-1 and are redirected as needed. And you pay extra for that redirected request. The security aspect We now understand why my S3 bucket was bombarded with millions of requests and why I ended up with a huge S3 bill. At that point, I had one more idea I wanted to explore. If all those misconfigured systems were attempting to back up their data into my S3 bucket, what if I opened it for public writes? I made my bucket public for less than 30 seconds, and within that timeframe, collected over 10GB of data. Of course, I canāt disclose whose data it was. But it left me amazed at how an innocent configuration oversight could lead to a dangerous data leak! What did I learn from all this? Lesson 1: Anyone who knows the name of any of your S3 buckets can ramp up your AWS bill as they like. Other than deleting the bucket, thereās nothing you can do to prevent it. You canāt protect your bucket with services like CloudFront or WAF when itās being accessed directly through the S3 API. Standard S3 PUT requests are priced at just $0.005 per 1,000 requests, but a single machine can easily execute thousands of such requests per second. Lesson 2: Adding a random suffix to your bucket names can enhance security. This practice reduces vulnerability to misconfigured systems or intentional attacks. At least avoid using short and common names for your S3 buckets. Lesson 3: When executing a lot of requests to S3, make sure to explicitly specify the AWS region. This way you will avoid additional costs of S3 API redirects. Aftermath: I reported my findings to the maintainers of the vulnerable open-source tool. They promptly fixed the default configuration, although they canāt fix the existing deployments. Additionally, I notified the AWS security team. I hoped they might restrict the unfortunate S3 bucket name, but they were unwilling to address misconfigurations of third-party products. I reported the issue to two companies whose data I found in my bucket. They did not respond to my emails, possibly considering them as spam. AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception. Thank you for taking the time to read my post. I hope it will help you steer clear of unexpected AWS charges!
Very interesting. Shouldnāt the VIP layer or CloudFront throttle a certain number of requests per sec to same bucket? Was there 429 throttling? Also, from my work in AWS Lambda team 3 years ago, there used to be ip based request limit and throttling also.
Interesting thanks for the insight
Pasted the text down there but format is horrible.
"Note: Unfortunately, I canāt disclose the name of the tool Iām referring to, as doing so would put the impacted companies at risk (as explained further)." Name and shame.
That's why you buy pure storage products instead of relying on aws
As a former AWS employee, I agree. Don't use AWS.
Sick exploit
Can a put (from random person) even access s3 bucket without role/policy in place ? Looks like this person gave full access to the bucket through internet.
Wrong, read again
lol, why donāt you just tell me why this is wrong ? I wrote this after hours of struggling with S3, I was unable to upload files due to missing perms.
Security group rules, sweaty.
This went completely neglected/underlooked due to the $0.0004 per 1000 requests cost? So about 100K requests to see a $40 in the bill. For most companies thatās again negligible costā¦ but aws keeps ripping their customers ($40 from over 1000s of users) And also the free tier that kept new non-enterprise users away from these charges.
Cant read due to paywall How is metering logic here? Even if a bucket is empty, does the cost happen when any url related to the bucket is hit? As in is empty bucket fetch also considered as a full metering?
We r dum we charge for 4xx lol
^^^ that! Also, no paywall for me... try another browser?