"Billing alerts" are a joke, give us hard spend limits. Then offer a way to set those limits during onboarding.
Building a business on blank cheques and accidental spends is shady. It's also a large barrier to adoption. The more times devs see reports like, "I tried [random 20-minute tutorial] and woke up to a bill for my life's savings and luckily support waived the fee this one time but next time they're coming for my house", the less they'll want to explore your offerings.
I know it’s minor in comparison, but I will never use AWS again after running up a $100 bill trying to get an app deployed to ECS. There was an error (on my side) preventing the service from starting up, but cloud waatch only had logs about 20% of the time, so I had to redeploy five times just to get some logs, make changes, redeploy five more times, etc. They charged me for every single failed deploy.
After about two days of struggling and a $100 bill, I said fuck it, deleted my account and deployed to DigitalOcean’s app platform instead, where it also failed to deploy (the error was with my app), but I had logs, every time. I fixed it in and had it running in under ten minutes, total bill was a few cents.
I swore that day that I would never again use AWS for anything when given a choice, and would never recommend it.
This seems like a glaring bug in the scripts run by that `npx` command. The author is correct, the scripts should 100%:
- Choose the lowest cost resource (it's a tutorial!)
- Cleanup resources when the `delete` subscript is run
I don't think it's fair to expect developers to do paranoid sweeps of their entire AWS account looking for rogue resources after running something like this.
If a startup had this behavior would you shrug and say "this happens, you just have to be paranoid"? Why is AWS held to a different standard by some?
I've been putting off digging into AWS for years now, and it's because of stories like these. There really should be a standardized training course that requires no credit card info and lets people experiment for free.
Instead they have some pencil pushers calculating that they can milk thousands here and there from "user mistakes" that can't be easily disputed, if at all. I'm sure I'm not the only person who's been deterred from their environment due to the rational fear of waking up to massive charges.
Par for the course for AWS. I tried following their quickstart Sagemaker guide to run Llama 2 a few months back. And it certainly spins up quick, but next day I realize it's running me $400/day.
I was able to get the charges reversed, but definitely learned not to trust their guides.
needed to send "raw" http requests instead of using their bloated sdk for reasons, and requests failed with "content-type: application/json" header, but succeeded with "content-type: application/x-amz-json-1.0". get out of here with that nonsense.
Every official AWS guide is designed to make you use as many AWS services as possible, which increases the risk of spend. You have to be extremely critical of anything they recommend (GUI defaults, CLI tools, guides, recommended architectures etc).
There's a reason there are very well paid positions in companies to guide colleagues on how to use AWS cost-effectively and with lower risk.
I'm on the AWS Amplify team and wanted to give folks an update. First off, definitely empathize with the pain that Elliot went through. The referenced blog post is part of our advanced extensibility documentation, which covers how customers can use AWS CDK to add features that are not directly supported by the Amplify tooling, such as integrating with OpenSearch. Our initial OpenSearch extensibility documentation did not include the removalPolicy config, which led to the issues Elliot experienced. To mitigate this, we updated our documentation to include `removalPolicy: RemovalPolicy.DESTROY` for all stateful extensibility resources, ensuring they are cleaned up when the stack is deleted. Additionally, we will be updating the default behavior for `npx ampx sandbox` and `npx ampx pipeline-deploy` to apply this removal policy.
> I’ll admit that I myself am only using OpenSearch because it supports geo_point bounding-box queries, a subject that I don’t have a full understanding of. Perhaps there is a way to do these with a simpler product, and OpenSearch is overkill.
> Even if you are not using Amplify/OpenSearch, I recommend getting familiar with AWS budgets.
> It’s so difficult to be paranoid about every single technology you use.
I do not know what others feel but with this kind of frictionless setup, plus low intuitivity in the UX/UI of those services, people are not concerned about setting up a credit card, and billing bundling between services (e.g. AWS batch + Lambda + EC2) is part of the business model.
I do not know how to articulate it, but it's more or less like those modern amusement parks where you pay to enter the facility, and for every attraction and even the toilet you pay to go.
One thing I find people don't talk enough about is how bad AWS documentation is, across the services. They contain outdated information about services, inconsistencies for the same feature, and lack explanation and meaningful examples to help you understand how things work and work together, even for common workflows. Example: try to make Sagemaker use EMR for Spark analysis, as a beginner who has little idea what IAM is and how to grant permissions for different services. It will be fun.
I am just amazed that people are able to navigate the services and configure them properly.
It is sad experience but I disagree with the author here
> It’s so difficult to be paranoid about every single technology you use.
I would be paranoid with anything related to AWS, I don't want to risk my bankruptcy (or near bankruptcy experience) on small mistakes or the goodwill of the AWS support.
And here I'm concerned about the 30€/mo I pay for running my (quite fat) home server... at least the metal can't randomly expand and consume 10kW all of a sudden.
If the original author reads this comment, I have a question.
One of the problems highlighted was that the documented teardown procedure did not properly delete the OpenSearch domain. Would AWS Nuke (https://github.com/ekristen/aws-nuke) correctly destroy everything that the tutorial sets up?
I remember my first usage of S3 close to its initial launch. It was the first time I came across the very concept of pay-as-you-go. I remember being anxious about unexpected charges and the forum was full of people begging for a cost cap feature.
That was 18(!) years ago. It's still nowhere to be found.
There's like 17 ways to do analysis, some of them paid, but none address the actual problem of capping a bill. It's pure malice.
I set up a private account on Azure to host a small, static website using 100% FREE components. However, it wasn't possible to do this without registering a credit card. Even when I tried to add a different card from Revolut, which is a prepaid card, I received a message stating that such cards are not accepted and that it has to be an actual CREDIT card.
Warning: As an aside, this setup creates mid-price r5.large.search OpenSearch instances by default. Nowhere in the boilerplate code or guide is this mentioned. That will run you $134 per month at minimum.
A good rule when working with any sort of cloud service: Everything that can be charged for, will be.
There are plenty of stories of people getting charged massively, and one may wonder whether this has any negative effects on them getting new customers. Unfortunately it's usually not the ones working with it who are the ones making the decision to use AWS or other cloud services, and the ones who are have their minds fully clouded by the propaganda --- I mean marketing.
AWS has good base building blocks (ALB, EC2, Fargate, RDS, IAM etc). But it takes knowledge to put the pieces together. Thus AWS tries to create services/tools that orchestrate the base blocks (Amplify, Beanstalk) for you, which in my experience always becomes a mess where you don't actually understand what you are running in your cloud setup.
I'd recommend either learning the basic building blocks (these skills also transfers well to other clouds and self hosting) or using a higher level service provider than AWS (Vercel etc) - they do it better than AWS.
Cloud computing is ridiculously expensive. The other day I wanted to increase Mongo Atlas iops by 1000 on a 3 node cluster and it costs $3000 a year. How does that make any sense?
The repeating problem with cloud is that it no longer stops my bad code by OOM, rather the sky is the limit for both the memory and the cost. I heard funny story about people who bought badly written code which was constantly going up through the roof with resources. They got angry with my friend, told him the software is not bad, but the hosting is (software was bought with hardware from the manufacturer) and migrated to cloud, only to meet the hard iron hammer of karma (which in this case was embodied by high bill)
Also in my opinion billing is the new perf test but post factum and obscure, i.e. it is super easy to miss some key points in the development and then wake up with the costs falling down the responsibility sink (https://news.ycombinator.com/item?id=41891694)
Some student / beginner trying to learn and then getting smoked by a footgun is a weekly occurrence on the big cloud subreddits.
It's better business to have people beg for mercy and then magnanimously waive fee than to have any discussion about actual hard limits (which would be used by big corps too not just students).
Yes it can be done technically - Azure already has a not loudly advertised account type that is hardcapped. And no billing alerts aren't a solution. Hell you could even do opt-in "yes I understand my data will be deleted" hardcaps.
This is a fixable problem - they just don't want to because a fix would be bad for earnings.
While learning the AWS Python Boto3 library I once accidental subscribed to the $3,000 a month AWS Shield Advanced DDOS protection service and THAT was stressful. But I did get a full waiver.
> It’s so difficult to be paranoid about every single technology you use. When using new technologies that promise to speed up the developer flow, I already expect them to be more expensive than bare metal, but I think this is beyond the pale.
It's always been. They are always pushing boundaries and checking what they can get away with. The response "we’ve processed a billing adjustment for the unexpected charges as a one time courtesy." even though it looks like a bug and it hasn't been fixed since is already telling.
I think anyone doing anything with anything AWS amplify will get burned sooner or later. It seems purpose designed to be easy to set up, and absolutely nothing else.
It is unfortunate that most of these services don't have a pre pay tier.
On the one hand I get that if your business depends on such a service you don't want it to suddenly go down. But on the other hand there is almost never a hard mechanism to limit your risk. Or if there is, it is opt-in. The conspiracist in me says this is working exactly as planned for AWS as they have no financial incentive to limit customer risk.
As someone new to cloud services, I'm curious are there better experiences with the billing of GCP, Azure or Oracle Cloud? Also, is the multi-cloud approach doable?
oh god , what a timing , I had actually created a blog post on why I didn't want to host a hobby project with credit card because I didn't want to even risk a 0.01% chance of me getting an insane bill and here aws's official guide is causing such things.
After handling so many of these cases I decided to build a solution that helps with cost monitoring and optimization. It's a single click integration with AWS and Azure.
We're currently working on a solution for these specific cases as well, would love to hear some feedback.
CloudExpat - www.cloudexpat.com
While I don't like Amazon one but, I appreciate that you don't go into the Evil Big Tech trope. Things at that scale are indeed complicated and hard to coordinate.
That said: fuck, that's expensive and poorly explained! Not doing anything cloud without hard limits!
The story of "you won't need highly paid technical experts to maintain things" is fake.
The story of "it's cheaper" is fake.
The story of "you can't run your own computers it's too complex for ordinary companies to work out" is fake.
Its all fake and people still are diving headlong into the clouds, falling through and hitting the earth hard.
There's enough discussion in the community about the risks and hazards of major clouds - you only have yourself to blame when that huge bill hits because you did some thing that would have not cost an extra cent on self hosted systems or virtual servers.
Go learn Linux. Go buy virtual servers from IONOS where they charge zero for traffic.
This is the reason I use AWS wrappers (render.com/fly.io) for small projects.
It may be more expensive but you can't pop the free tier/selected machine.
billing surprises is among the top 5 reasons I keep a homelab for experimenting. if your project can't be deployed to non-cloud infra, I likely won't be using it in the future.
All this pay-as-you-go BS is such a scam (note: I'm not blaming victims here.) There's just so many ways to screw you. Like a program gets bug and starts sending out a metric crap load of traffic. Or you get DDoSed and charged for the bandwidth. Or your disk fills up from log file spam, it gets 'backed up', you get charged for it. Or a program is caught in a loop and uses limited CPU usage.
I think its all hiding the fact that people don't want to take the time to design (and maintain) scalable infrastructure and instead rely on fake abstractions that pretend to be infinite, always-available, magic, or w/e. I'm sure there is some open source software that helps here.
I remember checking with and calling an amazon rep on the phone - he assured me I could use one of the "heavier" graphic card instances and would only be charged per minute usage at the rates shown.
I ran it for 1 minute expecting to be paying the $5 or whatever it was per minute and was charged around $100 for it to "boot up". Cancelled it. Never trusted amazon billing again = (
Bezos keeps waxing lyrical in all his interviews on how he "tests" his company services by calling them on the phone to make sure the SLAs or whatever are accurate. But they aren't. TBH I was kind of confused how proud he was that it took 10 mins to get through to someone on the phone instead of 1 minute or something, on how they noticed it and had to "rearrange" things. Like wtf, I would have fired ALL of my executive below me if such an egregious false advertising existed.
It can't be that bloody hard as one of the richest people on the planet to just pay some dude $5 an hour to make sure services are billed as expected, run as expected.
I am sorry to complain, I know they have all done great jobs, but it makes me wonder whether I would be "out of touch" if I was ever in a C-suite role. From what I see around me, I definately would be. But maybe those margins don't matter?
I honestly am confused after decades in IT why management is never held responsible. If I ran a company mgmt would be the FIRST to be fired if there were any issues. I swear I read a comment on HN once from a manager saying why should they be held responsible if there is a fuck up lower down the chain, I was like wtf, the whole point of being a manager is to be RESPONSIBLE. Management isnt a luxury "earn the big bucks" because Im better than everyone else, and thus should be protected.
The easiest way to diagnose this as a CEO is to see how often MGMT have been let go at different tiers, and if there havent been any, well there must be a form of corruption / nepotism occuring.
Been burnt as a "small" entrepeneur by all of the greats, Google ( shutting down my instance when it went viral because I decided to upgrade the hosting which for some unknown reason mean it had to be shutdown WITHOUT warning for 24 hours, possibly to transfer it or something god knows. AWS etc.
I know it might seem like a small gripe, but as a millionaire now I remember how I was treated by these companies.
Maybe I should just be grateful I could use them at all.
I think I'm just saying its crazy how the "low" b2b customer is treated when it would be so cheap to just make sure these collosal fuck ups don't happen.
I Followed the Official AWS Amplify Guide and Was Charged $1,100
(elliott-king.github.io)435 points by thunderbong 19 hours ago | 271 comments
Comments
Building a business on blank cheques and accidental spends is shady. It's also a large barrier to adoption. The more times devs see reports like, "I tried [random 20-minute tutorial] and woke up to a bill for my life's savings and luckily support waived the fee this one time but next time they're coming for my house", the less they'll want to explore your offerings.
After about two days of struggling and a $100 bill, I said fuck it, deleted my account and deployed to DigitalOcean’s app platform instead, where it also failed to deploy (the error was with my app), but I had logs, every time. I fixed it in and had it running in under ten minutes, total bill was a few cents.
I swore that day that I would never again use AWS for anything when given a choice, and would never recommend it.
- Choose the lowest cost resource (it's a tutorial!)
- Cleanup resources when the `delete` subscript is run
I don't think it's fair to expect developers to do paranoid sweeps of their entire AWS account looking for rogue resources after running something like this.
If a startup had this behavior would you shrug and say "this happens, you just have to be paranoid"? Why is AWS held to a different standard by some?
Instead they have some pencil pushers calculating that they can milk thousands here and there from "user mistakes" that can't be easily disputed, if at all. I'm sure I'm not the only person who's been deterred from their environment due to the rational fear of waking up to massive charges.
I was able to get the charges reversed, but definitely learned not to trust their guides.
needed to send "raw" http requests instead of using their bloated sdk for reasons, and requests failed with "content-type: application/json" header, but succeeded with "content-type: application/x-amz-json-1.0". get out of here with that nonsense.
There's a reason there are very well paid positions in companies to guide colleagues on how to use AWS cost-effectively and with lower risk.
How about postgres with postgis? https://postgis.net/docs/using_postgis_query.html
I do not know what others feel but with this kind of frictionless setup, plus low intuitivity in the UX/UI of those services, people are not concerned about setting up a credit card, and billing bundling between services (e.g. AWS batch + Lambda + EC2) is part of the business model.
I do not know how to articulate it, but it's more or less like those modern amusement parks where you pay to enter the facility, and for every attraction and even the toilet you pay to go.
I am just amazed that people are able to navigate the services and configure them properly.
> It’s so difficult to be paranoid about every single technology you use.
I would be paranoid with anything related to AWS, I don't want to risk my bankruptcy (or near bankruptcy experience) on small mistakes or the goodwill of the AWS support.
One of the problems highlighted was that the documented teardown procedure did not properly delete the OpenSearch domain. Would AWS Nuke (https://github.com/ekristen/aws-nuke) correctly destroy everything that the tutorial sets up?
That was 18(!) years ago. It's still nowhere to be found.
There's like 17 ways to do analysis, some of them paid, but none address the actual problem of capping a bill. It's pure malice.
A good rule when working with any sort of cloud service: Everything that can be charged for, will be.
There are plenty of stories of people getting charged massively, and one may wonder whether this has any negative effects on them getting new customers. Unfortunately it's usually not the ones working with it who are the ones making the decision to use AWS or other cloud services, and the ones who are have their minds fully clouded by the propaganda --- I mean marketing.
I'd recommend either learning the basic building blocks (these skills also transfers well to other clouds and self hosting) or using a higher level service provider than AWS (Vercel etc) - they do it better than AWS.
Also in my opinion billing is the new perf test but post factum and obscure, i.e. it is super easy to miss some key points in the development and then wake up with the costs falling down the responsibility sink (https://news.ycombinator.com/item?id=41891694)
It's better business to have people beg for mercy and then magnanimously waive fee than to have any discussion about actual hard limits (which would be used by big corps too not just students).
Yes it can be done technically - Azure already has a not loudly advertised account type that is hardcapped. And no billing alerts aren't a solution. Hell you could even do opt-in "yes I understand my data will be deleted" hardcaps.
This is a fixable problem - they just don't want to because a fix would be bad for earnings.
It's always been. They are always pushing boundaries and checking what they can get away with. The response "we’ve processed a billing adjustment for the unexpected charges as a one time courtesy." even though it looks like a bug and it hasn't been fixed since is already telling.
--> https://files.rombouts.email/IMG_0092.jpeg
On the one hand I get that if your business depends on such a service you don't want it to suddenly go down. But on the other hand there is almost never a hard mechanism to limit your risk. Or if there is, it is opt-in. The conspiracist in me says this is working exactly as planned for AWS as they have no financial incentive to limit customer risk.
That said: fuck, that's expensive and poorly explained! Not doing anything cloud without hard limits!
I just don't get it.
The story of "it's easier" is fake.
The story of "you won't need highly paid technical experts to maintain things" is fake.
The story of "it's cheaper" is fake.
The story of "you can't run your own computers it's too complex for ordinary companies to work out" is fake.
Its all fake and people still are diving headlong into the clouds, falling through and hitting the earth hard.
There's enough discussion in the community about the risks and hazards of major clouds - you only have yourself to blame when that huge bill hits because you did some thing that would have not cost an extra cent on self hosted systems or virtual servers.
Go learn Linux. Go buy virtual servers from IONOS where they charge zero for traffic.
billing surprises is among the top 5 reasons I keep a homelab for experimenting. if your project can't be deployed to non-cloud infra, I likely won't be using it in the future.
I think its all hiding the fact that people don't want to take the time to design (and maintain) scalable infrastructure and instead rely on fake abstractions that pretend to be infinite, always-available, magic, or w/e. I'm sure there is some open source software that helps here.
I ran it for 1 minute expecting to be paying the $5 or whatever it was per minute and was charged around $100 for it to "boot up". Cancelled it. Never trusted amazon billing again = (
Bezos keeps waxing lyrical in all his interviews on how he "tests" his company services by calling them on the phone to make sure the SLAs or whatever are accurate. But they aren't. TBH I was kind of confused how proud he was that it took 10 mins to get through to someone on the phone instead of 1 minute or something, on how they noticed it and had to "rearrange" things. Like wtf, I would have fired ALL of my executive below me if such an egregious false advertising existed. It can't be that bloody hard as one of the richest people on the planet to just pay some dude $5 an hour to make sure services are billed as expected, run as expected.
I am sorry to complain, I know they have all done great jobs, but it makes me wonder whether I would be "out of touch" if I was ever in a C-suite role. From what I see around me, I definately would be. But maybe those margins don't matter?
I honestly am confused after decades in IT why management is never held responsible. If I ran a company mgmt would be the FIRST to be fired if there were any issues. I swear I read a comment on HN once from a manager saying why should they be held responsible if there is a fuck up lower down the chain, I was like wtf, the whole point of being a manager is to be RESPONSIBLE. Management isnt a luxury "earn the big bucks" because Im better than everyone else, and thus should be protected.
The easiest way to diagnose this as a CEO is to see how often MGMT have been let go at different tiers, and if there havent been any, well there must be a form of corruption / nepotism occuring.
Been burnt as a "small" entrepeneur by all of the greats, Google ( shutting down my instance when it went viral because I decided to upgrade the hosting which for some unknown reason mean it had to be shutdown WITHOUT warning for 24 hours, possibly to transfer it or something god knows. AWS etc.
I know it might seem like a small gripe, but as a millionaire now I remember how I was treated by these companies.
Maybe I should just be grateful I could use them at all.
I think I'm just saying its crazy how the "low" b2b customer is treated when it would be so cheap to just make sure these collosal fuck ups don't happen.
Best run company in the world /s