User got a $104K bill from hosting provider(Netlify): “I thought it was a joke”
User got a $104K bill from hosting provider(Netlify): “I thought it was a joke”
"Netlify CEO here.
Our support team has reached out to the user from the thread to let them know they're not getting charged for this.
It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
Apologies that this didn't come through in the initial support reply."
This was posted 4 days ago in hackernews.
And that's all he posted. I think he responed to one of two comments and then ignored everything else. I really dislike that 'CEO here', 'I'm important, listen to me' means that always ends up on HN. Then they disappear like their mere response to the post is enough. After all, they're very important people.
If he thought 'damaged control done' he was sadly mistaken.
I host my site on Netlify. I'm moving. If they think that it's acceptable to bill somebody $104k for a small site, at all, without it tripping some alarm for a human to look at before it goes out, then they're doing it wrong. Something that says 'Month 1 bill = $20; Month 2 bill = $104,000' could be a problem isn't difficult to do. And that they have 'done this often' (my words) highlights it's a problem.
There are many bullshit hosting companies out there I can use who don't do this sort of thing. Why is Netlify special.
What's the questions-to-answers ratio to make it acceptable to write "CEO here" in a topic, in your opinion?
How much and what would he need to post so damage control would have been done (and successfull) in your eyes?
Just asking in case I become a CEO and see someone having a solvable issue with my company's sercice.
I find it astonishing that Netlify had no safety mechanism in place to prevent this.
Saddling customers with unbounded liability is irresponsible; arguably negligent.
Definitely negligent, I still remember the young adult who killed himself when he thought his Robinhood account was negative nearly 3 quarters of a million dollars.
Oh jesus, THAT’S why there’s a million hoops to jump through before they even give you access to LOOK at options trading now. I always just figured someone lost a bunch of money and sued, that’s so grim.
“After looking into this, it seems you have a hit song on your site,” the email from Netlify customer support reads. “Maan Bou Jan Sang Lou by Teresa Tang. I was not aware of her, but she seems to be a popular Taiwanese singer. This song is 99% of your bandwidth usage over the past 30 days.”
The letter further explained that a lot of bandwidth was generated from user agents that “are quite ancient using Google Cloud addresses”.
“This would include devices such as circa 2010 iPads, Windows 98 & Windows 6 computers. So either you have a fanbase with a passion for older technology, or this was likely a DDoS attack. To me, this seems to be the latter,” the email continued and suggested hosting such files on third-party platforms, such as YouTube or SoundCloud.
After explaining the standard practice of reducing the bill to 20% after such attacks, which would be $20,900 in this case, the Netlify support team offered a better deal.
“I've currently reduced it to about 5%, which is $5,225. I know this is still a lot of money, and I apologize for the inconvenience. If you like, I can raise this internally to see what else can be done.”
The user wasn’t happy with that and decided not to pay but post their story on Reddit and Hacker News instead.
“Since the user opened a ticket with us this past Sunday, we’ve been actively researching this situation. Initially, we thought it might have resulted from a DDoS attack, which we stated in our first response. After some investigating, it looks as though the spike in traffic was not caused by a DDoS after all,” Dorian Kendal, CMO at Netlify, told Cybernews.
Instead, now they believe that this was a sustained download event of an mp3 file over a stretch of multiple days.
“We’re working directly with the user to better understand what’s happening on their end, so we can uncover what caused the dramatic increase in downloads,” Kendal said.
I'm confused, what is this supposed to mean? Some sort of non-distributed DOS attack? How would working with the customer help there? If they're susceptible to a denial of service, isn't that entirely an internal problem?
No service was denied!
Fair point. DOS is perhaps the wrong word for it. But from that quote, it sounds like it's a similar behaviour to DOS tactics which involve finding ways to transform a relatively simple request into a large amount of work (or in this case, network traffic) for the server.
They are saying that it wasn't a ddos at all but organic use. The user was notified but did nothing. So they think their notifying stuff isn't good enough.
Sorry, but what exactly is a "sustained download event" supposed to be? It sounds like they're describing some sort of DOS-like attack that isn't a DDOS, where a user manages to force the server to serve up way more data over a sustained period of time than would be reasonable for downloading a single MP3 for normal use.
But maybe that's not what they mean. It's very unclear.
I am too. Is the agreement to charge per mb downloaded? Do they not have some sort of "turn it off if I hit this max?* feature?
I usually avoid hosting solutions like this just because of this shit. I wanna know how much I'll owe before the month starts even. Anything else feels like gambling.
Of course they do but they can make 104k if they don't turn it on.
There are plenty of bandwidth restricted hosting sites out there. Sounds like that is what you want. Maximum speed regardless if that's used 24/7 or not. If more users request your site than that bandwidth allows - oh well.
Makes you wonder how many customers were wrongly charged some other less insane amount, and no one noticed because it wasn't jaw dropping.
The most expensive mp3 of his life..
You wouldn't download a $104K hosting bill...
"Instead, now they believe that this was a sustained download event of an mp3 file over a stretch of multiple days."
Apparently the same mp3 downloaded/uploaded over and over again.
The most expensive mp3 of his life.
I use Netlify to host my frontend projects and portfolio. Does anyone have a way to prevent something like this?
Not use a hosting provider that charges by the amount of traffic?
This appears to be an extreme edge case but overall there is nothing preventing you from waking up to such a huge bill if your site turns into the most popular page on the internet over night.
I didn't even think commercial host providers would do this.
The only service I knew about that had limit to transferred amount of data was grex.org, a non-commercial public unix shell. It had limit of 10MB/day for your web page, but it also didn't allow stuff like images.
However, that wasn't anything commercial. And I think before the shutdown it was just a single computer sitting in someone's basement.
Not that it helps but the CEO claims they forgive for this type of attack/event. https://news.ycombinator.com/item?id=39521986
Netlify CEO here.
Our support team has reached out to the user from the thread to let them know they're not getting charged for this.
It's currently our policy to not shut down free sites during traffic spikes that doesn't match attack patterns, but instead forgiving any bills from legitimate mistakes after the fact.
Apologies that this didn't come through in the initial support reply.
And later they were asked if they would have responded if it didn't go viral. https://news.ycombinator.com/item?id=39522029
Question:
There are only two questions everyone have:
-
Would Netlify forgive the bill if this didn't go viral?
-
How do you plan to address this issue so that it never happens again?
Everyone here knew someone from Netlify would come and say OP wouldn't have to pay. That was a given. Now we want to know the important answers.
Answer by CEO:
-
Yes. We've forgiven lots and lots of bills over the last 9 years and they haven't gone viral
-
While I've always favored erring towards keeping people's sites up we are currently working on changing the default behavior to never let free sites incur overages
-
You can put the site behind cloudflare for DDOS protection. Unfortunately, it's not good for user privacy and it will make the site difficult to access over VPNs, proxies, and TOR.
Netlifiy is very expensive for bandwidth and the free bandwidth can be exceeded very quickly. I would look for something with a hard bandwidth cap. Then your site will just go offline if the bandwidth is exceeded.
Unfortunately, it’s not good for user privacy and it will make the site difficult to access over VPNs, proxies, and TOR.
Difficult, but not impossible (unless the site owner also goes and futher implements additional measures like ASN blocking for known proxies/VPNs/etc), just solve a captcha and you should be on your way pretty much.
You should take a look at GitHub Pages
I recommend hosting your projects on Cloudflare Pages, as it is a free service provider to the best of my knowledge.
I actually really respect their policy. Keep the site active and then forgive stupid bills if there was an error.
To shut down or disconnect a cloud service is terrible as usually it's in error. The errs on the side of the user knowing their stuff better than the hoster which is what I want in a provider.
That "policy" was to reduce his bill from 104k to 5k initially. It was not "forgiven" until his story went viral.
So their actual policy was to send a user paying $0/month a bill for 5k for malicious behavior they didn't cause on the site. Thats not something to respect.
The CEO playing at "no no, we always forgive these" was not at all what he was told until after thousands if not hundreds of thousands of their potential customers saw how screwed he was by their non existent tools to rate limit bandwidth.
A 9yr old hosting company not having any programtic tools to limit biling is an intentional choice, not some oversight. It's clearly their buisness model to have people go over the free tier unintentionally, just not unintentionally enough to go viral.
Is it possible to avoid that hotliking of a file? in this case was a heavy mp3 file, but it easly could be a heavy image or a video.
The email he got from support suggested hosting the mp3 on a third party site like youtube or soundcloud, which is a good idea I reckon. Youtube would work for videos, and I guess imgur or imgbb for images?
Not ideal, but all free!
“After looking into this, it seems you have a hit song on your site,” the email from Netlify customer support reads. “Maan Bou Jan Sang Lou by Teresa Tang. I was not aware of her, but she seems to be a popular Taiwanese singer. This song is 99% of your bandwidth usage over the past 30 days.”
The letter further explained that a lot of bandwidth was generated from user agents that “are quite ancient using Google Cloud addresses”.
“This would include devices such as circa 2010 iPads, Windows 98 & Windows 6 computers. So either you have a fanbase with a passion for older technology, or this was likely a DDoS attack. To me, this seems to be the latter,” the email continued and suggested hosting such files on third-party platforms, such as YouTube or SoundCloud.
After explaining the standard practice of reducing the bill to 20% after such attacks, which would be $20,900 in this case, the Netlify support team offered a better deal.
“I've currently reduced it to about 5%, which is $5,225. I know this is still a lot of money, and I apologize for the inconvenience. If you like, I can raise this internally to see what else can be done.”
The user wasn’t happy with that and decided not to pay but post their story on Reddit and Hacker News instead.
One user on Hacker News with the alias ‘bobfunk’ introduced himself as the Netlify CEO and assured users that the bill would be forgiven. Cybernews was unable to verify the CEO’s identity independently. However, many previous posts from the same user and his bio support the claim of him being Matt Biilmann, the founder of Netlify.
In another twist, the DDoS attack version of the story is being ruled out
“Since the user opened a ticket with us this past Sunday, we’ve been actively researching this situation. Initially, we thought it might have resulted from a DDoS attack, which we stated in our first response. After some investigating, it looks as though the spike in traffic was not caused by a DDoS after all,” Dorian Kendal, CMO at Netlify, told Cybernews.
Instead, now they believe that this was a sustained download event of an mp3 file over a stretch of multiple days.
“We’re working directly with the user to better understand what’s happening on their end, so we can uncover what caused the dramatic increase in downloads,” Kendal said.
“We’ve confirmed that the user was notified multiple times about the additional bandwidth that was being consumed on their site, but given their lack of response to these notifications, we believe that we should revisit and improve the messaging and urgency that’s being communicated.”
I'm kind of impressed by the amount of research they did to figure out why this guy's bill was so high, then immediately offered a resolution, and then immediately offered another avenue if the resolution wasn't good enough. Shout out to the customer service rep.
Just because I know most won't actually click on the article
In a quirky response, the company’s customer support team reduced the bill to $5,225. And when the story started trending online, the CEO decided that the user wouldn’t be charged at all.
