
How not to get a $30k bill from Firebase - squidrings
https://medium.com/@PurpleGreenLemon/how-not-to-get-a-30k-bill-from-firebase-37a6cb3abaca
======
paulgb
I really wish cloud providers would allow users to set a hard budget that just
stops the service if you exceed a threshold. I got a surprise bill from AWS
this month (fortunately orders of magnitude less than this, but still ~15x my
usual) and am thinking of moving from Lambda to a VPS just so the possibility
of this doesn't keep me up at night.

~~~
stickfigure
You can? I have daily limits for my Google App Engine apps.

~~~
EugeneOZ
Don't forget they are only warnings, they will not shut down your app and
their update rate is not as frequent as you might expect.

~~~
asciimike
App Engine PM: the GAE Daily Spend limits are hard caps that will shut down
services when hit. They are different from the broader GCP Billing Alerts
which are simply notifications.

The issue is that they only apply to certain GAE services (compute, legacy
APIs, etc.) and not across the platform.

~~~
EugeneOZ
Thank you for correction!

------
paxys
Billing problems aside, it's amazing that web development has reached a stage
where it's almost _too_ easy to scale a site to 2m active sessions and 20m+
page views in two days with just off-the-shelf tools and barely any
specialized skills.

~~~
cryptica
Except for very simple use cases, serverless platforms in general encourage
bad software design patterns.

For example, in this case, if a regular database was used instead of Firebase,
"counting the number of supporters" would have been done inside the database
using a query. The worst that the developer could have done is forget to index
to the relevant column; performance would have been sub-par but still orders
of magnitude better than Firebase because the bulk of the data would not leave
the database if there was a proper query language. With Firebase, when doing
complex calculations, not only does the data have to leave the database, it
often reaches end users; the calculations then happen on the front end and
this is a huge problem in terms security, performance and even correctness
(due to latency and the fact that other users may be making conflicting
calculations simultaneously based on slightly outdated data).

Also, based on the article's description of the application, it sounds like it
may have exposed potentially sensitive data (about supporters) to all users so
it may have introduced a security vulnerability. I would not be surprised if
this was the case.

~~~
bmelton
> With Firebase, when doing complex calculations, not only does the data have
> to leave the database, it often reaches end users

Any time sensitive data reaches untrusted users it is a bad practice, but this
needn't be the case on Firebase just because it is serverless.

Firebase cloud functions can be triggered as a response to document saves (or
authentications, or a plethora of other things) and run outside userspace, and
are available for exactly this sort of work -- keeping the minutiae of upkeep
away from the client.

That said, I agree that it's easy to make this kind of mistake with serverless
development, but mostly due to lack of existing domain knowledge. It's
trivially easy to make similar (or worse) mistakes without a serverless
environment by untrained developers, too. It's just a matter of becoming
familiar with the toolset, and because serverless tech is newer, fewer are as
familiar with them as they are the old things (and there are fewer people to
catch those mistakes when they see them.)

------
intricatedetail
That's why I don't use cloud offerings. Their pricing models are designed to
trap someone with a exorbitant bill and you don't have control over it. For
example let's say you put an image on a CDN and someone who doesn't like you
runs AB for a couple of days making billions of requests to bankrupt you or
someone finds a page that runs costly queries and sets up curl against it. No
thanks.

~~~
StreamBright
Interesting, I saved roughly 2M USD / year for customers moving them to the
cloud. Your example is silly, all of the CDN I set up has very strict limit on
how much a single IP can use and have multiple alerts if you are passing 100,
200, 500, etc. USD limits. On the top of that if that is not enough you can
add more limitations to avoid that exact scenario that a public resource can
be abused to cause you financial troubles. It won't "bankrupt" you if you do
it right. Just like pretty much every other technology, you need to know it.

The other problem with your comment is that you try to make it sound like it
was a single dimension decision to use the cloud. It is never a single
dimension questions though.

~~~
BartBoch
Not taking sides here, but I think that the cloud is complicated enough
(especially for a person that does not specialize in it), to miss one of the
edge cases that can lead to the huge bill.

In a few minutes, I can set simple PHP script with curl, that will launch 100
requests each second, using a pool of hundreds of thousands of IP's thank to
the rotating VPN.

This is an edge case of course, but it can happen.

I use cloud myself, but only cloud servers, which allows me to control budget
better and provides me with an "escape plan", where I can just switch to
dedicated quickly.

~~~
cc81
There is a difference between "cloud" and serverless though. I bet most people
are not using something that can autoscale infinite by default.

------
laken
I have to mention this every time I see it... The crowdfunding campaign
mentioned in this article took place in "Colombia" not "Columbia." There is no
country by the name of "Columbia." This problem is so aggravating to
Colombians they even sell t-shirts in their airports that say "IT'S COLOMBIA
NOT COLUMBIA"

~~~
nestorherre
Why is this getting downvoted? As a colombian myself I find it frustrating
everytime it's spelled wrong. Upvoted.

------
quickthrower2
The Cloud. The sky is the limit for how much you can run up on your credit
card. The silver lining is what you're going to be paying to your cloud
provider.

------
klaaz0r
Offtopic

I need to register with medium now to actually read the article?

~~~
comprev
It looks like you get 1 month "free". Open the article in incognito mode and
it's available without registration.

~~~
labster
> Get one more story in your member preview when you sign up. It’s free.

Oh wow, one free story for logging in! Such generosity of content other people
wrote for free. Why did we all leave LiveJournal again?

~~~
55555
Medium pays the writers who opt-in to the paywall by dividing rev on a pro
rata basis a la spotify.

------
crankylinuxuser
Well, simply put: don't use Firebase.

I know I've seen at least 4 articles about 'OMG Surprise horrific billing"
with them. And to be quite frank, you can buy a EC2 instance, throw Postgres
on it, and have a stable bill (Except for bandwidth costs, but that's
trivial... usually).

I'm sure Firebase has good features, but this surprise billing is terrifying.
They can't even offer a warning "10 min average indicates a bill roughly
XX,000$/mo". I could not suggest it in good faith for anything, especially
since it doesn't have a hard cutoff.

------
reimertz
I just want to point out to people who are new to Firebase that you can query
your Firestore collections. Also, Firestore Queries are cached by default, so
if you try and fetch data that hasn’t changed, you shouldn’t have to pay for
that read.

db.collection(‘Payments’).where(‘id’, ‘==’, ‘payment
1’).onSnapshot(console.log) // 1 read, then cached

Whereas specifying a document and then manually get it always counts as a
read. Example;

db.collection(‘Payments’).doc(‘payment 1’).get() // always counted as a read

It's funny.

I've had multiple discussions with people who build apps using Firebase and
then not being able to scale to 1000 concurrent people without the BE falling
apart. I want to tell them that they've done something wrong, but since none
of my apps haven't reached those kind of usage levels, I really don't know.

I think Firebase comes with a lot of power, and as long as so plan to scale,
design your database model, have proper security rules and cache as much as
you can, you can probably host a 1M concurrent users without getting scaling
problems. Cost though, ooh yeez. :)

------
nicebill8
This video [1] goes into more detail as to what exactly they did wrong and how
to fix it.

[1] [https://www.youtube.com/watch?v=Lb-
Pnytoi-8](https://www.youtube.com/watch?v=Lb-Pnytoi-8)

------
sessy
TLDR: The huge bill was a result of an improper way the application was coded.
They contacted Google/Firebase who were gracious enough to waive off the bill.

~~~
genericacct
That's very nice of them. However I'd have to read all of firebase's small
print before i would consider using it. Does their SLA guarantee data
availability even if google dcide to spin it off or sunset the way they did
with google+, reader and such?

~~~
kkarakk
Does it matter what their fine print says if you have to bang your head
against a buggy automated bot process to get support things done? Switched
over to MS appcenter recently(they added support for cosmosdb +
authentication). Their customer care is so nice by comparison

------
lofo
Writing a cost calculator for firebase is a baffling experience. The relative
difficulty to gather precise information and model costs are intentionally
discouraging you to undertake any estimation.

------
Gustek
@squidrings I believe you are the author?

Very interesting post especially as I am myself currently working on my first
firebase based app and making sure I don't get huge bills is one of my
worries.

Just little correction needed:

> July last year, a crowd funding campaign went viral in Columbia

It's Colombia not Columbia.

