
A startup’s Firebase bill suddenly increased from $25 to $1750 per month - lpellegr
https://medium.com/@contact_16315/firebase-costs-increased-by-7-000-81dc0a27271d
======
flipp3r
Reminds me of getting Bait-and-Switch'd by Google App Engine in 2011. From
10's of euros to 1000's.

> They have no phone number to contact, no way to dispute this other than
> email — which they have ignored us for over a month now without replying to
> our continued requests. Trapped. Doomed. We have no further options.

Sounds like Google did a great job with Firebase. Definitely gives you that
Google-feel of unresponsive support and no hope when you have problems with
any of their services. I bet they also have a "community" support forum or
Google Group full of helpless people talking to a virtual wall. (Edit: Yep,
they recommend Stack Overflow, Quora, and then their Google Group.. lol. Some
things never change)

~~~
corford
I learnt this lesson the hard way. Solution: get a separate debit (not
credit!) card for all cloud stuff. Make sure you only transfer enough money on
to it each month to cover what you reasonably expect your bills to be (and
that you can quickly top it up in hours if you legitimately need to).

Worst case, if AWS or Google decide to fuck you over, let the bill bounce.
This way you've still got funds on hand to deal with the fall out and some
leverage to negotiate a solution.

~~~
Moru
In Sweden we have at least one bank that has e-cards where you can set up a
new card for each transaction. You decide how long it is valid in months and
for how much total and recieve a unique card number and ccv. I have been using
this at least 15 years for payments online. You set it up, make the payment
and then immediately close the card again if you want to be extra safe.

~~~
sync
Final is doing a take on this as well for the US:
[https://getfinal.com/](https://getfinal.com/)

~~~
comex
There's also Privacy:

[https://privacy.com](https://privacy.com)

------
mayop100
[Firebase Founder here] I’m very sorry for the surprise and frustration
experienced by the poster, especially due to problems working with Firebase
support. We’re embarrassed by the level of communication on our side, and
we’ll be working directly with this developer to resolve the issue.

There are a couple of things I’d like to clarify for the group, to help folks
understand what happened here, and hopefully help others avoid the same
problem.

1 - The Firebase Realtime Database charges for SSL overhead on all requests
(we charge for all OSI Layer 5 network usage). We’ve always had a policy of
charging this way. Unfortunately, we introduced a bug late last year that
began undercharging for SSL, so when the bug was fixed it surprised many
people who had gotten used to the lower numbers. For most people, the change
was very small. For a small number of people though with exceptional use cases
(ie. tons of small network requests from IoT devices without support for
session tickets), this can result in a larger change. We identified and
contacted developers who were significantly affected, but this customer didn’t
get the email. We should have done better. (we mention this change in our FAQ
--
[https://firebase.google.com/support/faq/](https://firebase.google.com/support/faq/)
\-- under “Why was my Realtime Database reported bandwidth lower than
average”)

2 - We recently started actually enforcing overages on legacy plans and our
current fixed-price ($25/mo Flame) plan. This is new -- in the past, we
allowed unlimited usage on every plan, since we hadn’t built the tools to
control this. This meant that many of our developers were getting far more
than the listed limits for free. So you may start receiving emails now warning
you of bandwidth overages, not because your usage patterns changed, but
because we’re now enforcing limits for your existing usage. If you upgrade to
the Blaze plan, you’ll start being billed for your full usage amounts -- so
double-check your database’s “Usage” tab before upgrading.

I’ll be looking into our profiling tool to see if we can improve it to give a
more complete picture of costs.

Again -- I want to apologize to the poster for the poor support experience. If
others on the thread have had similar problems and feel they are not getting
the attention they want from support, feel free to email me directly as well:
andrew@firebase.com

~~~
omgwhat
What support? Google has no support, ever. You care about "users" but don't
give a shit about any one user.

I'm starting to really fucking hate Google.

~~~
Veratyr
Google doesn't have support for free products but they certainly do for paid.
Whether the support is good or not is another question but it's there.

Just go to
[https://support.google.com/googleplay/?hl=en](https://support.google.com/googleplay/?hl=en)
and click "Contact Us" in the top right. All the menu items I clicked that
don't have straightforward solutions went straight to a menu where within 2-3
minutes I can get live chat or a phone call.

I've also had no problems getting help with my paid Google Apps account when I
need it. For cloud there's paid support with a 4-hour response time:
[https://cloud.google.com/support/](https://cloud.google.com/support/)

~~~
funkyy
They do have support. I was stuck once with a friendly fella, that spent 45+
minutes talking about weather and weekend adventures to me, rather than
solving my issue. It took less than 5 minutes to solve the issue, almost an
hour of talk time. I would even bet money that the guy was heavily stoned.

At the end he asked me to provide positive feedback to him in exchange for
friendly service. He said he will redirect me somewhere and I should give him
a proper feedback. This is even while I was telling me I have no time and this
is an urgent matter. At the end of the call it turned out it took him 5
minutes to solve the issue and he just wanted to chat while I was stuck at
talking to him, thinking he is waiting for a solution to propagate.

I can't say this is a good support.

~~~
Moru
This is what happens when you try to drive work-morale with a stick and carrot
approach. Teach your workers good morale and lead with good example, every
customer is important. But also your employees.

------
shaqbert
Pricing mistake 101: You never ever change those old plans. Instead, you
grandfather them. Especially so if you're still a young platform and have most
growth ahead of you, the "loss" of not charging the new - and supposedly
higher - pricing is gonna be trivial 2 years from now, with lots of new
projects coming in. But the NPS hit from pissing your most loyal users off and
the subsequent damage to your growth curve are huge.

Also, I do love Bezos mantra of always becoming cheaper over time. That is the
core of Amazon's brand. Whereas Oracle, IBM, and the old guard always make you
pot committed and charge an arm and a leg, the Amazon guys will leave money on
the table. That is how you earn trust.

~~~
delhanty
Tell that to FastMail, who have pissed me off by deciding to end-of-life the
"lifetime" 16Mb member account I set up for my father with a one-time $15
payment.

16Mb is modulo zero these days, but it's enough for him - he just deletes some
emails when it gets full. He also gets imap access and FastMail's spam
filtering, which is really very good. I can also assign him an email address
from a domain that I own that is associated with my FastMail enhanced account.

I don't want my $15 back. What I DO want is for FastMail to honour our
existing agreement, which was a LIFE-TIME 16 Mb member account in exchange for
$15.

See also here:

[http://www.emaildiscussions.com/showthread.php?p=599512](http://www.emaildiscussions.com/showthread.php?p=599512)

Text of their latest email is below the line:

=============================================

Back in January we contacted you with some important information about your
_@_.* FastMail account.

We're now contacting you again to remind you that from 31st July, 2017 our
'Member' level plans will be discontinued.

To continue using your FastMail account after this time you'll need to select
one of our current plans. Please note: your email address will remain the
same.

As a long-time FastMail user we're offering you 50% OFF all upgrades until the
end of July, which includes our already discounted multi-year subscriptions.

And if you upgrade now we'll also add all the remaining time until 31 July for
free!

As an added bonus we've also given you $15 credit to put towards your
subscription, which is equal to the amount you originally paid for your
account.

This means if you were to upgrade to a Basic account today for one year, with
the discount and account credit you are effectively getting FastMail at no
charge until 31 July 2018 with a massive 2 GB of storage (up from 16 MB)!

~~~
muraiki
Thanks for sharing this. I had recently set up an account with FastMail, but
now that it's clear that they won't abide by their own promises, I'll look
elsewhere.

~~~
FireBeyond
Yeah, don't be so fast. I've had a legacy Enhanced account which they show no
signs of removing - I went to renew it today and could (though I went to the
Standard).

I've had nothing but excellent, and personal, service from them, for years.
Very high availability, no canned responses.

Even when bitching about Google's support of "this address used to be with
Gmail but is now external, but we'll make it problematic for other users to
send it calendar invites", though in no way did I imply Fastmail was at fault,
they jumped into the HN thread to say "Hey, let us know if we can be of
assistance figuring out the issue".

------
roadbeats
I have a game being mostly used by kids and it doesn't generate revenue, and
it has been in the free tier of Firebase for about 2 years. This month
suddenly my account got disabled due to DB usage (I only keep sessions and
ids, still can't believe I exceeded DB usage). But the traffic was same as
old. About 1.5 weeks ago I woke up at 6am with an e-mail telling me my account
was disabled. And I had to switch to something else, spending hours. I don't
recommend using Firebase to anyone anymore.

~~~
YCode
They don't seem to understand that once you lose trust and reputation in a
community you will struggle to get it back.

For example this thread has more or less convinced me to take Firebase off the
table for any future projects I might have used it for.

~~~
noja
Yet they keep doing it, so are their numbers showing otherwise?

~~~
metaphorm
not all business decisions are well-grounded in financial upside. a lot of
dysfunctional outcomes are a result of organizational dysfunction and broken
culture. a company can be making bad decisions for reasons related to internal
dysfunction, and be losing money on it, and persist in this because of their
dysfunctional culture.

as many in this thread have pointed out: Google has a dysfunctional culture
w.r.t. customer service.

------
apatters
Wowow! In this position my first step would be to determine if there's any
legal basis for disputing the charge (an informal conversation with a lawyer
or even just someone who's been through a similar experience is a fine start).
Even when a billing change is disclosed in advance and the customer actually
uses the bandwidth, massive spikes in metered pricing are a tricky area and
have attracted the attention of regulators in the past--for instance telecoms
have come under fire by charging sky-high overage fees for mobile data and not
warning you until you've already racked up a few extra zeroes worth of charges
on your next bill.

Also, immediately dispute the charge with your credit card company, as you
want this on their radar. Their interests are aligned with yours to at least
some degree (if Google's causing financial distress for one of their customers
with thousands of dollars in unanticipated charges, how many others may get
hit?).

Aside from that, if things played out as described: major dick move, Google!
Is "do no evil" a bad joke these days? Seriously, though, Google is not a dumb
company and my guess is that if the right person was made aware of this case
(e.g., anyone who understands that the most important word in "customer
relationship" is the second one), they would make it right.

I'm totally on the side of the author: when possible, it's worth a little
extra elbow grease to build your business on top of free, self-hosted software
instead of proprietary SaaS. Think of it as insurance against cases like this
one.

~~~
pbhjpbhj
"Don't be evil." was dropped in 2015 or thereabouts IIRC. I'd guess when they
realised it was becoming more of a joke than a motto: "Google, the company
who's motto is 'don't be evil' angered customers again today when ...

~~~
logicallee
Can you link to a report or some source, stating they dropped the "don't be
evil"? where do you get that, and the date, please?

Thanks.

~~~
ninju
[http://fortune.com/2015/10/05/alphabet-google-
evil/](http://fortune.com/2015/10/05/alphabet-google-evil/)

... The new code of conduct has a close approximation of the philosophy—though
perhaps more formally phrased—in the very first sentence of the preface:
"Employees of Alphabet... should do the right thing – follow the law, act
honorably, and treat each other with respect."

~~~
grenoire
"treat _each other_ with respect"

Sounds like a loophole, not just 'more formally.'

~~~
pluma
As opposed to "evil", which is extremely subjective already?

------
factsaresacred
I use Firebase and find it easy, useful and constantly improving (cloud
functions etc.). Their support, however, is a joke.

5 questions can be logged _per year_ for one-on-one email support and then
you're on your own. And this is for urgent requests.

I can understand not wanting to be inundated with wasteful questions but when
Firebase is changing their API and pricing plans and you're trying to scale a
product, and _paying them lots of money_ in the process, support should be
forthcoming.

They also don't respond on their Twitter handle, which is a further crime.

~~~
danholevoet
Dan from Firebase here. I’d like to clarify a few details about the support
options that are available. First, the limit of five questions per year
applies only to technical troubleshooting questions (i.e. some indeterminate
issue in your code that you’d like help with). Developers are always able to
contact us an unlimited number of times for issues relating to identified bugs
or to request features. Additionally, any questions relating to your account
or bills (such as the issue in the original post) are also unlimited.

With that said, it’s been a year since we introduced the five-a-year limit on
technical troubleshooting questions. I’ll see if this is actually a useful
limit to keep in place and remove it if not.

~~~
factsaresacred
Thanks for explanation, Dan. My issue is that the switch to the 3.x SDK and
moving my assets to Firebase hosting depleted 4 of my 'technical
troubleshooting questions' (two of those were bug reports which seem to no
longer deduct from the five-a-year limit).

So now I'm looking at a message telling me that _' for all urgent
requests...you have 1 question remaining'_ and hoping that nothing bad happens
more than once between now and whenever you decide to allow me, a paying
customer, to ask for help again.

To make clear: 99.9% of the time Firebase just works, which is great. And when
I did receive support, it was excellent. It's the support model that's weak.
Making billing and bug reports unlimited while limiting _urgent_ requests to
five a year is poor policy, and not very useful - for your customers at least.

~~~
danholevoet
Yes, I understand completely. The intent of the limit is obviously not to
discourage legitimate issues from reaching our team. Limits like this are a
blunt instrument, and it's possible that the time has come to remove it.

I can't promise a change immediately, but I am taking a very close look at
what we can do here.

------
neftaly
We've also gone from ~$25/mo to $1000-2000/mo with the pricing change. While
it's not the end of the world (we're near shipping replacements for FB stuff
anyway), there's also been 3-4 incidents of downtime (> 1 hour) in the last 2
months, with no way to contact anyone except their stupid web form.

~~~
brianwawok
Do you not pay for support? It starts at $150..

~~~
ocdtrekkie
The idea of having to pay extra to talk to them about their screw up is pretty
laughable.

~~~
brianwawok
Why?

I like the fact I can pay one price for self support and another for 15 minute
resolution. Why force people to pay for support if they don't need it?

~~~
ocdtrekkie
I think there's a major difference between say, support figuring out how to
implement their product, which is a you problem, their service is working
fine.

And a wholly different one if they screw up and you can't talk to them about
it without paying them.

The former is an incentive to self-serve and investigate yourself, the latter
is an incentive for the company to provide shoddy service to people pay to
resolve it.

~~~
theptip
Tiered support contracts are common in enterprise software. If the base level
of service were bad, nobody would use the software, and that's clearly not the
case here.

(More common is the Oracle/IBM-style approach of making your docs vague enough
that you need professional services to actually integrate the product).

~~~
kuschku
A free level of support that requires answering within of 14 days to any
issues via email, letter, or phone, and providing an address that a letter can
be sent to, is required by law in the entire EU for any paid services.

Just FYI.

Google doesn’t even provide that.

------
CodeWriter23
So, they discovered they were under billing by measuring bandwidth via the
server logs instead of at the switch. Then proceeded to not tell a soul they
would be making a change that has the potential to greatly increase your bill.
Do I have that right?

~~~
yunolisten
Pretty much :)

------
DomBlack
> Always build your architecture in a way that will avoid becoming trapped
> into a specific service. Amazon’s AWS Lambda sitting between any services
> and your app is a strongly recommended path!

Isn't using AWS Lamba as your gateway, trapping you into using a specific
service, just as much as firebase was?

~~~
moduspol
You wouldn't have clients talk to AWS Lambda directly. You'd put it behind API
Gateway, which would allow you to use your own domain in front. Your clients
would just know to send requests to
[https://myapi.mycorp.com](https://myapi.mycorp.com).

Since you control the domain, you could then stop something like this more
quickly.

~~~
bradynapier
yes, this is exactly how it is implemented now

------
jamesk_au
Possibly relevant (from the Firebase FAQ[1]):

 _Why was my Realtime Database reported bandwidth lower than average between
September 2016 and March 2017?

For our bandwidth calculations, we normally include SSL encryption overhead
(based on layer 5 of the OSI model). However, in September 2016, we introduced
a bug that caused our bandwidth reporting to ignore encryption overhead. This
might have resulted in artificially low reported bandwidth and bills on your
account for a few months.

We released a fix for the bug in late March 2017, returning bandwidth
reporting and billing to their normal levels._

Does not excuse the issue with support.

[1]
[https://firebase.google.com/support/faq/](https://firebase.google.com/support/faq/)

~~~
Aissen
I'm guessing they had to update this FAQ after they started ignoring the OP
(and other customers) messages. Also, something is wrong with the timeline
with regards to the OP's blogpost where they say they've been using Firebase
for a while.

~~~
infogulch
Yeah OP says they've been using FB with similar costs for over two years, so I
highly doubt they're complaining about something that was only a problem for
the last 6 months.

------
jakub_g
> It was extremely hard to write this article for fear of being looked down
> upon for our mistakes. I can only hope that some of you may learn from our
> embarrassing mistakes and implement solutions to the growing problem of
> service trap.

Kudos for writing it down. This kind of cautionary articles are very valuable
and much needed.

Moreover, getting to front page of HN is one of the very few ways to have your
case resolved by some googler accidentally passing by - hopefully someone will
help the guys.

~~~
holtalanm
I don't think this should be considered embarrassing. It is a situation that
almost anyone could find themselves in, and it sucks.

Honestly, the fault it purely with Firebase.

------
cwmma
They said the issue was due to TLS tickets and not setting keep alive values.
TLS tickets are await of resuming a TLS session without having to renegotiate
it, considering that you are only polling for a boolean, that negotiation
overhead could be much greater then the actual being passed. This isn't
something specific to firebase so that's why no library would mention it.

Keep alive is a flag that allows you to reuse a single connection instead of
closing it, if it's closed then again you have to renegotiate things. So it
sound like you may have been inadvertently spamming them with tls connection
requests, the fact that they said they were counting blocked requests makes it
passable that you were also sending invalid tls handshakes.

This would make the real issue that they had been previously underreporting
your usage and now are correctly counting it. Now if they had wanted to keep
your business they should have, you know, warned your first or helped you fix
your code instead of just surprising you like that. The fact that the change
was going to cause a customers bill to spike should have been a red flag to
them.

~~~
tamalsaha001
Interestingly I was researching TLS forward secrecy. And the recommendation
seems to be to disable tls tickets. [https://timtaubert.de/blog/2014/11/the-
sad-state-of-server-s...](https://timtaubert.de/blog/2014/11/the-sad-state-of-
server-side-tls-session-resumption-implementations/)

~~~
cwmma
probably not something they need to worry about here as it's just polling for
booleans so forward security is probably not as high a priority as other use
cases.

------
kbart
That's the same, old vendor lock-in approach. It has happened with the
servers, databases, network equipment etc. now it happens with cloud --
nothing unexpected, prepare to see more of these when cloud market stabilizes
and clear leaders emerge. That's why I always prefer open solutions, that can
be moved/migrated/replaced easily and advise others to do the same.

~~~
moduspol
Eh, it looks like the service provider hadn't taken into account an oddball
use case in the pricing / metering, and then suddenly did.

They certainly should have communicated that better and their tools / charts
should have a way of showing the billable data, but it's a bit of a stretch to
call it vendor lock-in. It's not like everyone on Firebase is complaining and
the author admits they made major mistakes.

~~~
princekolt
But why do providers get a free pass when they do a major mistake (à la GitLab
losing tons of not-backed-up data), but customers are required to accept
whatever the provider throws at them? It seems pretty unfair to me.

~~~
MaulingMonkey
Providers _to some degree_ get a free pass when they do their best to fix
their mistakes.

In Gitlab's case, their response was excellent and their total data loss was
less than a day's work - and even that only for a full time manipulator of
gitlab metadata (project managers? some parts of QA?): "Database data such as
projects, issues, snippets, etc. created between January 31st 17:20 UTC and
23:30 UTC has been lost. Git repositories and Wikis were not removed as they
are stored separately." ( [https://about.gitlab.com/2017/02/10/postmortem-of-
database-o...](https://about.gitlab.com/2017/02/10/postmortem-of-database-
outage-of-january-31/) )

Considering that "[...] out of 5 backup/replication techniques deployed none
are working reliably or set up in the first place." (
[https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-
VCx...](https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-
VCxIABGiryG7_z_6jHdVik/pub) ), this was a pretty amazing result, and about the
best a response they could make. That, combined with their transparency, earns
something of a pass (especially when data loss _I care about_ is already
limited by git's DVCS nature, even if they had obliterated the repository
data.)

In Firelab's case, if no other action is taken, they've actively mislead their
customer in failing to follow through on crediting their account, charged them
an obscene and unsupported price hike, and gone incommunicado. That's like...
an _anti-_ pass.

~~~
brazzledazzle
Chances are they also knew when they fixed the bug that it would result in
price hikes. They could have easily generated a report showing which customers
would be impacted, allowing them to contact them ahead of time.

~~~
MaulingMonkey
From the followup updates, it sounds like Firebase actually tried to do this:
[https://medium.com/@startupandrew/firebase-founder-here-
im-v...](https://medium.com/@startupandrew/firebase-founder-here-im-very-
sorry-for-the-surprise-and-frustration-experienced-by-the-poster-b0065c99f53e)

Sounds like they've processed the credits as well, so that's good.

------
ChicagoBoy11
I've only used Firebase for really small pet projects, and one of the big
drivers making me hesitant to actually use it at work was the really large
difficulty in actually transparently getting a grip on what my usage looked
like.

A while ago their pricing model was based on concurrent connections to the
live DB, but then you read the docs and it seemed to suggest that you had to
do a different kind of math to actually figure out what your REAL usage was.
Well, they decided that was way to complicated and so they nixed it. This
scenario just reveals that this isn't really the issue -- the issue is trying
to obfuscate what the real factors that the billing is based on. If they
literally just gave you a dashboard with a breakdown of where the source of
your bill would come out to, in detail, it would alleviate 100% of these
issues. All we need/ed was better monitoring!

------
jakozaur
Yet another example that Google so far doesn't get the public cloud and may
have hard time to catch up with Amazon Web Services.

E.g. shutdown Prediction APIs, charge 70x, not enough support.

I believe Google got much better tech than AWS, but the culture, support and
strategy has blind spots.

~~~
steego
I think Google gets the public cloud. What they struggle with is lower margin,
revenue driven, customer obsessed businesses.

First, it's important to understand unless you're buying ads from Google, or
renting their machines, you are not their customer. Users are inventory and
they spend a lot of money trying to procure inventory (users) for their
customers (advertisers).

When you look at it through this lens, you can see how Google as an
institution can tolerate experimenting with ideas that are disconnected from
revenue models and dropping failed ideas like hot potatoes. It makes sense for
them to behave that way.

Amazon's services are almost always designed with either a concrete or direct
path to revenue upfront and they have a huge customer support infrastructure
they can tap into. They move a little slower in certain areas of R&D because
customer obsessed businesses are tolerant of deprecating technology when it
doesn't work out. That means they'll do more market research, take less risks
and focus on tenaciously steady growth rather than trying to always hit it out
of the park.

When choosing a vendor, it's important to understand how their offerings
affect the solvency of their business and whether it fits into their core
competency.

In this case, Firebase's prices were too low. That should have been a warning
sign. The other warning sign is Firebase is a such a small percentage of
Google's revenue, Google doesn't have a _huge_ incentive to fix customer
service issues.

~~~
ChicagoBoy11
*"customer obsessed business are <less> tolerant of deprecating technology..."

~~~
steego
Yes. That's what I meant.

------
jondubois
Synchronizing data in real-time is computationally expensive (this applies to
both Operational Transform and Differential Synchronization approaches). The
price of Firebase was probably kept artificially low to attract new users but
this was a ticking timebomb. Those people/companies who are using Firebase as
their primary database are going to suffer now. It should really only be used
for specific applications where datasync makes sense such as collaborative
text editing. I wouldn't even recommend using it for chat because it's really
easy for the complexity to get out of hand there and you can usually achieve
similar results using a combination of pub/sub and REST.

~~~
josephg
> Synchronizing data in real-time is computationally expensive

No, its not. I was responsible for implementing, deploying and managing the
infrastructure at lever.co when we were a tiny fledgeling startup. The entire
application is built on top of JSON OT. I took some measurements at one point
when we had ~thousands of active browser sessions of our app. At the time we
were seeing about 1-2 OT merges (transform + re-apply) per _day_. All the
other non-concurrent operations can get sent straight to the database. I don't
know what the numbers are now, but I'd be shocked if OT ever becomes the
bottleneck.

For text based OT you'll see more concurrent edits. But for text based OT, I
have a little C library that can comfortably do 20 million simple text OT
transforms/second on a single core of my old 2012 macbook air. Good luck
making that a bottleneck.

~~~
jondubois
I guess it somewhat depends what kind of OT we're talking about (text or
JSON). In any case, I don't think that the bottleneck would be the OT
algorithm itself - More likely, the bottleneck would be the number of messages
(HTTP requests or WebSocket frames) required to send each individual operation
between the client and server. If you have text-based OT and you send an
operation over the wire each time a user presses a key, and if you do this for
every single input field in your app, it's going to add up. Not saying it's
not feasible but it's not going to be as fast as making a plain REST call for
those use cases that don't require synching. That said, I wouldn't be
surprised if OT turns out to be much faster than differential transform in
terms of raw algorithm speed but again I think the bottleneck will be the
number of frames/requests that need to go over the wire.

~~~
josephg
> Not saying it's not feasible but it's not going to be as fast as making a
> plain REST call for those use cases that don't require synching.

The whole point of OT is that it lets you fearlessly merge concurrent edits.
You can lean on that to rate limit messages to or from the client, if you need
to. Because the client can always apply its own edits immediately to its own
local model, there's no perceived latency. So, if you design your system right
you can batch up changes at any granularity you like (per page, per form, per
second, dynamically based on load, whatever). Of course, the tradeoff is that
you lean on the OT system more by batching up edits. But it can be made
effectively free to transform if the edits modify different parts of the
database (using range trees to cull, etc).

But yeah - it is much faster than differential transform. I'm not sure how
good diffing algorithms are in practice, but best case they need to scan the
entire document. On the other hand OT only requires size & computation
proportional to how much was changed. If we're collaboratively editing a 10kb
text file, changes will mostly only be a few bytes each. And applying each
change using a good rope library is a O(log n) operation.

Of course, all this requires a database which supports OT out of the box
_whistles innocently_

------
bradynapier
I authored the post and yes you are correct about the AWS Lambda being another
service. I have since modified it based on a users suggestion to reflect.

I guess the point was that by adding the middleman between it there was more
control --- We are self-funded and small group of people. We can't afford to
hire a bunch of people to know about containers and servers and every little
detail.

Thanks for your comments! Didn't expect so many views so quickly on this...

~~~
geggam
Investigate SRV records for load balancers as well.

------
jayd16
The author talks about using their own infrastructure and open source
alternatives but that's a revisionist fantasy. They would have never gotten
off the ground as a start up by buying a bunch of servers and spending time
spinning up infrastructure. Not to mention they would have been paying the
full costs of their bandwidth from the start.

The real lesson to learn here is to never hit a paid service going through a
domain you don't control. DNS and proxy tricks would have saved the day.

~~~
twic
> The author talks about using their own infrastructure and open source
> alternatives but that's a revisionist fantasy. They would have never gotten
> off the ground as a start up by buying a bunch of servers and spending time
> spinning up infrastructure.

"We went from a Skype group of 10 beta testers to hundreds of active users,
then thousands in only a few short months."

Getting off the ground required supporting thousands of users. That doesn't
require buying a bunch of servers. Rent one basic physical machine from
Hetzner or whoever, spend an afternoon setting up PostgreSQL, and you're done.
Reactionary, perhaps, but hardly revisionist.

~~~
sergior
And that machine could serve all the traffic they need and also pay full time
dev op guy if needed still for fraction of the cost of the cloud

~~~
tschellenbach
Seems hard at $25 a month :)

~~~
sergior
You can get devops for $1 a month from Asia and dedicated server from an
auction. You could still have some $$$ for a coffee!

~~~
sushid
Where can you get devops for $1? I'm talking about actual DevOps, not a
Mechanical Turk version of Jenkins.

~~~
sergior
Jokes aside, you could get an apprentice for free

------
throwaway13337
Look to stories like this before you go 'serverless' in your organization.

Building on top of a proprietary stack with no guarantees on pricing and
future availability will lead here.

~~~
Nemo157
It doesn't sound like it's the 'serverless' hosting that is their issue,
rather that they have no direct control over the endpoint that their difficult
to upgrade client software is hitting. They actually fix it in future versions
of the client software by putting controllable serverless endpoints in front
of the firebase endpoints they're hitting, so if something like this occurs
again they could just change their endpoint to not hit the third-party
resource that is costing them so much money.

~~~
raesene6
The more general problem with serverless architectures that this post
illustrates is that you're essentially running on a series of FAAS/SAAS style
services which can change their billing at any time.

So you either have to code a portability layer so you can move to another one
in the event of an unwanted change (which could be quite a lot of work), or
you accept the risk that a 3rd party could cost you a lot of money until you
can change off their service (which could be quite technically challenging
depending on the complexity of your use case and the availability of
comparable alternatives)

Whilst with things like container services you retain more control (ultimately
you can self-host if need be) as soon as you step onto pure serverless you'll
always be at the mercy of the providers of the services you use.

~~~
brazzledazzle
I think they make it pretty clear the problem is with their inability to
change the endpoint (i.e. the URL). Switching from lambda to something else
could be a pain in the ass but they'd be hitting their own domain and changing
where that points is relatively trivial. Right now they have clients out there
hitting a Firebase endpoint that they have no control over and that they can't
update quickly.

------
betageek
The detail that amazed me is the lack of visibility of the issue in reporting
tools - it's all very well changing the terms, but not giving your customer
any ability to see what is increasing the bill seems massively unfair.

~~~
Hedja
Exactly the reason why I decided not to pay for Firebase and instead went with
gradually rolling my own solutions.

Their Realtime Database is extremely convenient and pretty useful for web
apps, but there's no way to limit the connections and track bandwidth usage in
real time. It's basically an open door for anyone with only your Security
Rules to block data access. There's no logs to even see if your rules are
working as expected, nor where your connections are coming from. They didn't
even have that simple blue line showing basic usage in OP's post until
recently. The profiling tool mentioned in the OP is also recent and doesn't
give a proper picture.

It's just really pathetic. They sell themselves as a service company for
startups, but one of the biggest things startups need to do is manage their
costs, which Firebase doesn't provide in any form. And now since Google bought
them, their support will be near non-existent as with all Google products.

Having said all that, their Free tier is great for live prototypes and doesn't
require adding payment information. Just make sure you're not tying yourself
in if you want to progress. Make thin wrappers around their libraries and
export your data regularly if it's not throwaway.

~~~
kefka
"Welll, we have this there chalkboard that we write what we think your price
is. And some intern draws lines on a graph to match what we think it is. Oh,
proof? Nahh, we dont do that. Oh, logs? Nope, you'll just have to _trust
us_..."

------
staticelf
I think Firebase was kind of douchebags assuming the story is correct, if you
change how people are billed you should give people time to adjust their
software.

Stuff like that makes me want to host everything myself as it will for sure be
the cheapest option.

~~~
tmikaeld
Hosting by yourself is always the safest long-term solution, even if you are
hosting it in the cloud, you at least protect yourself from this "Company
reserves the right to change the TOS at any time..."

------
pg_bot
Evaluation of technologies is an important skill for any developer to learn.
Lock-in on a proprietary technology that powers your company should be a giant
red flag. In some cases, you are obligated to use a third party (Payment
Processing) but if it is a core part of your business you should really think
about why you are unable to do it yourself. I've seen a few demos of firebase
before and while their demos are highly polished, there is nothing they offer
that couldn't be done by open source tools.

A hard lesson learned, but hopefully this company will pull through.

~~~
aeorgnoieang
> In some cases, you are obligated to use a third party (Payment Processing)
> but if it is a core part of your business you should really think about why
> you are unable to do it yourself.

I'd think getting paid is a core part of every business! Tho I'll admit
basically no one can do it themselves.

------
IanDrake
I think there might be an advantage to either being older or perhaps being
more experienced, I'm not sure which. I find people my age automatically go
for renting equipment vs the cloud.

I've never understood the advantages of these services except for when you
have drastic usage changes throughout the day.

But all the younger folks seem to think it's awesome:

"Look ma, I didn't even need to register my own domain, wait for DNS
propagation, or install software, or wait a few days for hardware to be
provisioned. That cut my MVP time from 30 days to 25 days!"

For about $250/m I can get a dedicated windows server that can handle just
about anything a startup can throw at it. Sure it's work, but not too much
work.

Hardware matters. I would estimate the cost of running in the cloud to be 3-4x
more expensive than rented hardware. The $60/m VPS you're getting with a
virtual core is usually one hyperthread on one actual core. That's a lot of
money for so little and hourly billing doesn't help if you're using it 24/7.

The proprietary SAAS situation is even worse than VPSs. Besides the lock in,
the billing is so abstracted from reality that you'll never realize how much
you're overpaying.

~~~
brianwawok
Except the young ones that know how to use the tech, can go:

Hey this portion of my load is HIGHLY flexible. I can use a very simple celery
queue to scale this out to 1000 preemptible instances, and pay literally
peanuts to harness the power of 1000 computers for the 30 minutes I need it,
then shut it down for the other 23.5 hours a day.

There are a wide range of ways to do things. From Heroku to Bare Metal to EC2
to Lambda to managed services (like Firebase). I think they all can be a win
in the right case. If you are always using the same solution (in your case,
maybe rented window servers?), I would heavily question if you are using the
right tech. And personally as soon as I hear people running Windows to run
actual code that does work, I shudder a bit.

~~~
IanDrake
I think I exempted the exact scenario you mentioned.

Windows, like Linux, is a great OS, IF you know what you're doing and a
terrible OS IF you don't.

My point is, hardware matters and abstracting it has real cost in terms of
control, performance, and dollars as the OP has now learned.

~~~
brianwawok
Depends. You can use docker on top of say a container-OS and not need to know
anything linux (and in fact you can't even do anything linux, it is all locked
down and readonly).

Of course hardware matters. Of course it is embarrassing to have your entire
business model crushed by how you built your product. It happens in many ways.

------
theptip
I can appreciate the appeal of a Backend-as-a-Service offering like Firebase,
particularly if you're a one-person frontend-heavy dev team; it seems like it
gives you a lot of leverage in the early experimentation stages. I'd like to
play with it some day soon.

But this sort of incident really underscores the business value of having an
exit strategy from whatever cloud you're running in.

That's why I like Kubernetes, since you can just spin up a new cluster on
whatever cloud you want (or on-prem), and all your app deploy scripts remain
the same.

Likewise for the choice of boring data store (Postgres/MySQL). I'm happy to
pay a cloud provider to run my DB for me, because I know I can trivially
redeploy in another hosted DBaaS or on-prem solution if I had a need to do so.

You could make the same case for running your stack on VMs that you manage;
pick your poison.

------
danny_taco
This is exactly why we moved out of Firebase once it got acquired by google.
We knew support would become nonexistent and we would also run the risk of
Google shutting down the service for no good reason.

~~~
pranavsinghca
did you go to self hosted or is there a firebase style real time db provider
that is already at scale? basically it would have to be AWS or Azure for me to
feel comfortable. Does AWS have something now?

------
stemuk
Did a Firebase team member already comment on this? In a few hours Google will
present some new Firebase features at io17, and I would expect them to avoid
bad press now as much as possible...

~~~
kuschku
That's probably the purpose of this post — the best way to get support from
Google, after all, is still by posting a story on reddit, twitter or HN.

Which is pretty sad.

~~~
stemuk
I think it's kind of weird to see Google/Firebase team members swarming around
in dozens on HN when they present a new feature, which seems to me like a
coordinated marketing effort by Google. But once problems arise or they get
criticized by their own users, no one seems to be available for a comment.

~~~
redwood
I was thinking the same thing. There's a giant circle of patting each other on
the back every time they roll something out, and always have that very precise
disclaimer that they work for the big G.

Then again, as a meta comment I will point out that the fact that both you and
I so clearly notice this suggest that we are following these updates much more
closely then most of the folks here.

~~~
stemuk
True, I used to be really interested in Firebase and their set of products.
But because of several pretty bad experiences related to vendor lock-in and
(very) high prices I switched over to an open source alternative.

It might not be so well known, but I found
[http://www.deepstream.io](http://www.deepstream.io) to have a relatively
similar feature set, without having to deal with bad support (if it bothers
you, just fix it yourself) and high prices, since you are the one who picks
what suits you best.

~~~
SparkyMcUnicorn
I'm going to give a shoutout to
[https://feathersjs.com](https://feathersjs.com)

I have yet to see a platform that's as flexible and robust, with the added
bonus of being real-time.

------
kelvin0
Yeah this 'bug' has also bit us once, when we first started using LogMeIn. At
first the fees were reasonable and we decided to go with them to do customer
support on their PCs running our software.

Every year since then, the cost has gone up significantly (ransomware?) and we
always bit the bullet, until we finally dropped them.

Beware: [http://www.computerweekly.com/news/450297646/LogMeIn-
custome...](http://www.computerweekly.com/news/450297646/LogMeIn-customers-
angry-over-unexpected-price-hikes-and-contract-renewals)

------
dismantlethesun
> They have no phone number to contact, no way to dispute this other than
> email — which they have ignored us for over a month now without replying to
> our continued requests. Trapped. Doomed. We have no further options.

Google Cloud support[1] comes with phone numbers to call at $400/month. It's a
minimum requirement for anything that's mission critical. I understand that
this poster was paying $25 a month typically, so it's not unreasonable that he
didn't also shell for the $400/mo support package though it left him in the
lurch when he had a time critical problem.

Realistically, I don't see why he didn't simply shut down the service the
first day usage numbers spiked. Something that's only costing $25 a month to
host, shouldn't have a huge opportunity cost when shut down.

Isn't Firebase part of Google Cloud?

[1] [https://cloud.google.com/support/](https://cloud.google.com/support/)

------
octo_t
I find it hard to reconcile this suggestion at the bottom of the article

> Always build your architecture in a way that will avoid becoming trapped
> into a specific service. Amazon’s AWS Lambda sitting between any services
> and your app is a strongly recommended path!

\- Build without depending on external services

\- You can do this by depending on this external service

~~~
chaostheory
I don't feel that relying on an external service is always bad as long as
there's no vendor lock-in with technologies that are open standards. Case in
point the managed postgresql and mysql services that Amazon, MS, and Google
all support. That said I'm extremely wary of Cloud Spanner and Azure Cosmos DB
or any specialized service that the big 3 offer.

------
dullgiulio
Using Firebase or similar is akin to opening a restaurant that serves take-
away pizzas from a nearby Pizza Hut, after nicely putting them in a porcelain
plate.

If Pizza Hut changes the pricing, you either close shop or adapt.

Who would open such a restaurant?

------
duncan_bayne
"They have no phone number to contact, no way to dispute this other than email
— which they have ignored us for over a month now without replying to our
continued requests. Trapped. Doomed. We have no further options."

I really don't understand why anyone expects anything different from Google
these days. Or possibly ever. "Don't be evil" ironically turned out to be a
particularly evil marketing ploy.

------
youngtaff
"This includes failed attempts which are blocked by their security rules."

Ouch!

~~~
kogepathic
_> Ouch!_

Yeah, no kidding. It means someone nefarious (or bored) can easily run up a
huge bill by just directing a ton of invalid requests to their API.

Many inexpensive hosting providers provide you with a lot of bandwidth (e.g.
Scaleway offers unlimited bandwidth at ~200MBit/s). Assuming you can saturate
the link, that's around 64TB per month of traffic. Quite the Firebase bill
they'd have.

~~~
bryanrasmussen
well it's not like anybody has the business model of doing denial of service
attacks that stop when the ransom is paid, because if someone did have that
business model this seems like a great way to make sure people will pay you. I
mean just do like enough to cost $500 and say pay me $1000 today or you will
pay google $10000 tomorrow.

------
yunolisten
Shame on them for not giving notice and a proper explanation of the change.
They were vastly under reporting the cost of providing the service. Consider
that they are a business.

Shame on you for not using resumable TLS sessions/Keep alive. You're hammering
their infrastructure. The change in how they meter usage is seeing you having
to compensate them for the resource they provide you.

~~~
awjr
Be very careful here. The developer may not have used TLS BUT any failed
authorisation attempts are also counted in the bandwidth.

So a bot net could absolutely wreck your credit card by just repeatedly trying
to access your API with invalid credentials.

~~~
yunolisten
> So a bot net could absolutely wreck your credit card by just repeatedly
> trying to access your API with invalid credentials.

You could argue that for pretty much anything being hosted, anywhere.

~~~
stemuk
No, because most self-hosted services are 10-20x cheaper than comparable SaaS
offerings. In the realtime space Firebase is particularly known for being
really expensive for the scalable plans (blaze plan).

~~~
yunolisten
> No, because most self-hosted services are 10-20x cheaper than comparable
> SaaS offerings.

This has nothing to do with the fact that it could be hit by a botnet, as per
the exact point I commented on, could 'wreck your card', it's simply a
question of scale.

~~~
kuschku
No. Most self-hosted services have no bandwidth costs, at all.

Or they have bandwidth costs around a dollar per terabyte. Which, even when
maxing your connections, would always be below your actual server costs.

~~~
yunolisten
If you read the fine print of the ones with "no bandwidth costs" you'll find
that service becomes throttled after a certain level of usage. These are
businesses, they have to make money to operate, they're not in this for
charity

~~~
kuschku
Dude, I’ve used 180 TB of traffic in one month on a 16$/mo server, and still,
no throttling.

I’ve read the fine print, and called them.

Online, Scaleway, OVH, do not ever throttle you.

Hetzner requires you to buy traffic, but there it costs 1$ per 1TB of traffic,
which is 1000x cheaper than Firebase.

~~~
yunolisten
> Dude, I’ve used 180 TB of traffic in one month on a 16$/mo server, and
> still, no throttling.

But legitimately using lots of bandwidth isn't the same as a DoS attack. Try
and remember that bandwidth isn't the only resource being used.

> Scaleway

In my experience they throttle your CPU usage after a while.

> Hetzner requires you to buy traffic, but there it costs 1$ per 1TB of
> traffic, which is 1000x cheaper than Firebase.

At no point did I suggest using Firebase was a good idea. I said it's always
cheaper to run your own services in the long run, and that they'd have found
out their own problems (see my first reply) sooner.

~~~
kuschku
But my point is that you won't even ever have an issue with overusing traffic
or CPU during a DoS, and the issue will be purely that your bandwidth will be
saturated.

Vs. AWS, Firebase, etc where your limit will be your bank account instead.

------
r1ch
I feel like a dinosaur running all of our services on dedicated server
infrastructure (OVH) but every time I see stories like this I'm glad I do.
Unlimited bandwidth, a fixed monthly fee, absolutely zero surprises. I think
far too many startups jump immediately into the cloud, where costs and
volatility are magnitudes higher than your own infrastructure.

------
forgottenacc57
Google Compute Engine is Bing.

One day they'll just realize they've lost, partly due to their shit attitude
to customer support, and close their public cloud.

FFS if you run a public cloud you've just damn well GOT to understand you can
never cut off the customer service for any reason, especially "because your
site got too busy".

What a bunch of PhDs.

The reason Google is so deeply unwilling to engage its customers to provide
support is in the DNA of the company - they are all PhDs, and how many PhDs
have the slightest idea of how to help and talk to a customer.

~~~
jhall1468
> The reason Google is so deeply unwilling to engage its customers to provide
> support is in the DNA of the company - they are all PhDs, and how many PhDs
> have the slightest idea of how to help and talk to a customer.

No they aren't lol. The majority of Google employees have B.S. degrees. The
reason Amazon is so good at this stuff is because, at their core, they do
everything they can to make the customer happy. Google doesn't really
understand that. Not because they are PhD's (like, what the hell does that
matter), but because customer service was never a thing they _needed_ to do.

------
bmpafa
I got burned by firebase in a similar way. I was using their worker queue
library and found my bill for a zero user app had gone from $0 to $100.
Couldn't get a straight answer from support on why the worker queue was eating
so much bandwidth.

Did some experiments and learned the queue downloaded every task in the queue
each time it started. The authors assured me that wasn't the case at first,
but eventually acknowledged that this was exactly how it worked.

Terrible experience overall, glad to have moved away from firebase.

------
piokoch
I am so surprised why people buy into some SAAS software that does simple
things instead of hiring 2 programmers from cheap location (somewhere between
India and Poland western border), getting Hetzner VPS or dedicates server
(even the second option is cheap, including a few administrator hours) and be
done.

It is not so hard to predict failure/buy out of some SAAS provider, it
happened so many times. I am curious why Venture Capital investors are not
looking at that problem closely.

~~~
bolololo1
Still cheaper for a single dev starting a startup to hook up to
Firebase/AWS/Azure than hiring 'cheap' devs from Poland.

~~~
camus2
> Still cheaper for a single dev starting a startup to hook up to
> Firebase/AWS/Azure than hiring 'cheap' devs from Poland.

Yes but you gotta be smart here. It's not like Firebase is the only database
out there. And once your product grows, and you hit Firebase limits which
forces you in the "pay as you go" plan, whatever cheapness you thought you
bought into quickly becomes a huge expanse.

~~~
bolololo1
Of course it's not and I feel sorry for the author, but being smart doesn't
mean that you know everything from dev to setting up servers. Most startups
are just quickly glued up. What the author and team could do after they
started growing is to start slowly moving to own infrastructure/backend. I'd
advise any startup anyways to start with Saas and when money starts showing up
then hire the remote guy to set up infrastucture.

------
CodeWriter23
$1/GB AYFKM? That's ridiculous. Should be about $0.01/GB.

~~~
motoboi
It looks like a good pricing model.

To me it seems they correlate bandwidth usage with system usage and charge you
using bandwidth as a surrogate of computing time, hardware and lahour (plus
the bandwidth).

------
alexfi
> They said in most cases it doesn’t make a big difference… unless you use the
> REST API.

So in the end it was a coding problem, because you forgot to add Keep-Alive =
true? If you use the provided SDK's it should be no problem then right?

~~~
Buge
I don't think Keep-Alive = true would be enough. The article also mentions TLS
session tickets would be necessary.

And it's hard to call this a coding problem if the requirements of the API
never specified that session tickets and keep alive were necessary.

~~~
alexfi
Yeah it's no coding problem on purpose, but sth you could actually fix. But
it's not good manner to not mention it from Firebase.

~~~
mining
It's absolutely good manners for Firebase to tell you "If you fix this thing,
your costs will go down."

~~~
izacus
> It's absolutely good manners for Firebase to tell you "If you fix this
> thing, your costs will go down."

I wonder how many HN startup founders go around telling their customers on how
to pay less.

------
mSparks43
i really really hope all you lot complaining you got burned are the same
people who downvoted my previous account out of existence when i said you
would get burned 12 months ago or so.

Sadly I reckon it was more likely the people who burned you who were doing the
down voting.

meanwhile, the servers i bought have now more than paid for themselves many
times over and our operating costs have fallen to nearly zero.

Feeling vindicated.

~~~
flukus
The reaction seems to be mostly apologetics. How did we get to the point where
running your own servers was such an alien concept?

------
coldcode
When the original vendor gets purchased by one of the big companies, it's time
to move on. If it's Google, run.

------
manishsharan
Yikes ! Do these(Google Cloud) guys not know that they are competing with AWS
and Azure ? Do they not realize support and billing practices and trust are
competitive advantages ?

------
samdung
We started using Firebase when it just launched, primarily because the top end
plan said $200 for (almost) unlimited use. For what we were doing at that time
this was a huge deal. Firebase have changed their pricing a couple of times
since then. All said and done it's still a great service and we use Firebase
in almost all our projects.

Maybe unrelated, but at the same time we started using Firebase, we started
using (then) another startup called Intercom (www.intercom.io). Intercom was
priced a flat $ 49 (or 50, do not remember exactly) then and it was a bargain
for us. But since then Intercom has changed their pricing so many times that
we ended up paying more for Intercom than for all our servers put together.
Intercom is again a great service but we stopped using it because it wasn't
worth the price.

------
dhawalhs
Something similar happened to me early this month. Embedly went from $99/year
to $99/month. I barely use it for a few image manipulation calls.

------
Dolores12
Going cloud is like selling naked options, the risk is not limited.

~~~
victorNicollet
I was wondering: who is really allowed to sell naked options (as opposed to
options that get killed if you can't afford a margin call) ?

~~~
patio11
Depends on whether it is OTC options versus a more bespoke product, but in
general, counterparty risk is something which investors have more or less
appetite for. If you're engaged with OTC options via the Options Clearinghouse
Corporation, you're engaged in a system with "approximately but not exactly
zero" appetite for counterparty risk.

If you're e.g. the sovereign wealth fund of Saudi Arabia and you call up
Goldman Sachs and say "Make me a market in naked puts on Citi, at-the-money,
with expiry in a year; I'm thinking about a billion dollars worth", I give you
excellent odds on them being able to put that trade together for you. I mean,
it will cost you a lot of money, but Saudi Arabia doesn't call Goldman Sachs
up and expect it to not cost them a lot of money, now do they.

------
Ygg2
Can't wait for the Spanner to take a price hike. You think $7786/year is bad?
Try 5.5 million.

------
troika
You could cut down costs or at least improve cost-predictability by hosting
something like [https://deepstream.io/](https://deepstream.io/) instead of
Firebase. The problem however is that you are still exposed to your cloud
provider's fees - if AWS or Google Cloud up their band-with cost, you're in
the same boat. Than again, the independence gained by running a server in a
basement comes with a very hefty penalty.

------
abrookewood
This is why Google continue to lag behind AWS and Azure for cloud services -
people live in fear that they will prematurely terminate the service or
abruptly change the price without warning. Throw in poor marketing, terrible
support and incomplete documentation and they have a perfect recipe for
failure. I really wish some senior people over there would get their shit
together. I WANT them to be a compelling competitor, but right now they are
miles away from being that.

------
Skylled
This seems like a pretty big stink to be dropping on the first day of Google
I/O, with it's spotlight on (among other things) Firebase.

I hope this article forces their hand.

------
Kiro
> Obviously our app being shut down and our users being cut off was not an
> option.

I wonder how many companies shut down due to this. My company is in a similar
situation (not a tech cost but still) and it feels unbelievable that we may
need to shut down even though we have so many customers. I never hear about
these stories though, as if all companies have a large buffer that can handle
these increases.

~~~
jstanley
Wouldn't it be better to port your app to another platform than to shut down?

------
gonational
They've been upgraded to the Conflagration plan.

------
troncheadle
I have to say... I don't understand what about Firebase's offering is SO
killer or unique. I feel like it would be very easy to roll one's own custom
'Firebase.' I'm unsure as to why anyone would ever choose to predicate their
businesses continued existence on another platform, especially one that can be
replicated in like, an afternoon.

~~~
ruslan_talpa
Can you replicate it and open source it tomorrow afternoon please? :) I'll pay
for your time.

While what they did was certainly a dick move, this type of comments
trivzializes and diminishes the work the firebase engineers put it.
Incidentally it also implies that google is full of stupid execs with money
that pay millions of tech that can be build by one dev in an afternoon.

~~~
troncheadle
I can definitely do that. I only collect payments up front, so fork it over
and I will have it for you tomorrow, likely even tonight.

I think what I said ("I don't understand what firebase does that is so killer
or unique") is very different from what you are reading ("firebase is useless,
firebase's engineers do stupid work, and google is full of student execs with
money that pay millions of tech that can be build by one dev in an
afternoon"). I think your characterization of my statement is more than a bit
unfair.

I'm still at a loss for what firebase is beyond a key value store with web
sockets. I think it would be a problem to scale a platform like this on ones
own -- maybe that's why to use Firebase? I've used it in hackathon settings
where I didn't want to roll my own backend, or to mock web-sockets pre Django
1.9 / Channels.

Still, if I had a business that depended on a key value store with web sockets
I would definitely build it myself.

------
soheil
Ally bank has an app that allows you to verify debit card transactions and put
a cap on recurring ones. It's called Debit Card Controls [1].

[1] [https://media.ally.com/2017-03-02-Ally-Bank-Launches-
Debit-C...](https://media.ally.com/2017-03-02-Ally-Bank-Launches-Debit-Card-
Controls-Mobile-Application)

------
px1999
Stories like there are why I won't consider Google as an infrastructure
provider for our SaaS, regardless of the technicals around their services.
Thank you to the original author/s for writing the post, it reinforces the
warning to avoid providers like this.

In HomeAutomation's case, the amounts are relatively small (though still
potentially massive to them). If their monthly infra spend for firebase
started out at $10k a month and went to $1m, would only way to have issues
resolved still be to hit the front page of HN?

Risk factors into our technical decisions, and trust factors into risk.
Particularly when billing is concerned, needing to wait a month is ridiculous.
Assuming the post is accurate, ignoring billing-related emails is inexcusable.
Any organisation this incompetent is one that you _should not trust_ with your
critical infrastructure.

------
Yizahi
Google is famous for not having any humans in support. One robot will write
abuse on you and another robot will happily oblige and ban you with no
appellation possible. Building whole business on one of such platforms takes
some "courage" :) .

~~~
victorhooi
What you said is quite untrue.

I know - because I am one of those humans in support _grins_. I work in the
support organisation for GSuite (formerly Google Apps). And I work across the
hall from the guys who support GCP (Google Cloud Platform). And there are an
entire army of us, helping support many, many enterprise customers.

Even our consumer products have real people manning support phones - I know,
because I've had to call them before (mostly Google Play Music issues - this
was before I joined Google, so I couldn't use my internal pull...lol).

Regarding your second point - abuse is a real problem for any cloud provider.
I know, because I happen to work in anti-abuse. It's a difficult problem to
crack, and there's a lot of smart people (at many companies) trying to solve
it. But please understand there are real humans at the other end, trying to
provide a good service - and it's not very kind to take cheap shots at them.

There usually are appeal processes available for nearly any kind of action
taken - if you were actually personally affected by an issue, or feel unsure
about how something was handled - please feel free to reach out to me. I can't
guarantee I can wave a magic wand and fix it, but I'm certainly willing to
reach out to see what can be done.

Disclaimer: I work for Google, but the views expressed here are purely my own.

~~~
Top19
I hired a freelancer from the Philippines on Upwork to work with me on some
Google App Engine issues because Google's documentation is incredibly
unhelpful at times and as well as bizarrely organized.

He was from the Philippines, and it turned out he was a contractor providing
"support" for Google Cloud. So I guess Google is outsourcing their support?
Isn't this something Dell tried to do in the 90's with disastrous results?

~~~
victorhooi
Regarding your first issue - I'm sorry to hear you had documentation issues.

There should be a huge "Send Feedback" widget at the bottom of every single
documentation page:

[http://i.imgur.com/MhWbY0W.png](http://i.imgur.com/MhWbY0W.png)

[http://i.imgur.com/Gu8GzLB.png](http://i.imgur.com/Gu8GzLB.png) (If you click
it, it will pop up this modal form, which also allows you to just screenshot
the offending part).

Did you try using this link on any of the pages that you found unhelpful, or
bizarrely organised?

I'd encourage you to use this - as all of this feedback comes through to us
via an internal system, and it does get reviewed (although I'm sure you can
imagine the volume).

Otherwise feel free to reach out to me if you have specific concerns and feel
more comfortable doing it that way, or you're unsure about the response you
received. (Although quite frankly, I think the form is best, as that gets
tracked, and anybody can jump on it).

~~~
Top19
Actually a PM I reached out to a couple of months ago did get back to me. I
was disappointed at first because I heard nothing from Google Next could be
uploaded online.

But Google did come through and they posted the images here:
[https://cloud.google.com/storage-options/](https://cloud.google.com/storage-
options/)

That had been my complaint...that the documentation was hard to get and that
when I saw the images presented at Next I understood things that I had
previously given up on.

------
archiepeach
Is there a large difference between using Firebase Database and another hosted
DBaaS, such as MongoDB Atlas?

I use Firebase Database but keep it arms length by avoiding any of the
proprietary features it offers. I use it exclusively as a JSON store which I
feel I can easily export at any time if I wanted to switch to another NoSQL
DBaaS.

A few people here are mentioning how it's crazy to depend on external services
for an app, but as an 'indie' dev, I can't spend time maintaining a secure,
high-availability database. Or a mail server when I can use SendGrid, or a
logging platform when I can use Sentry, or any of the other service my app
depends on.

~~~
benjaminl
Firebase is one of the very few products to offer controllable client data
replication for full offline first functionality.

For apps that have significant offline requirements, this functionality can
save a significant amount of time and significantly reduce the amount of code
that needs to be written and more importantly maintained.

One of my clients just recently seriously considered going with Firebase. The
choice came down to Firebase and Couchbase, which has an even better offline
functionality. After reading this post I am glad that we didn't going with
Firebase.

Of course Couchbase has their own problems. The supported version of Couchbase
is prohibitively expensive for small organizations and Couchbase themselves
strongly recommended against deploying the community edition. (The community
edition also artificially lags behind the supported edition 6 months.
Couchbase even delays critical security fixes that same 6 months.)

~~~
HodGreeley
Thanks for the kudos. The team works very hard to make the offline first
capabilities best in class.

I'm not sure who at Couchbase is "strongly" recommending against the Community
Edition. Of course we'd like to have people on the Enterprise Edition, but I
think the information is out there to make an informed decision. You should
only be getting a hard push away from CE if they really believe you'll need
the support. Anyway, this is good feedback.

------
geekme
Google is notorious when it comes to support.

------
mastef
Also discussed over on reddit :
[https://www.reddit.com/r/androiddev/comments/6bnup0/firebase...](https://www.reddit.com/r/androiddev/comments/6bnup0/firebase_costs_increased_by_7000/)

As noted there I've always received pretty good support from FB ( and recently
also Google ) for what it's worth.

------
sergior
I understand the convenience of not having to manage such product, but more
often than not is way more cost effective to rent some dedicated servers and
run infrastructure for fraction of the cost. It is fairly easy to create
resilient infrastructure nowadays and hw failures are very rare anyways.
People are being fooled by fancy menus to overpay a mile for mediocre service.

------
imb
Others have had similar complaints for a while and Google/Firebase has not
resolved them: [http://stackoverflow.com/questions/38959321/firebase-
databas...](http://stackoverflow.com/questions/38959321/firebase-database-
bandwidth-usage-growing-rapidly-even-when-when-the-database-is)

------
mattbillenstein
Eh, typical convenience versus control trade-off.

IMHO, the time needed to build and deploy your own thing using open-source and
open protocols is well spent when compared to being at the mercy of
BigPaasCorp when it comes to stuff like this.

A VM running nginx, an app in your language of choice, and postgres isn't that
hard to setup -- and the costs and scaling of such is well understood.

------
pranavsinghca
Does AWS (or I guess _maybe_ Azure) offer a Firebase alternative?

For an IRC style public chat app (the popular Firebase use case), is there a
simple-to-use AWS solution that doesn't involve setting up the
software/configuring it?

If not, then is setting up DeepStream on an AWS the best answer?

Serious question. Is there something available that can scale as easily?

~~~
pranavsinghca
Hmmm. Just checking and it looks like the EC2 data rates are $0.09GB up to
10TB and even less thereafter..

So.. if I understand correctly, Firebase is 10x more expensive??

It seems like it might be worthwhile to roll my own then..

------
davidlee1435
Relatively new developer here- does anybody have any advice for someone who
eventually wants to migrate a web app from Firebase to a custom server if I
start to see good traction on the project? I was initially drawn to Firebase
because I could iterate quickly, but now I'm not so sure if I should continue
with them.

~~~
mcescalante
You could take a look at Parse - it is a backend as a service that Facebook
open sourced that still gets active development
[http://parseplatform.org/](http://parseplatform.org/)

------
dbg31415
This is always the concern with services... and why I like containers over
building on some platform-as-a-service model.

I want to be able to insulate my projects from risks like this, worst case
they jack the prices and I move hosts.

This is also why I think AWS Lambda is cool, but why I'm really hesitant to
dive in. I like keeping my options open.

------
kalleboo
> They have no phone number to contact, no way to dispute this other than
> email — which they have ignored us for over a month now without replying to
> our continued requests. Trapped. Doomed. We have no further options.

Usually getting to the front page of Hacker News is the only way to get
support from Google. Hope it works this time.

------
Can_Not
Had anyone actually tried something like this and had success?
[https://blog.feathersjs.com/using-feathersjs-as-an-open-
sour...](https://blog.feathersjs.com/using-feathersjs-as-an-open-source-
alternative-to-firebase-b5d93c200cee)

------
kpennell
As someone who was considering using Firebase, I guess I should
reconsider...ugh...

------
piyush_soni
Hey Firebase people. I know you're here, reading this quietly. Gonna respond?

------
dutchbrit
Is there any open source product that offers everything that firebase does?

~~~
jazoom
From the article:

>Always heavily consider open source alternatives (something which didn’t
exist for Firebase at the time… but now alternatives like Horizon and
Backendless exist, for example.

~~~
brango
Looks like you have to pay for Backendless. I can't see an open source
version. Also, Horizon looks dead.

------
themihai
This is nothing new to me. I'm still working on my GAE/datastore
migration...Price spikes, platform changes and deprecation are real risks when
you use proprietary services/platforms.

------
tomc1985
Or they could have self-hosted their database (not necessarily FireBase) on a
VPS and never have to deal with _any_ of this BS.

But, as I'm sure I will be informed, that is just _too complicated_

------
ajohnclark
Sticking with my dedicated, almost went to GAE.. thanks for post.

~~~
H1Supreme
I like GAE for low traffic websites. The free tier is quite generous, and can
serve up a small business site (or 5), no problem. I run one for my personal
site, and use it as a proxy to my home server.

Only problem is: You gotta play by their rules. The databases, in particular.
Datastore was weird to get used to.

------
TaliaNa
Is it the same problem
[https://news.ycombinator.com/item?id=14358275](https://news.ycombinator.com/item?id=14358275)?

------
user5994461
Dirty hack: update the app to make a request per 10 minutes, instead of a
request per minute.

Costs goes down 10 fold. Problem solved!

------
solomatov
In general, you shouldn't use services which you don't understand completely,
can debug, and have source code for if there're other options. I think that's
why open source took over the world. It might seem to be more expensive, but
in the end, it's a better option. Even if the service you use have a great
support and fixes bugs instantaneously, it's better to have more control.

------
fimdomeio
Lesson learned here: Don't use SASS for hard to replace parts of your
infrastructure.

------
onmobiletemp
Firebase never made sense to me. I know nothing about web dev though. It seems
like you are handing over your project to some nebulous entity and they can
change the deal, stop updating or patching or raise the price. It seems better
to simply build the whole project yourself.

------
tschellenbach
Does anyone know if Firebase is profitable nowadays?

------
kukabynd
So glad I didn’t choose it for my product.

------
omgwhat
This is what fucking terrifies me about using anything Google. They love their
users, but they won't give two shits about any one user.

------
dboreham
Clearly someone needed a larger boat.

------
r0m4
What were you getting for 25?

------
internalfx
Have you tried RethinkDB?

------
systematical
Latest and greatest

------
marknadal
James and team has done an absolutely amazing job with Firebase, very much a
pioneer and industry leader. It makes me sad that Google then price gauges it.

I believe James is working on a solution/work-around for everybody, I chat
with him occasionally and trust him, so I still encourage people to keep using
Firebase. And that is coming from me (I work on gun, an OSS alternative with
graphs that you can self host -
[https://github.com/amark/gun](https://github.com/amark/gun)), so while I have
an incentive to get people to use gun, I have to keep plugging Firebase
because they are amazing (I don't say this about everyone, as I have said
pretty harsh words about other database systems. But there are some that
deserve legitimate respect, like Firebase, RethinkDB, and others).

------
korzun
They should have been monitoring the traffic within their infrastructure. I
can understand rolling something out for a couple of months but two years is a
bit excessive for this type of use case.

It's like going to a pump that counts two gallons as one for two years,
pretend you don't know any better then have the nerve to ask the owner for a
refund after the meter is fixed.

------
williamle8300
The price spike is dramatic... but is anyone actually surprised? Firebase is
holding all the cards and users are expecting them to not play their hand.
It's silly.

The solution is to build your own infrastructure. If you don't have the
resources to do that, then find a lower-cost competitor.

------
futun
... Remember when software wasn't cloud based?

