

So long AWS, and thanks for all the fish: FathomDB (W08) focusing on its new DB - justinsb
http://blog.fathomdb.com/53681308

======
ChuckMcM
This reads poorly.

FathomDB customers saw outages because the implementation Fathom did using
MySQL on AWS went out when AWS went out.

So in conclusion Fathom is going to build something which Oracle, IBM, SyBase,
and every other database company in the history of the planet has tried and
failed to do, build a relational database on a distributed infrastucture that
is more reliable than the underlying infrastructure? Uh, good luck with that.
Seriously, its Turing Prize material if you succeed.

I was thinking the punch line would be "Gee Netflix uses AWS and they didn't
go down so we're going to more what they did." I guess they are going to
compete with MongoDB and Riak.

~~~
justinsb
Thanks for the good-luck wishes. Obviously it's a big undertaking, but we're
seeing very promising results, so it's time to go all-in. Take that as you
will :-)

Incidentally, Oracle and IBM both have clustered database products, though
they can certainly be improved upon.

~~~
jacques_chester
Oracle and IBM do have clustered stuff and it's amazing, but it still tends to
assume closely linked networks of high-reliability hardware and have trouble
with very large, slowly linked networks of low-reliability hardware.

You might find Greenplum and VoltDB to be more interesting models to study.

The key question is: are you aiming at OLTP or OLAP? Because, as those boring
mainframe-era guys have discovered, it's difficult to serve two very different
purposes.

~~~
justinsb
OLTP. I personally believe Hadoop has got the OLAP problem-space covered.

~~~
jacques_chester
What are your most important non-functional requirements?

In particular, how do you see the tradeoff between durability and performance?

~~~
justinsb
You've totally hit the nail on the head there. Up until a month ago, it was a
fairly obsessive compulsion around performance. Durability and reliability are
features of the design, so it was about getting performance under the design.

However, the AWS outage has (we hope) put reliability back on the map. So
maybe now we don't have to be the fastest database, if we're the fastest
reliable database. That's a liberating change of perspective, which has been
behind many of the recent advances.

~~~
jacques_chester
Does durability absolutely trump performance, or are you prepared to trade
off?

For example, if durability is an absolute must (as it is for current
databases), then your main bottlenecks would be a) spinning rust and b)
network traffic to distribute copies of the data throughout the network. SSDs
will help this a lot, but already you're back where almost everyone has
already started.

If you're prepared to accept highly replicated in-memory copies as "durable",
you can already make your overall system hundreds or even thousands of times
faster.

So how do you define durability?

    
    
        1 copy on 1 disk?
        x copies on y disks in the same computer?
        x copies in y disks in different computers?
        x copies in RAM of y different computers?

------
unshift
sorry, this is probably tangential, but how is the AWS outage any different
from any other hardware outage, regardless of whether or not you actually own
the hardware?

why does it seem that everyone expects 100% uptime from a VM just because it's
in "the cloud"? shouldn't "the cloud" be used to make fault-tolerance even
easier, because you have access to multiple geographic regions and multiple
providers with little fuss?

they're still computers, and they're still bound to go down occasionally. i
don't see how running leased VMs somehow absolves you of doing basic
operations work and guarantees a bulletproof experience. regardless of where
you host it, writing a nicely distributed and fault-tolerant system is and
always has been difficult. the only part that's gotten easier is finding rack
space.

~~~
justinsb
With physical hardware, the basic failure characteristics etc are well known.
When the hardware is virtual or abstracted as it is on the cloud, you have no
choice but to rely on the promises made by the provider. If you can't build a
reliable system on the abstraction provided by the cloud, you can't reliably
use the cloud. But with a good cloud, you can architect a reasonable solution
based on the provider's promises.

However, if the provider doesn't keep those promises, then all your hard work
and calculations go totally out the window.

At that point, you have to figure out a way to run a database on a system that
effectively offers you no guarantees. That's what we're working on.

~~~
dialtone
For your application though a failure is a failure in any scenario. EBS went
down in one zone and was slow in a couple others for a few hours, it's a bad
downtime but some services survived the problem relatively unscathed. This
suggests that something could have been done to avoid any trouble at all.

If the datacenter you are in has a slight conditioning problem this summer and
10% of your drives breaks down due to excessive heat, how quickly will you be
able to re-provision the data center?

About a week after AWS outage the italian ISP Aruba had a UPS failure due to a
fire in the UPS room and the entire datacenter switched off automatically
during the night. For the following 8 hours that datacenter was off for every
customer. How would your new solution handle such a situation?

Designing a hot standby replication solution for MySQL/PostgreSQL that works
across regions seems easier to me rather than implementing a database from
scratch that should solve a very complex problem.

~~~
justinsb
Certainly it is possible to engineer a hot standby solution for MySQL
databases using DRBD, or synchronous replication etc. There's a choice between
engineering an endless series of fixes like that, or re-examining the problem
space and building a new database. After years of doing the former, we chose
to examine the latter, and we're seeing good indications from that approach.

~~~
jacques_chester
I have misgivings because you are not the first to try and re-examine the
problem space. A lot of the endless series of fixes have arisen from just such
attempts.

I want you to succeed, but you're dealing with seriously hairy deep magic
issues that the best minds in the industry and academia have been chipping
away at for decades.

~~~
justinsb
A healthy skepticism, then: nothing wrong with that. It's a speculative
project with a big payoff if it succeeds, and I wouldn't ask you to believe
until we've shown a working product. Your good wishes are appreciated though!

------
Joakal
This author blames Amazon for a poor SLA yet continues using their service?

What cloud provider offers a more reasonable SLA?

~~~
justinsb
I'm the author. I think you're conflating the SLA with the guarantees the
cloud offers: one of the points I tried to make was that an SLA doesn't offer
you any real guarantees; they only really benefit the service provider. If
every SLA gave you a "50 times" penalty, as FathomDB is paying, then they
would be interesting, because at that level the provider has to take uptime
seriously.

In terms of who offers a better cloud, clouds based on OpenStack are 'the
great hope'. IaaS is about commoditizing infrastructure, so open source seems
necessary; rather than AWS's very closed, secretive approach.

~~~
sunchild
I deal with SLAs in my job all the time. After over a decade negotiating
uptime SLAs, I'm firmly convinced that the best protection against downtime is
a huge, vocal userbase, and the best remedy for downtime is an unqualified
termination right and refund of prepaid fees.

An SLA (even with severe penalties) is certainly setting up the right
incentives on paper, but in practice it has little effect on the results. Why?
Because enforcing a promise on a piece of paper is bothersome, expensive and
sours the relationship.

Also, saying you'll pay 50x penalty is only half the story. What are the
triggering events that entitle customers to payment? Is the SLA measured end-
to-end (i.e., from the user's endpoint to yours)? Is the SLA penalty
cumulative? Can your customers actually profit from your downtime (i.e., you
would end up owing them more cash than they paid you)?

Without more details, your post suggests that each fraction of a second of
downtime experienced by any single user of your service entitles them to a
credit of 50x what they paid for that fraction of a second. Is that your
policy?

~~~
justinsb
Thanks for the expert input. This is exactly why we don't have a SLA at
FathomDB, because I don't think that they're worth the paper they're written
on. It seems like you almost agree (?)

Most cloud providers (including FathomDB) charge by the hour anyway, so you
have termination rights for any reason, and there are no prepayments to
refund.

We think that 'doing the right thing' is the only workable policy, and it
shouldn't require an outcry from a vocal userbase. We should have handled the
outage better, and we're making a big payment to demonstrate that. We're not
going to let any new customers use a system where we can't be confident in its
uptime (MySQL on AWS); we're building a new system in which we can be
confident.

~~~
sunchild
Dead on. I went back and re-read the article, and yes – we are in agreement.
It's about doing whatever it takes to keep the relationship with your
customers on an even keel. I couldn't agree more that SLAs are generally bad
for customers, even though the perception is the opposite. I can't tell you
how many times I've had a client demand an SLA from their vendor, and when
that SLA arrives, it reads like a commitment, but actually it's impossible to
fail. And these almost always are wrapped in "exclusive remedy" language,
which quite literally means the customer would be better off without it. The
devil is in the details.

Cheers!

------
soamv
Can you actually tell us a bit about the new DB you're building? (What is it
designed to run on, what kind of failures are you aiming to tolerate, and so
on.)

~~~
justinsb
Sorry, not too many details yet - let us finish it first :-) Commodity
hardware though, and quorum-style failure-tolerance characteristics.

