
Trello has moved to AWS - gecko
http://status.fogcreek.com/2012/10/fogbugz-kiln-and-copilot-continue-to-run-on-backup-power-trello-moving-to-amazon-aws.html
======
gecko
There's a little bit more information in the last status post, plus a picture
of Fog Creek cofounder Michael Pryor hauling 5-gallon jugs of diesel up 17
flights of stairs. (We have other employees down there, too, and I know Stack
Exchange's NYC employees were excited to help out, so they're probably there
by now, too.) [http://status.fogcreek.com/2012/10/diesel-bucket-brigade-
mai...](http://status.fogcreek.com/2012/10/diesel-bucket-brigade-maintains-
services.html)

~~~
grey-area
I'm sure you've thought of this, but now that you're up and running, isn't it
possible to buy/hire a pump, run it off the electricity from the already
running generators, and pump it up from street level instead, instead of
walking up stairs with buckets?

~~~
Dobbs
The pump is underwater.

~~~
culturestate
OP is suggesting using a new pump to pump from the trucks, in lieu of buckets
- not the pumps that are submerged.

------
rdl
"Trello is in the process of being moved to Amazon AWS where it will not be
affected by further data center issues."

I guess that's called the power of positive thinking?

~~~
sturadnidge
Oh, the irony. Interesting they didn't move to Azure though.

~~~
dagw
Unlike their other products trello isn't .Net based. It runs on a
node.js/mongoDB/Redis (and I'm guessing Linux) stack . My understanding was
that they're using Trello as a test platform for playing with and evaluating
different new technologies.

~~~
smiler
You can run Linux VMs on Azure.... [http://www.windowsazure.com/en-
us/pricing/calculator/?scenar...](http://www.windowsazure.com/en-
us/pricing/calculator/?scenario=virtual-machines)

~~~
bdavisx
"Let us redefine progress to mean that just because we can do a thing, it does
not necessarily mean we must do that thing."

------
Killah911
Would love to hear about what it took to transition to AWS. Any chance Joel
would divulge some of the driving factors (reliability being one)? How about
cost? Unlike my paltry webapps, trello gets a lot of activity and no current
revenue stream, so I'd love to know more on what the cost difference is
between running your own vs in the cloud. How long did it take to move over?
Are the old servers being used as backups in the event Amazon has their very
rare aws outage?

I know that's a lot of questions, but heck it would be an article I'd love to
read and something I believe would be very relevant to the community as well!

~~~
thedufer
> Are the old servers being used as backups in the event Amazon has their very
> rare aws outage?

AWS is essentially acting as a backup of the old servers, as the datacenter is
in danger of losing power.

That said, the cost, reliability, etc. had been carefully thought out, albeit
as a somewhat longer-term project than it ended up being. More info will
certainly wait at least until the current state of things is cleared up -
Fogbugz and Kiln are still in the datacenter.

~~~
Killah911
Thanks for the info. I know things must be pretty crazy right now. Quite
literally in the trenches. Just goes to show what an amazing team behind an
amazing product can do. Kudos!

------
citricsquid
Not entirely related, but regarding the backup fuel situation: how long does 1
"bucket" of fuel keep the power going for and what is the cost of 1 bucket of
fuel?

~~~
gecko
This information is from earlier today, so I don't know how accurate it is,
but:

    
    
      - They burn 44 gallons per hour
      - The tank adjacent to the generator holds ~400 gallons
      - Gas is currently VERY expensive, but I don't have a real number for you at
        the moment.

~~~
jspthrowaway
Diesel, on-highway in New England, is averaging $4.205/gal[1] (probably more
in the city, especially more in a disaster situation). Applying your
information to the above question and assuming they can get just shy of five
gallons per bucket, it's probably a fair estimate that each bucket carries
more than $20 of diesel fuel and accounts for seven minutes of generator time.
So, about $3 per minute or 5.1 cents every second.

Really puts things in perspective.

[1]: <http://www.eia.gov/petroleum/gasdiesel/>

~~~
lmm
I got a similar feeling at my last^2 job when we had a client run a superbowl
ad, for which they were paying a million dollars. That works out as something
like $30/ _millisecond_.

------
rurounijones
Be interesting to see how this was migrated, method, what had to be changed,
lessons learned etc.

Would also be great to see a follow-on report after a month or so of lessons
learned from hosting on AWS vs own infrastructure.

I will also be keeping an eye out to see if it remains as responsive.

~~~
Killah911
What would be really interesting is if they thought of a Hybrid-Cloud
approach. I recall Joel talking about adding a new server every week due to
Trello's ever growing popularity during his presentation at Startup School.
Being able to spool up on-demand should really help with performance or for
times when the datacenter is flooded. But wouldn't it be more cost effective
to run it on your hardware during off-peak times and spool up some new servers
on AWS when performance starts to lag?

Disclaimer: I'm a Trello fanboy, it's like having container classes in STL,
only more visual!

~~~
gecko
Technically, it _is_ running hybrid now, albeit for somewhat silly reasons:
the mail server is still at Peer1.

~~~
trimbo
Yeah it's hard to migrate email given the anti-spam measures ISPs use like IP
reputation.

Good luck. Maybe a quick migration to Sendgrid for that system? (Edit: or
Amazon SES, of course)

------
modarts
Unrelated to the post, but I hope you guys are all doing okay out there.

I have a huge amount of respect and appreciation for the level of commitment
you have for your customers, despite these horrific circumstances. Stay safe!

------
Tyrannosaurs
I'd be interested to know whether there has been a significant surge in demand
for AWS since Sandy hit.

Having the cloud as your back up makes a lot of sense but if everyone did it
could it cope with the additional demand? Sure the AWS infrastructure is
massive but I'm guessing the load that would be dumped on it if a significant
chunk of New York's web hosting was suddenly moved across would be massive.

------
mgkimsal
Haven't we seen more AWS outage issues than "east coast shutdown"-level storms
over the past, oh, say, 5 years?

EDIT: Thinking now this may be a temporary move, not a permanent one. As such,
it probably makes sense to have a tested process to be able to move between
cloud/dedicated/whatever as quickly and painlessly as possible.

~~~
acdha
We've seen many people who failed to read AWS' documentation and put all of
their services in one data center. If you follow Amazon's guidelines and used
multiple AZs (i.e. if you use RDS check that box), you've had very little (~20
minutes) downtime; if you went multi-region it's even less.

It's not like this is a well-hidden secret - head over to AWS's whitepaper
section

[http://media.amazonwebservices.com/AWS_Cloud_Best_Practices....](http://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf)

“Be a pessimist when designing architectures in the cloud”

[http://media.amazonwebservices.com/AWS_Web_Hosting_Best_Prac...](http://media.amazonwebservices.com/AWS_Web_Hosting_Best_Practices.pdf)

“As the AWS web hosting architecture diagram in this paper shows, we recommend
that you deploy EC2 hosts across multiple Availability Zones to make your web
application more fault-tolerant.”

~~~
griffordson
Do you know something that Netflix doesn't? They avoid EBS dependencies for a
reason:

[http://techblog.netflix.com/2011/04/lessons-netflix-
learned-...](http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-
aws-outage.html)

[http://techblog.netflix.com/2012/10/post-mortem-of-
october-2...](http://techblog.netflix.com/2012/10/post-mortem-of-
october-222012-aws.html)

~~~
acdha
Avoiding EBS - or at least planning seriously for how you'll handle failures,
as with any storage system you use - is a good idea but it's unrelated to my
point.

------
mgamer
I wonder, how technically can you move something like Trello to AWS in such
short period of time? Do you think they had a plan prepared for such
contingency?

------
amalag
The real question is "is it permanent?"

------
dancryer
Did they move to AWS to ensure that their services go down more efficiently
next time there's a hint of a natural disaster?

~~~
zwily
You do know that AWS did not go down during Sandy, right?

/me knocks on wood

~~~
Xylakant
Well, but it's not like AWS didn't have its share of downtime in the last
couple of weeks. Back then the recommendation of the arm-chair-admins was to
move to dedicated hardware you own. Oh the irony.

~~~
pyre
Not necessarily. If they only have their setup in a single AWS datacenter, and
_it_ gets hit by a hurricane, then you still have issues.

The real issue is geographic redundancy, regardless of using AWS or dedicated
hardware.

~~~
Xylakant
With the last round of failures some of the people affected reported that they
couldn't start up instances in other datacenters since those were overloaded.
Still, I agree - the underlying problem is cross-datacenter redundancy but
that's a very difficult problem to tackle and might just not be economically
feasible.

