
The Trello Tech Stack - elehack
http://blog.fogcreek.com/the-trello-tech-stack/
======
edw519
_Use things that are going to work great in two years._

This is one of those remarks that is obvious to someone who knows what it
means but mysterious to someone who doesn't. So what does it really mean?

\- Don't use a technology that, no matter how good, is just too niche for
widespread adoption?

\- Don't use a technology that we'll have trouble finding programmers to
support?

\- Look for stuff that will take better advantage of impending improvements in
hardware and communications?

\- Avoid frameworks for which support may wane?

\- Avoid technology that we may have to support ourselves?

\- Avoid anything proprietary whose owner may disappear?

\- Make sure that whatever you choose, it will work well on mobile
technologies, even as native apps?

\- Choose technologies abstract enough to minimize our costs but not too
abstract to be inefficient?

\- Any combination of the above?

\- What considerations have I missed?

~~~
frabcus
My (minimal) criteria is it has to be packaged in Debian.

A couple years ago I considered both MongoDB and CouchDB immature for that
reason. The recent confusion of CouchDB/Couchbase etc. shows that was a
reasonable view.

~~~
mnutt
I'm not sure when they started, but 10gen now packages and distributes their
own MongoDB Debian and Redhat packages. It seems like the right move, so that
they're not beholden to the update schedule of the distros.

~~~
wavetossed
Yes!!! The development cycle of software like MongoDB, RabbitMQ and so on, is
much faster than that of Debian or any other Linux distro. The Debian package
is fine for dabbling or low volume use, but for any serious app, you MUST go
direct to the developers and use their latest stable release. That is what
they support best on their mailing lists, and that is where most bugs are
already fixed.

A lot of software development teams are releasing their own RPM and .DEB Linux
binary packages for just that reason, to encourage people to use up to date
packages instead of the stale OS distro packages.

In a way, it's rather like security updates. Who would refuse to install
security updates because it's not part of the Ubuntu 10.4 LTS release? Almost
nobody even thinks of doing that. So why would you use old obsolete releases
of mission critical software?

~~~
meatmanek
> why would you use old obsolete releases of mission critical software?

"If it ain't broke, don't fix it"

Because it's mission critical, and you can't afford for it to break. Once you
hit a certain complexity, upgrades almost always break something:

APIs change. Undefined behavior changes. New bugs are introduced. A feature
critical to your app starts performing worse. The above changes break
something else you depend on (libraries, proxies, etc.)

Upgrading to a significantly changed version of a mission-critical
app/library/language is a lot of work, and is sometimes impossible: many
projects couldn't be reasonably ported to Python 3 if they wanted to; a lot of
important libraries don't work on Python 3.

This is exactly why bug and security fixes are often backported into old
versions. Python 2.5 is still receiving security patches. Apache 1.3 was
maintained for years after 2.0 was considered stable.

~~~
mnutt
Yes, especially with a database you can't just pull in new updates and expect
them to work. It involves reading the release notes and doing a lot of
testing. (much of it is automated)

------
davesims
Not to pick on him, because I think the overall received wisdom on new tech
has shifted and morphed a great deal over the last few years, no doubt
including Joel's...but I can't resist pointing out that, my how times have
changed...

<http://www.joelonsoftware.com/items/2006/09/01.html>

In other words, it's nice to see Joel greenlight something like this, I'd say
it's kind of a sign of the times in terms of the industry's overall comfort
level with what would be termed 'hip' technology, or 'new' or whatever moniker
you want to attach to it.

~~~
bigiain
To Joel's credit, that was over 5 years ago - which is, like, 35 dog-years ago
(which are even shorter than tech-years).

~~~
davesims
Agreed, completely, just pointing out, again, time's is different...

------
nirvdrum
This part caught my eye:

"The Socket.io server currently has some problems with scaling up to more than
10K simultaneous client connections when using multiple processes and the
Redis store, and the client has some issues that can cause it to open multiple
connections to the same server, or not know that its connection has been
severed."

I wonder if they ran into redis's hard-coded 10k connection upper-limit. As it
turns out, their configuration for "unlimited" connections actually has a cap
of 10k. I believe in master this is going away, but if you need more than 10k
connections on redis <= 2.4, you need to manually patch the daemon, in case
anyone else runs into this.

------
trotsky
How vulnerable are all javascript approaches in terms of injection type
attacks? Do apps like node and mongo effectively prevent them, or is it still
possible to shoot yourself in the foot? I've read some off hand comments along
the lines of "as long as you're not a total moron you have nothing to worry
about", but that sounds a lot like what was said about sql injections and xss
before exploiting them went mainstream and it turned out everyones apps were
filled with them. Has anyone audited a real world app built with a stack like
this and come away with any experience to share?

~~~
radagaisus
You CAN do injections to MongoDB code. An injection is basically 'allow user
input to interfere with code' so for mongoDB assuming the query is a string
you can do '{name: ' + user_input + '}'. and user_input, without sanitizing it
(which is simpler, just converting it to a string) could be: {'$where': ...}

[http://www.mongodb.org/display/DOCS/Do+I+Have+to+Worry+About...](http://www.mongodb.org/display/DOCS/Do+I+Have+to+Worry+About+SQL+Injection)

~~~
rjrodger
Unlike SQL, which you have to build as a string, the natural approach in
JavaScript (even for junior devs) is to use an object literal to build the
query. And then you get escaping for free.

------
radagaisus
Just had to share this:

Our stack is Redis, MongoDB, Nginx, SCSS, HAML, Coffee, Rails and NodeJS. I'm
extremely happy with these choices.

Recently me and a friend did a small weekend project: www.bubblefap.com (nsfw)
The design and code is a homage to ugliness. We only used PHP-ActiveRecord and
that's it. and I had so much fun!

I just hacked away! I was cowboy coding, hacking away, and I didn't need to
think about frameworks and architecture and integration with fog and hacking
Rack to support flash uploads. Oh, good times :-)

------
jcromartie
Node.js, Redis, _and_ MongoDB? They've gone full web-scale!

In all seriousness, though, it looks like they are using Redis for exactly the
right reasons, and the larger architecture is pretty much the definition of a
sane forward-looking design.

------
padobson
I think this is the best write-up I've seen of a full Javascript/CoffeeScript
stack.

I've always been hesitant to get too far away from my LAPP(ython) stack, but
I'll almost certainly be hacking something together with these components to
see how I like writing everything in CoffeeScript.

------
DanielBMarkham
Nice job, guys. I love this architecture. This looks a lot like my dream stack
for future web work.

Would love to hear some more about the "bleeding" experiences you've had.

~~~
dodger
Socket.io: <https://github.com/LearnBoost/socket.io/issues/686>

MongoDB (on FreeBSD): <https://jira.mongodb.org/browse/SERVER-3927>

Redis: <https://github.com/antirez/redis/issues/91>

. . . are a few.

~~~
jashkenas
dodger: I'd be curious to know where y'all bled over CoffeeScript or Backbone,
if you don't mind sharing.

~~~
dodger
Actually, those have been great (and thank you for making them)! We had an
outage during beta because somebody forgot to wrap an array comprehension
assignment (CoffeeScript) and we didn't test right. Backbone has been really
good, and while we had some complaints, I think many have been resolved in
more recent versions - we're on something ancient.

~~~
robocat
> "we're on something ancient"

Isn't that one of the major drawbacks of using a bleeding edge Tech Stack?
Albeit mitigated by choosing to use code that is easy to comprehend/maintain
(like backbone!)

What do you think you will do? E.g. With backbone: Merge up, status quo --
maintain your branch, change libraries, or something else?

~~~
dodger
Agreed, it made some necessary, non-backward-compatible changes as a result of
being such a young project. I think it is likely that we will upgrade to a
newer version soon.

------
drewda
"We custom-built a client-side Model cache to handle updates and simplify
client-side Model reuse."

Is this related at all to backbone.iosync.js?* Or, if not, is it something
that Fog Creek will be open to sharing in the future?

* <https://github.com/logicalparadox/backbone.iobind>

------
techscruggs
Question: They tout MongoDB's "generally fast" read speed. Any idea what this
is in relation to? In "fast", are they suggesting that it is faster than a
typical RDBMS? If so, does anyone know of any supporting benchmarks or
breakdowns of how they accomplish this and to what extent?

Its news to me.

------
geekfactor
Does anyone know of any open-source projects out there using a similar stack?
I'm particularly interested in learning more about client-side view rendering
with Backbone and Mustache... server side can be any of Node, Rails or Django.

~~~
jwarzech
I've been using CoffeScript and backbone.js a lot lately, it really has
changed my approach to web apps (keep the server light, just transfer json
back and forth). The approach that has worked the best for me is to keep pages
such as the landing and authentication as traditional rendered pages and save
the client side rendering for the user-authenticated 'meaty' parts of the app.

If you want to look a couple really-small examples I have a couple apps on
github I used for learning backbone.js/coffeescript. They both use the
underscore.js template engine.

<https://github.com/jwarzech/realtime_hn> Backend is sinatra, takes a HN story
url and polls the comments in reverse chronological order.

<https://github.com/jwarzech/inspire_board> Backend is rails, polls images
from dribbble and lets you save your favorites.

~~~
randito
I like the idea of doing authentication and OpenId in a traditional web stack,
while doing the "app" portion in a javascript backbonejs stack.

That lets you use existing libraries (Devise, etc.) for the authentication and
once that's out of the way, you can go have "fun" with coffeescript and
mustache. :)

------
praxxis
"We use a really excellent async library"

Does anyone happen to know if this is an open source or in house library?

~~~
gecko
It's open-source: <https://github.com/caolan/async>

~~~
wildmXranat
Yes, I've seen it on HN before and it's the bee's knees.

------
alexhaefner
This is a great write up. We have a very similar stack for a separate team
based project we're working on. The only difference is the front end stack,
and we have tornado sitting back on the backend to serve http requests when
needed. What's the benefit of Mustache/Backbone over just writing your own
client side templating, in a practical sense? Last time I used a KVO
framework, there was a performance penalty that I did not like. Does Backbone
have any performance issues? Or even mustache? What's the trade off of having
these frameworks versus generating custom tempting code?

------
apg
This post raises a few questions for me... and perhaps some one more versed in
these stacks can provide answers.

\- Does the use of CoffeeScript alleviate the MC Escher-esque quality of
callbacks within closures involved in working with Javascript on both the
server and client (and the data store)? I can totally see the appeal of
CoffeeScript's syntax. Giving the programmers something different to look (and
learn) at probably provides some cognitive benefit as well.

\- Is there an acronym/name (ala LAMP) for the Node/MongoDB/Redis stack?

~~~
jeswin
1\. CoffeeScript doesn't alleviate this yet. I doubt will be much progress on
this until we find a good way to implement defer/async.
[https://github.com/jashkenas/coffee-
script/issues/350#issuec...](https://github.com/jashkenas/coffee-
script/issues/350#issuecomment-322321)

I am writing a sizable Node app myself. In the end, you just get used to the
callback style.

~~~
apg
I am assuming that CoffeeScript should be able to add new syntax where the
need arises, as the number of CoffeeScript users is probably still pretty low
and able to adapt to change.

From what I can tell, CoffeeScript - and the Javascript/Node world - would
really benefit from something like F#'s workflow/computation-expression
syntax, which will take a fairly straight-forward readable statement and
behind the scenes de-sugar the hell out of it into a bunch of closures.

[http://en.wikibooks.org/wiki/F_Sharp_Programming/Computation...](http://en.wikibooks.org/wiki/F_Sharp_Programming/Computation_Expressions)

EDIT: search for "De-sugared syntax" on this page for an example.

~~~
jashkenas
That's correct -- because we compile to "standard" JavaScript, we're at least
somewhat comfortable introducing significant changes (even to the syntax)
where desirable. Even if you never get around to updating older pieces of
code, all of the compiled JS continues to be fully compatible and
interoperable with the newer stuff.

------
prodigal_erik
"Client-side MVC" is kind of unfair to backbone.js. I haven't confirmed it
myself, but I've seen people explaining how it's also suited to server-side
rendering, which means it's not ruled out for competent authors doing
progressive enhancement.

[http://lostechies.com/derickbailey/2011/09/26/seo-and-
access...](http://lostechies.com/derickbailey/2011/09/26/seo-and-
accessibility-with-html5-pushstate-part-2-progressive-enhancement-with-
backbone-js/)

------
djb_hackernews
Great write up. Hopefully that ephemeral data in Redis can scale.

They could have used something like Pusher for about half of their
implementation (the websockets, message pushing, polling, etc)

I bring this up because I built an OSS Pusher clone[1], for those people that
want to deploy their own, built on the Play! framework.

[1] <https://github.com/danbeaulieu/PushPlay>

~~~
jurre
There's also this ruby implementation that plays nice with all the pusher libs
<https://github.com/stevegraham/slanger>

~~~
djb_hackernews
Yes, I built my version almost in tandem with Steve. Pusher hosts their js lib
on github under the MIT license which made building these things supremely
easy.

------
eaurouge
If you could do this over again, would you still choose socket.io for
websockets, or would you go with something else like PusherApp?

~~~
dodger
Today I'd consider going with a service, but first I'd see how
<https://github.com/LearnBoost/websocket.io> looks. Most of our problems have
been with abstractions in the socket.io client and scaling issues with server
process chatter. The chatter shouldn't be necessary with websockets-only.

~~~
Rauchg
With <https://github.com/LearnBoost/engine.io> the abstractions are greatly
simplified, which is what I'm announcing today.

The socket.io codebase has been shrunk dramatically, and it's as a result
easier to scale/maintain.

~~~
dodger
That's great!

------
wowzer
When I first saw Trello I "felt" that something cool was going on under the
hood. So I did a little poking to see what JS technologies you guys were
using. The piece I was most excited to learn about was backbone. Had never
seen it before and was really impressed by the space it was filling and what
it's capable of. Thanks for the write up.

------
barmstrong
Great writeup!

Can you guys share what (if any) test suites were used? Was curious on this
point.

------
karterk
I have read about various issues in using HAProxy and socket-io (websockets
mode). I am currently working on a project that's heading towards that
direction - anyone has anything to share on that front?

~~~
feralmoan
nginx supports http1.1 reverse proxying as of a few weeks ago
<http://wiki.nginx.org/HttpProxyModule>

Mikito Takada (Zendesk) has some really helpful information regarding
socket.io + haproxy specific workarounds via
[http://blog.mixu.net/2011/08/13/nginx-websockets-ssl-and-
soc...](http://blog.mixu.net/2011/08/13/nginx-websockets-ssl-and-socket-io-
deployment)

~~~
karterk
Thanks. Looks like it's available only on the development version - I wonder
if it's production ready...

Also - I took a look at the article - it does not mention using a total node
approach of using something like node-http-proxy for load balancing. Any idea
on such a set-up? I need HTTPS as well.

~~~
mixu
A pure-Node approach is a lot simpler: you can just use Node's HTTPS (if you
don't care about load balancing) and attach Socket.io to it. Just don't put
your Socket.io server behind Nginx if you want WebSockets support.

With load balancing, I would recommend going with Stud/Stunnel and HAProxy.
Terminating SSL with a specialized piece of software is nicer (separate SSL
overhead to another box), and using a separate load balancer allows for more
flexible options, e.g. serving static assets from Nginx. There is nothing
wrong with node-http-proxy, it's just that these two projects have been around
for longer and are better understood from an ops perspective.

You cannot use round robin load balancing, you need to have at least IP-based
stickiness with Socket.io for now. Well, you can round robin if you use the
Redis store but you'll run into inefficiencies with
<https://github.com/LearnBoost/socket.io/issues/686> . I'll do a bit more
coverage on other SIO deployment-related issues once I get the chapter on
Socket.io finished for my (free) book in a few weeks.

If I were starting now, I'd just ignore the Flash sockets transport since it
makes the whole stack more complex due to not looking like HTTP to load
balancers, and start with Engine.io / WebSocket.io to take advantage of their
simplicity.

------
jslabaugh
Great write-up... It made me want to go play with these exact components!

------
jroseattle
Love the stack, already working on something similar (mine involves nginx,
though.) Given we haven't deployed, it's refreshing to see a similar diagram
from those already out in front.

------
wuher
Great writeup, I wish more companies did this: be open about their tech stack
and experiences.

This inspired me to write about how I think, they could've gone even further:
[http://wuher-random.blogspot.com/2012/01/single-page-web-
app...](http://wuher-random.blogspot.com/2012/01/single-page-web-apps-and-
rest.html)

~~~
dodger
Yep! We intend to use our shiny, well-organized REST API for future
development and probably port existing functionality to it, too!

------
Smrchy
I am wondering where and how you terminate the HTTPS connections for
websockets. Can you share some details on that?

------
strictfp
" sending changes to Models down to browser clients". _Down_ to the client? He
draws the stack with the client on top and the storage at the bottom, and then
says 'down to the client'? I have a collegue who also does this. I never
figured out why. Can anyone explain possoble reasoning behind this wording?

~~~
Flenser
Because from the client downloads assets and we are more used to thinking
about what happens from the client's pov.

Alternatively, it could be for the same reason people usually say they're
going "up" to a big city, regardless of the direction of travel. The server is
the hub and all the clients are _down_ from it. This possibly dates back to
when towns/fortifications were usually located on hills.

------
sifi
Awesome job guys. I'm really digging this stack. It is giving me motivation to
learn more about Backbone.js

------
crescentfresh
Does it just go without saying these days that jQuery is required? Trello uses
it, as well as jQuery UI, date.js, and highcharts.js far as I can tell.

I mention it because they mention other parts of the client stack (Backbone.js
for ex), but not said libs.

------
swah
It seems Socket.io has not had any contenders yet..

~~~
yesbabyyes
Socket.io is a great library. When it comes to a contender, I see Server-side
Events [1] as a contender to WebSockets. I feel it's a simpler architecture
which fits HTTP/REST better than WebSockets in the general case. I built a
couple of small experiments/examples using CoffeeScript and Redis [2]; the
middleware.coffee in particular is designed to be used together with REST,
where if you send "Accept: text/event-stream" you will get updates to the
resource(s) on that URL.

1: <http://dev.w3.org/html5/eventsource/> 2: <https://gist.github.com/1560654>

~~~
josephg
Unfortunately, eventsource isn't supported by any current version of IE. It'll
be awhile before ES can replace your AJAX/websocket libraries.

<http://caniuse.com/#search=eventsource>

~~~
yesbabyyes
On the plus side, it's supported by Opera Mini and iOS.

For IE and others (Android, I'm looking at you!), I have yet to try it and so
can't vouch for it, but there exists a polyfill:
<https://github.com/Yaffle/EventSource>

------
skeletonjelly
Surprised that they didn't incorporate any ASP.NET MVC in there. I was under
the impression they were a Microsoft shop?

------
sandwiches
Pretty cool; any insight as to how they deploy code to production servers?

------
nigma
Very cool. What do you use for search?

~~~
dodger
Right now? Just what we can get out of MongoDB. We need better search, and it
should be coming soon. Vote it up! [https://trello.com/card/board/better-
search/4d5ea62fd76aa113...](https://trello.com/card/board/better-
search/4d5ea62fd76aa1136000000c/4e9ec6c79335096aa4658e32)

------
jeswin
The UI could be less cluttered.

