
How We Make Trello - spolsky
http://blog.fogcreek.com/how-we-make-trello/
======
edwinnathaniel
I'm more interested to know the actual technical details of the build and
testing process than the "Task-List" software development based approach
because that can be done by any software project management tools (even JIRA).

What's missing in this article is the whole Continuous Delivery technical
aspect of it.

What do you guys use to build the NodeJS app?

What do you guys use to test the NodeJS app?

What do you guys use to check the code coverage of the NodeJS app?

What do you guys use to test the front-end?

What is the automated testing strategy?

How do you store artifacts of builds, schema migration (if using RDBMS) or
handle different model versions, how do you rollback (what's the rollback
strategy)?

~~~
bobbygrace
I hope Doug writes a more technical post about the multi-client build/release
process. We have unit tests for the whole API and very few automated tests for
the front-end. Client builds are stored in S3. We use mongoDB and will do
backfills if necessary (pretty rare). Rolling back the client is just pointing
the stable branch to another build.

~~~
bib971
Bobby, you said every app is a client of the API. I notice Trello.com consumes
API from [https://trello.com/1/xx](https://trello.com/1/xx) while an OAuth
client (from one of your Jsfiddle examples) consumes from
[https://api.trello.com/1/xx?key=xx&token=xx](https://api.trello.com/1/xx?key=xx&token=xx).
I suppose the former just passes through to the later? If so, does the former
needs to pass over the key and token (I suppose you can generate on the fly
based on auth cookie)? I'm trying to get my head around the fact that your app
provides OAuth for other clients and at the same time (from what you're
saying) your app is also one of such clients. Not sure how it really works.

~~~
thedufer
api.trello.com is just a CNAME to trello.com, in case that difference was
confusing you. The authentication part is taken care of differently depending
on the kind of tokens we get (the web client uses a cookie) but we turn that
data into a standard authentication object and the rest of the route uses the
same code regardless of request type.

~~~
bib971
Got you. This is very helpful, thank you!

------
basicallydan
Fog Creek are the _kings and queens_ of dogfooding. Spolsky, you sure have
nurtured a group of very loyal team players. I applaud you all. It must be
really nice to work at a place where the love of the process and the product
are both so strong.

In my opinion if there's one thing a reader should take away from this it
should be that Single Page Apps and separation of server and client are The.
Best. Thing. Ever. From the start, design your system this way.

Good post, and an entertaining read.

~~~
iLoch
While I don't think client side apps are the only way of doing things, I will
say that I think client side apps will become much more popular once browsers
get a little better.

I love the separation of concerns that is possible with JS apps - you can have
one team working on the API and one on the interface and the only place they
really need to communicate is in the API documentation. Once it's all done,
you've already got a fully functional and secure API (because it wasn't an
afterthought) that can be used for other clients.

~~~
copergi
>you can have one team working on the API and one on the interface and the
only place they really need to communicate is in the API documentation

That's already how you should be working anyways. Unfortunately, most language
don't seem to have support for a decent template system that makes this
natural. People need to start making heist clones in their language of choice
so doing things right becomes more common.

~~~
iLoch
I agree, but it's often too easy to just add another variable in your
controller then modify the view, and now there's undocumented functionality.
IMO the full separation is more helpful to stay DRY.

------
badman_ting
It's almost comical how nobody at my company would ever take a project this
seriously. Good for them, Trello is awesome.

~~~
smacktoward
Everything I've ever heard about Fog Creek indicates that they take their
people seriously. Which probably makes them more likely to take their work at
Fog Creek seriously.

Funny how that works, huh? A lesson many, many other companies could profit
from.

~~~
mbesto
I agree with your point, but it requires that you hire correct right from the
beginning. Spolsky (like pg, Atwood, Fried/DHH , etc.) has his pick of the
litter because of well thought out essays and large base of followers. You
can't simply take this attitude without having a large pipeline of people who
(1) agree with you and (2) who are good.

Lesson learned - creating a community or following of people (i.e. talent
marketing) is a very powerful thing.

~~~
rwallace
The essays and followers surely helped, but there are lots of basic pragmatic
things you can do to make yourself a desirable employer even if you aren't a
great essayist. Just some obvious ones off the top of my head:

\- Have sensible working hours (30 to 40 hours a week is optimal).

\- Either let people work from home, or give them private offices.

\- Don't have idiotic hiring criteria like buzzword matching or college
degrees. (Meta: don't have the personnel department doing the hiring.)

\- Get at least the basics of tools and process right. You don't have to let
people code in Lisp or Haskell, but when a candidate asks you about version
control, the answer better hadn't be "oh we don't have time for that here".

\- If you are requiring people to work in the office, don't quibble about
things like high-spec machines and good chairs that cost a small fraction of
the cost of hiring people.

Hit everything on that checklist and even if you're still not quite as sought
after as Fog Creek, you'll be well out in front of most of your competitors at
little cost in either time or money (and infinitely far in front of Fog Creek
for any candidates who won't or can't live in New York).

~~~
thedufer
> and infinitely far in front of Fog Creek for any candidates who won't or
> can't live in New York

Fog Creek actually hires remote people now. We made the change almost a year
ago and its been a great source of new candidates.

~~~
rwallace
Ah, then I happily stand corrected on that point!

------
steveklabnik
> > The Trello API is well-written, has no bugs, and is totally rock solid. Or
> at > least it doesn't change very often. That means we can put out new >
> clients all the time without having to update the API. In fact, we can have
> > multiple versions of the website out at any given time.]

A very counter-intuitive result: most people would not consider a stable API
to let you iterate quickly!

~~~
hiisi
I think the article says that the API is stable unless you have access to
"Alpha" and "Beta" channels.

~~~
thedufer
Alpha and Beta channels only affect what client you get. All client versions
talk to the same API, so it has to be very stable/backwards compatible.

------
allcentury
I've used Trello & FogBugz over the years and we've even modeled some of our
software after some of the practices they've written about. Amazing stuff!

------
sidcool
So they create a new branch for each fix? And all developers are supposed to
merge in their branch every few hours or so? Isn't it a bit of a drag?

~~~
dodger
Yes, more or less a new branch for each fix. But with git and Kiln, it's
really not a drag at all; the merges are generally really easy and mostly
performed by our release manager when the card makes it to 'Ready to Merge'.

~~~
ssi1111
How do you test the fixes? Which branch gets deployed to the test environment?

~~~
bobbygrace
Everyone runs dev environments locally and tests. A dev environment is really
easy to set up with mongo and node. Occasionally, we’ll put stuff on a staging
environment if we want to test a database or infrastructure change. Big,
experimental client changes go to the alpha channel on production.

~~~
ssi1111
So when you have 2 features/bug fixes in the same area, built in 2 different
branches and deployed to isolated dev/test environments... the first time you
might find out that they dont work well with each other is in Staging? Why
wait for that when you could deploy to a common environment and catch issues
sooner?

Btw... I am trying to get some teams in my company to get out of branching. So
trying to understand your view (which is exactly what these teams are doing),
hence these questions...

~~~
bobbygrace
That rarely happens in our experience. With our internal board, we know who is
working on what and in what area of the code. Plus, all code is reviewed by
someone working in the same area.

------
trustfundbaby
What I'm interested in is the mechanics behind how they know were to send a
user based on their channel (beta/stable/alpha). We wanted to do something
like this, but we couldn't figure out how to route users to the right app
server either using AWS ELBs or nginx proxying ... admittedly we didn't really
spend a lot of time thinking about it though.

~~~
thedufer
We do that decision-making inside the web servers. It only affects the client
you get, so when someone requests a page we do a lookup on the logged-in
member to decide which channel they get. API reqs don't care what channel
you're on. No need for any fancy nginx proxying/etc.

------
jamessocol
I'd love to hear—maybe I missed another blog post—why they went with the
single release manager, where only one person can merge and deploy. What
happens if Doug is sick or on vacation? Or even just in a meeting? What is a
typical amount of time for a change to sit in "ready for merge" or merged but
not deployed?

~~~
dpatti
The primary benefit is that it creates a single point of communication for the
developers, the designers, and the QA. It also helps with prioritizing changes
that could be conflicting and, in the same vein, helps prevent bad releases or
merges. And, of course, anyone can do this job. If I'm not available, someone
picks up the slack. I just volunteered because I was interested and available.

All that said, I think that letting every developer deploy would not be a bad
idea at all. The problem is that our team is too big to do that without
creating more robust deployment tools and too small to dedicate enough time to
doing so. My hope is that one day we can get there, though.

~~~
girvo
You should definitely work on the Kinect-to-deploy integration, then
_everyone_ would want to learn how to do it ;)

------
chrismorgan
I like the lighthearted approach to cross-selling in the article. ☺

------
jcastro
Are they on AWS or hosting your own hardware? What OS?

~~~
tghw
AWS, which came about when Hurricane Sandy took down the data center's backup
generator fuel supply and much of the team spent days bucket brigading diesel
fuel up 17 flights of stairs.

------
giulianob
How difficult is it to make the backend work with both new features and old
features at the same time?

~~~
thedufer
To add to what bobby said, changes to the API are far more likely to be
additions rather than actual behavior changes (except for bug fixes, which of
course all clients handle fine). This makes it easy; old clients just ignore
new routes/arguments.

~~~
steveklabnik
Thank you for this. I hope more people come to understand this more nuanced
understanding of change. Please talk about it more!

~~~
hamidpalo
Another Trello dev here.

Basically, we have many clients that are consuming the API, including our
mobile apps. We never change the published interface so that if you're getting
some board fields that will always work. This means that we don't change the
types of fields (for example string vs object) or the names of fields since
that changes the published service contract.

What we do instead is add more fields. A very good example of this is emoji.
We added emoji support a while back and instead of changing the "text" fields
to be objects or embedding HTML (gah) into them we added another field called
"textData" that has the extra info.

This is copy/pasted from an actual call for getting card actions:

    
    
            "textData": {
              "emoji": {
                "daft": "https://trello-emoji.s3.amazonaws.com/4f820a995a03e8e82d134ac4/ae846ca70bf7aa95dd3330b8fd1c70cb/art-daft-punk-steampunk-951631.gif"
              }
            },
            "text": "Custom emoji! :daft:"
    

This may not be sustainable in the long term and if it's not we'll version the
API if we have to at that point. So far though, the API has proven itself to
be very well designed and adaptable (thanks to @d_lec)

------
mikegioia
Does anyone know how the server determines which channel the client should be
using? Are they doing this check at the apache/nginx level, or on the server
right after the user is authenticated, and before the client code is sent?

~~~
gomox
Good question, I wondered about this as well. Channel switching would
certainly create issues on data model changes in the underlying store.

~~~
icebraining
From what I understood, the API doesn't have channels, so the store only
stores a single data model. The channels are only for the clients.

~~~
bobbygrace
Right. The there is only one version of the API. Multiple clients.

------
laurenstill
This is great and all, but I really wish I knew why they won't make the audit
logs accessible to users. If someone has a hack for this, I'm all ears.

Need it for compliance documentation.

------
banachtarski
> The Trello API is well-written, has no bugs, and is totally rock solid.

The no bugs claim indicates to me this wasn't written by a technical person.
Or at least, not very technical.

~~~
bobbygrace
It was a joke. =)

~~~
banachtarski
I reread that paragraph. In context, it doesn't seem so.

~~~
npizzolato
Read more than just the paragraph. The entire article is written in a "fun",
joking style.

> The revelation could result in a rippling shockwave that knocks you off your
> seat and may have troubling, unpredictable consequences for the time-space
> continuum. Possibly.

> You’ve probably downloaded our iOS, Android, Kindle, and/or Windows 8 apps,
> and are saying to yourself, “These are very well polished apps which I have
> rated or will rate favorably in their respective app stores

> The Trello API is well-written, has no bugs, and is totally rock solid. Or
> at least it doesn’t change very often. [Goes on to only reference the fact
> that it doesn't change very often]

~~~
avenger123
I like the passion for the product. Shines through very well.

------
jw2013
These tech intros are pretty cool, yet can I ask how does the team acquire
users? That seems to be harder than the tech challenges for Trello IMHO.

------
somid3
I don't get it. Why is having an API in the backend so that many clients can
use it a big deal?

~~~
morganherlocker
In trello's case they have a browser app, ios, and android. Having one well
designed api makes these easier to build and maintain. It appears to be
working pretty well for them. I use trello in browser, on an iphone, and on an
ipad, and they all work together very seamlessly.

------
kalnuezis
Anybody using FogBugz and Kiln from Fog Creek? Both seem to be a bit
outdated...

~~~
anton_gogolev
In what sense are they outdated? UI-wise (which I will agree with) or feature
wise?

------
nevster
I'd like to be able to write like this. It made me LOL.

------
aravindb
Trello using Trello,(Inception) awesome!

------
takacsv
Trello is using Trello to develop Trello. So meta :)

~~~
AznHisoka
they're using Trello to write a Trello post on how they're using Trello to
work on Trello.

------
haymills
So interesting!

