

Square open sources Kochiku - xophe
http://corner.squareup.com/2013/09/kochiku.html

======
pbiggar
CircleCI ([https://circleci.com](https://circleci.com)) offers this right now.
We automatically partition your test suite and run it across multiple
machines.

Except since it's in the cloud, you don't need to own 10 Mac minis to make
this happen. And if you decide you want to split it 20 ways, not 10, you can
do that in about 2 clicks.

[disclaimer: I work at [https://circleci.com](https://circleci.com), if that
wasn't already obvious]

~~~
statusgraph
It sounds like Square doesn't use 10 mac minis but rather EC2 spot
instances....

------
vscarpenter
I'm curious as to why Square would spend engineering talent on a distributed
CI system when you can buy something like Jetbrains TeamCity that does what
they wanted to do. This is not meant as a slam and I am also a developer who
loves to build things but I would think Square and other companies for that
matter would want to apply their limited engineering talents in solving bigger
problems that grow the business.

Not looking for a flame war - just curious to get different perspectives.

~~~
hkarthik
It makes a lot of sense. I'm currently working on a large Rails app and we've
felt a lot of pain with an off the shelf CI System (Jenkins) and the idea of
writing our own CI has been mentioned a few times.

For Team City, here's the cost breakdown:

TeamCity Enterprise = $1,999

229 Build agents @ $299/each = $68,741

Total yearly license cost ~ $70K

This doesn't include the time that a team of developers will spend configuring
the build and babysitting it during through the early growing pains of
adopting a CI system. Also you'll probably still need a build/release
engineering team to manage the server when you have a team of 100+ devs like
Square does.

When you add it all up, building a custom CI setup makes a lot of sense. And
I'm sure they grew it over time, with an initial version that was usable being
completed in just a few months. What they've open sourced is the end result of
spending that time.

When you really adopt testing as part of your culture, CI becomes totally
critical so engineering should spend the time to make it a solid and viable
solution.

~~~
bitops
A total yearly cost of $70K isn't all that bad when you break down the
numbers. Software engineer salaries are typically much higher, let alone
considering HR & insurance expenses.

So that's one employee you don't have to hire. I was on a team of 4 that,
among other responsibilities, managed several TeamCity instances, all of them
pretty large.

Generally speaking, once it was up and running, it was pretty low overhead and
easy to automate. So I think it really is worth thinking hard about the cost
before people dive in and roll their own solutions.

And there's always the argument of: instead of building your own, why not work
to improve an existing OSS solution?

~~~
hkarthik
> Generally speaking, once it was up and running, it was pretty low overhead
> and easy to automate.

This unfortunately has not been my experience with off-the-shelf CI solutions
(open source or otherwise). Of course in all instances, we were writing a lot
of new code and continuously enhancing old code so the projects under CI were
very active. We dealt with CI systems being overloaded, builds breaking for
cryptic environment setup reasons, and just an overall slowness due to memory
leaks that were hard to diagnose and only occurred in the CI environments.

In the smaller teams, one engineer would essentially become "the build guy"
that would babysit the CI system. In larger teams, this became a function of a
Configuration/Release management team. In all cases, CI was a regular pain
point unless there a lot of customization done over a long period of time.

At the surface level, a custom CI solution wouldn't take long to write. A
simple poller to check version control, a script to build, and some basic
notifications could probably be hacked together quickly and iterated on over
time.

------
joeblau
One of the things i'm really loving about Square is the utilities that they
are open sourcing that are not core to their business. I think this is great
for the community.

~~~
aguynamedrich
Their Android libraries and their involvement in the Android dev community are
incredible.

------
derwiki
When I was at Causes, we wrote a similar tool called Buffet
([https://github.com/causes/buffet](https://github.com/causes/buffet),
pronounced like the investor) for running test cases in parallel on an
arbitrary number of machines; dropped out total build suite time from ~20
minutes on a single machine to 2-3 minutes. We certainly didn't add the level
of polish that Kochiku seems to have, though.

~~~
aiiane
Yeah, we wound up adding similar functionality to Testify
([https://github.com/Yelp/Testify](https://github.com/Yelp/Testify)) before I
left Yelp.

------
polskibus
For distributed build, testing this is the best tool I've used:
[http://www.incredibuild.com/](http://www.incredibuild.com/). Unfortunately,
it is not open source. Does anyone know of a free alternative to incredibuild
with a similar feature set?

------
nobodysfool
So now we've gone from adding an 'er' to everything and dropping the 'e' to
using Japanese words and proper names for our products. And I was just going
to iPost that to my iBlog.

