
Ask HN: Flexible, lightweight build automation/CI tool? - jakobegger
I‘ve recently set up Jenkins to continuously build and test a Mac app (Postico), and it‘s not been a pleasant experience. Everything is poorly documented, you constantly need to switch between different paradigms (point and click interface, procedural pipelines, declarative pipelines), the API is atrocious, and I haven‘t figured out yet how to configure the dashboard.<p>So I‘ve started to look for more lightweight and flexible alternatives. My build process consists of a couple of shell scripts (bash), some ruby scripts, and even a python script. The individual parts work well, I only need something to tie it all together and display status.<p>It can‘t run on docker because the app is built with Xcode, but I do use docker to start up test servers that our unit tests run against.<p>I‘ve been looking at buildbot, and it looks like it does what I want. Does anybody have any experience with it?<p>I‘ve also considered using a general purpose task scheduling system, but I don‘t know anything about that space. Does anybody have good recommendations?
======
ctpide
I’ve been using Concourse CI and it’s fairly light weight and has great
declarative pipelines. Learning curve is not too steep and it has a command-
line tool to update your pipelines. Maybe it’s a stupid suggestion, but you
could connect the worker in that setup through ssh to a Mac for building, but
of course that takes away some of the comfort, but you will have to run a
dedicated build machine anyway.

Also, doesn’t Travis CI have macOS support ?

~~~
mmanulis
+1 to Concourse. Though if the OP needs OS X build support, there is no
escaping needing a Mac box to run it on.

One can still run Concourse on a Mac Mini if one chooses, not that much extra
work. This is the simplest solution but you will still need XCode Server last
time I checked (been a while). I'm sure there are ways to hack around it, just
a matter of time vs money question.

Both Travis CI and Circle CI offer Mac builds, so if one is willing to spend
some money, that's an option.

Having had to run my own CI infrastructure, I'd rather pay someone else to
deal with it. Especially if your deploys are tied to it, as once it breaks,
and it will do it, your entire pipeline is down.

I ended up spending a good chunk of my own time dealing with failing builds or
having a couple of people on my team be the point-persons for CI/CD infra.

------
Androider
CircleCI 2.0 has full support for macOS, sounds like a perfect fit for you.
It's also inexpensive (if you don't consider your time worthless, since it's
$39/mo, which is ridiculously cheap compared to time we spent babysitting
Jenkins previously), and just one of the best CI systems there exists in my
experience.

Just a happy customer.

~~~
circular_logic
Circle might be nice for a small project, but I would advise away from Circle
Ci for larger projects. Their feature set is immature if you want to do
anything that requires a moderately complex workflow or need triggered
deploys. Your builds are in a queue that can take ages sometimes. Their UI is
pretty poor. For example, branches show up forever on a project even if you
delete them from the repo. You actually have to ask their support to remove
them.

~~~
busterarm
Circle CI 2.0 and their triggered deploys work pretty well, actually. We've
been using 1.0 in production for over 2 years and we're about 3 months into
using 2.0 and it's working out great.

Our builds are pretty complex and we have tear down scripts that we always
want to run even if a build fails and CircleCI does that (on 2.0, finally).

The only thing I wouldn't advise CircleCI for is if you want to distribute
your tests across many machines -- and only because it's very expensive!

------
robertop
I have used buildbot for about 5 years on a cross-platform (mac, linux,
windows) project. It is definitely lightweight and flexible.

Pros: \- The schedulers and pollers is very stable. You don't really have to
worry about uptime. \- Good enough documentation.

Cons: \- Configuration is done in python. If you don't know python well then
be prepared spend time to learn it.

The biggest pain point, at least for me, has been that buildbot has no support
for caching of dependencies. If your project has many dependencies you will
spend quite a bit of time either installing dependencies on the CI builders or
having long build times because every build will download dependencies. This
is not really a buildbot issue itself, but it will definitely bug you if you
have used hosted CI products like TravisCI or Circle CI.

~~~
Rondom
My usage of buildbot is only very anecdotal (small hobby project). I consider
it very easy to get started with. Through its use of Python it allows for a
lot of flexibility, but I don't know how well that scales to very big
complicated organisations. But then again, Jenkins is not easy to work with,
either and you might not really need the complexity and feature-set that
Jenkins provides.

------
SeanA208
Buildkite ([https://buildkite.com/](https://buildkite.com/)) is so underrated
and really nice! Much simpler interface than most other providers like Circle
CI or TeamCity and works really well.

~~~
exidy
Buildkite is my go-to CI in my professional work as well. The hybrid model
(the master is SaaS, but you bring your own agents) works so well both from a
cost-efficiency perspective, and also when working with enterprises who don't
like builds occurring on machines they don't control.

------
bluehatbrit
Circle-CI ([https://circleci.com/](https://circleci.com/)) have a macos system
for building things like xcode projects. I've not tried it myself but it's
cloud based which saves self hosting. Unless you want to self-host in which
case this may not be a great option.

------
chuckhoupt
Have you looked at Xcode Server:
[https://developer.apple.com/library/content/documentation/ID...](https://developer.apple.com/library/content/documentation/IDEs/Conceptual/xcode_guide-
continuous_integration/) ?

~~~
jakobegger
Yes, and it was extremely frustrating to use (I didn‘t get it to work at all),
and it also doesn‘t cover things like testing against multiple servers.

------
zwischenzug
I wrote this in pure bash as a learning exercise some years ago:

[https://github.com/ianmiell/cheapci](https://github.com/ianmiell/cheapci)

it was surprisingly effective, but I capitulated to Jenkins eventually.

------
ctrlc-root
I know you said lightweight, but have you given TeamCity
([https://www.jetbrains.com/teamcity/](https://www.jetbrains.com/teamcity/)) a
look? It's much easier to configure than Jenkins in my opinion and very
powerful. I've used it at my last two jobs and so far there's nothing it
hasn't been able to handle, including some complicated inter-project
dependencies, and it's been relatively easy to set up and use.

~~~
jakobegger
Their screenshots scared me away initially — it looked too similar to Jenkins.

------
baronofcheese
You don't mention whether this needs to be on prem or if it is OK if it runs
somewhere else. Visual Studio Team Services work great for me. They also have
hosted Mac instances that you can run on. It is very flexible. You can also
install agents on your own machines to run on instead.

I've also used Bitrise in the past, it also has Mac OS and Linux machines
where you can run your builds on.

~~~
taspeotis
+1 VSTS. You also get 240 minutes of build agent time free per month. Their
hosted Mac agents are in preview right now.

------
perlgeek
Not the most light weight, but consistent and cross-platform:
[https://gocd.org/](https://gocd.org/)

It has pipelines as a first class citizen. The config is rather concise XML,
and can be hand-written or generated/modified through a web UI. There is also
the option to store pipeline config as XML/YAML/JSON in a git repo.

~~~
cygned
Did they add key based authentication with SSH yet? That is basically the
missing feature that renders it unusable for me.

~~~
perlgeek
You can give the go system user an ssh key, and then you can use ssh git URLs.

Is that what you want?

~~~
cygned
That's basically what I am talking about:

[https://github.com/gocd/gocd/issues/951](https://github.com/gocd/gocd/issues/951)

With Jenkins, I can create credentials with the key and use them upon checking
out. This feature prevents me from using GoCD.

------
mkesper
You don't need to click to configure jenkins jobs, they're yaml files
containing "builder" steps.

~~~
Androider
Jenkins setups invariably accrue a bouillabaisse of plugins and stews in a
"just so" environmental soup, that simply moves the "works on my machine"
syndrome to "works on our Jenkins (please, please don't break or we'll be down
for a week reverse engineering this Ubuntu/RHEL snowflake from 2012)".

~~~
maxxxxx
I have heard the best way to use Jenkins (or another CI server) is to avoid
any plugins and just have it kick off a few shell scripts. Seems also easier
to test the different steps that way.

------
atsaloli
GitLab CI is lightweight compared to Jenkins. I recently taught CI/CD using
both, on small instances, and the instances running Jenkins would occasionally
grind to a halt due to load while the ones running GitLab kept ticking. Same
CI/CD load, same instances. Just a different CI app.

~~~
jakobegger
It looks like Gitlab CI requires the pipelines to be inside your repo. I‘d
like to keep them in a separate repo, otherwise making changes eg. to a test
server configuration requires merging the new config into every branch.

~~~
jeremija
You can define build (secret) variables per project, which means you can
change the configuration without committing every branch.

------
vorpalhex
I use some Jenkins and some Travis-CI([https://travis-ci.org/](https://travis-
ci.org/)) with good luck. Travis seems capable of most things that Jenkins can
do, while still being a pleasure to configure and work with.

~~~
jakobegger
I‘ve tried Travis once, but the build didn‘t work and I just had no idea how
to debug it. Also, the build environment I wanted wasn‘t available, so I
really want to go with something that allows me to use my own machines for
building/testing.

------
alexjarvis
bitrise ([https://www.bitrise.io](https://www.bitrise.io)) is the best that
I've used so far for building iOS apps. Its elite package is also really fast
if you can afford it. I've used travis (slow, 45 min build times) and gitlab-
ci with a dedicated mac pro runner and bitrise wins hands down for speed and
simplicity. Also don't forget fastlane
([https://fastlane.tools](https://fastlane.tools)). Most of my build
configuration is in fastlane (for iOS and Android) and bitrise just triggers
fastlane via the script step. Good luck!

~~~
jakobegger
Doesn‘t look like bitrise supports macOS?

------
xpaulbettsx
I wrote Surf ([https://github.com/surf-build/surf](https://github.com/surf-
build/surf)) for exactly this purpose, it's very much inspired by Buildbot

------
andyana
I use [https://github.com/Strider-CD/strider](https://github.com/Strider-
CD/strider)

------
bm1362
Drone is an open source CI/CD alternative.

------
rc_kas
bash scripts are the most light and most flexible :D

------
philippz
Circle CI!

------
vletrmx
Check out [http://git.liw.fi/ick2/](http://git.liw.fi/ick2/)

~~~
jitl
This website lacks all information about the project. What does it look like?
How to use it? Googling returns only irrelevant results. No findable docs.

This looks like a toy or side project.

