
If a Build Takes 4 Hours, Run It Every 4 Hours - royosherove
https://pipelinedriven.org/article/if-builds-take-4-hours-run-them-every-4-hours
======
_ph_
I don't know what the author is building that a build is taking 4 hours, but I
am aware that there are huge software projects with build times like that and
much longer. I agree that getting quick feedback to changes is vital for quick
overall development. This shows the values of development stacks which reduce
the build times a lot. As a resultmore incremental development can be done.
For example, the build times were one of the architectural goals when
designing the Go language.

Dynamic languages are also a great tool to fight build times. Even if it is
unsuitable for large projects to be entirely built on dynamic languages, a
great technique is using a mix of static and dynamic. Build the core of the
application in a static language, but more as a set of libraries to be called
from the dynamic layer. The dynamic layer is where most of the application
logic resides, easy to be changed. And parts which have stabilized reasonably
can still be moved over to the static part.

~~~
mister_hn
Have you ever tried to build Chromium or Firefox yourself? If you have a slow
machine, the build can take even 10 hours or more

~~~
_ph_
No, I have not. But wouldn't be those code bases which compile very slow
because of being written in C++? C++ seems to baloon in building time for
large code bases. On the other side, the big Threadripper manages to compile
the Linux Kernel in 40s. I don't think those web browsers are that much bigger
than Linux than the compile times differ.

Firefox is further slowed down by using more and more Rust based code - Rust
is known to compile slowly. However I would say this is a good tradeoff, since
Rust brings much more safety to one of the most safety-critical applications
on a typical desktop.

~~~
zlynx
It is extremely easy to let your C++ code get into long build times. People
get fond of putting code into the header, for example. This adds that compile
time to every file that includes that file. And it prevents you from using
forward declaration, which then requires including those definitions in your
header.

With discipline and tracking the build time, and tools that record how long
each build step takes, you can keep people from doing things like declaring a
Boost multimap in a header file. They can go ahead and use it but hide it in a
cpp source file behind a pointer or a virtual interface.

I've noticed that some people fall in love with C++11 move constructors, and
they are pretty cool, but unlike a reference or pointer they require full type
definitions in order to pass "by value." So that's annoying.

C code is definitely faster to build, unless you've got some crazy programmer
who reimplemented C++ templates using five level deep nested macros.

------
noisy_boy
We have Maven builds that complete in a few mins via Jenkins but I have been
exploring if it can be done even faster. The builds are written using the
Jenkins declarative pipeline with parallelization etc but feels like it is
just not fast enough (the slowness is probably because of the whole lot of
intermediate IO/back-and-forth between Jenkins master and slaves). Wish there
was a build setup that was faster while being as feature rich as Jenkins while
supporting Maven builds.

~~~
mister_hn
Try the flag -T XC where X is the number of threads you want (similar to make
-jX)

~~~
noisy_boy
Yeah - I did try it + the other oft-recommended optimizations. I have arrived
at the conclusion that the actual build is not taking the majority of the
overall build time, it is the various before and after that happens (e.g.
checkout git repo, deciding build parameters, sonar qube scans, creating git
tags and so on). Since the Jenkins pipeline basically runs shell scripts to do
these things in each stage, the overall build just feels slow. Collapsing the
stages would make things more messy, keeping them separate is cleaner but
means more per-stage overhead.

~~~
mister_hn
I would suggest to keep the shell scripts calls to the minimum (e.g. compact
commands together if possible).

You can still have nice stages separations, but try to avoid the pitfall of
doing too much shell invocations.

A good trick is to create a fake "multi-module reactor" pom.xml and grouping
all the modules logically there, then you build like

mvn clean install -f reactor-pom.xml -T 8C

~~~
noisy_boy
Thanks for the suggestions. We have very few multi-module projects. Most of
the time, the time is spent in the various stages - almost all of them shell
scripts. I guess there is overhead of initializing the stages/spawning etc. I
also think that Jenkins could be doing some serialization to be able to resume
a failed build - which we don't really need as we'll probably start a fresh
build anyway. That's my next area to explore.

------
tedunangst
Tying your build frequency to build time _and then using that as motivation to
decrease build time_ is probably the important bit here, but only if you
include the second part.

~~~
royosherove
That's a good way of putting it. In TOC terms, the build time becomes the new
constraint to solve, but only after you solve the build frequency constraint.

------
teleforce
If the long build time for static typing programming language is the concerns
perhaps you can try have look at D programming language since it is optimized
for fast compilation [1].

Around last year someone even managed to speed up its compilation time even
further and it is also featured in Hacker News but cannot find the link now.

I think in the future most of the static typing programming languages will be
mainly based on Single Static Assignment (SSA) technique then the compilation
could really fly.

[1][https://www.drdobbs.com/cpp/increasing-compiler-speed-by-
ove...](https://www.drdobbs.com/cpp/increasing-compiler-speed-by-
over-75/240158941)

[2][https://www.cs.princeton.edu/~appel/papers/ssafun.pdf](https://www.cs.princeton.edu/~appel/papers/ssafun.pdf)

------
gregdoesit
Alternatively, you can parallelise parts of your build (eg the tests, static
analysis etc) and speed it up. Then, if you have more changes coming in than
you could parallelise is when you can start to do some advanced things like
trying to predict which changes might break the build.

There was a longer discussion on one of these approaches done by Uber[1] on
HN. Granted it’s a lot of investment, it’s interesting to explore ways of
dealing with long builds that need to run frequently.

[https://news.ycombinator.com/item?id=19692820](https://news.ycombinator.com/item?id=19692820)

~~~
larrybud
Or, just parallelize the entire build. eg, if you have four build servers, you
could kick off a build every hour. Easy using VM's, trivial in the cloud.

------
DmitryOlshansky
Why not every hour or even half-hour?

Provided that you can run multiple builds in parallel but each of them has
somehow limited parallelization.

------
tlarkworthy
Why don't we run it on every change?

~~~
bryanrasmussen
If a build takes 4 hours it argues that a build on that system is a pretty
expensive operation, unless changes are very infrequent, which in most systems
is not the case, a build run on every change would seem to be highly
problematic.

~~~
tlarkworthy
Bingo it's expensive, which is why they set it to every 24 hours.

Also coz they don't want to add the queuing logic, so it takes 4 hours to run
is an approximation. Also it messes with other systems so they prefer to run
it overnight. This blog doesn't solve a real problem.

~~~
royosherove
Author here. It's literally the first line in the blog post. And it links to
this blog post about static environments:
[https://pipelinedriven.org/article/the-four-biggest-
issues-w...](https://pipelinedriven.org/article/the-four-biggest-issues-with-
having-static-environments)

~~~
tlarkworthy
That doesn't answer anything. Either you: minimize latency (run every change
independent of build time) or run to minimize engineering (24 hours). I dunno
why you couple build time to a static period, it does not make sense to me.

