
Repobuild – Declarative-style build system, like Google Build - albertzeyer
https://github.com/chrisvana/repobuild/wiki/Motivation
======
stephen
Well, 1) I wish him all the best and would love another good build tool
alternative, but 2) I don't see his utopia of demonstrably making open-source
building easier happening because the OS world is a bazaar, not a Google-esque
cathedral.

Google's build system is awesome because it can force everyone in the company
to use it, going so far as to put every single project in one huge repository.

And, then, crucially, the model assumes that any dev workstation can compile
(or pull down via cache) every single upstream project.

In the non-catherdal/bazaar world, take any non-trivial Java project, and all
of it's upstream dependencies, and some are built with Ant, some with Maven,
some with Buildr, some with Gradle, some require thrift installed, some
require who knows what installed.

And you can't just go back and add some meta-data of "oh sure, every Java
project compiles src/main/java -> target"...their build files are going to
have all sorts of crazy one-off steps (Maven plugins, whatever).

There is no enforcement of "you must make your build use this BUILD file, and,
btw, only program in Java, Python, or C."

With all of Maven's warts, the jar-based repository with transitive
dependencies is, IMO, it's major innovation. I just don't see a transitive
upstream build-from-source model (like Google's) ever working in the bazaar
that is Maven central.

I don't at all mean to say Maven/Maven repository are perfect--for one, it's
ridiculous how many broken dependencies are in Maven central. Naively, I had
thought that was automatically enforced/not allowed, but, sigh, no, we all
have to deal with, say, the ASM developers completely breaking their API from
v3 and v4 and forever cursing the rest of us with jar hell.

~~~
alook
Agreed.

Transitively-resolved, platform-independent binaries are the big win coming
from maven. Only once have I worked with a project where I required new
system-level dependencies to compile & run a maven project; and I have never
had a cross-platform issue when carrying projects across Mac & Linux.

To be fair, the import-compiled-transitive-dependencies-as-binaries pattern
that Maven likes to employ is probably less easily adapted to natively
compiled languages like C/C++ than it is to managed runtime languages
(Java/Erlang) or scripting languages (Ruby/Python).

As much as it's a pain to work with "maven plugin spaghetti", it's at least
deterministic and keeps things easy to figure out.

There are those dependencies that you learn to remain wary of
(asm/cglib/antlr/jline/etc), but at the end of the day I'm happy to work in an
ecosystem where I don't have to compile every single line of third-party code
myself.

~~~
bjt
> To be fair, the import-compiled-transitive-dependencies-as-binaries pattern
> that Maven likes to employ is probably less easily adapted to natively
> compiled languages like C/C++ than it is to managed runtime languages
> (Java/Erlang) or scripting languages (Ruby/Python).

The "scripting languages" don't even have this as easy as it might seem. Half
the dependencies in my Python projects these days have important bits written
in C (e.g. psycopg2, gevent).

------
ChuckMcM
Interesting, if Chris can pull this off it will change the world in a huge
way. Having used the build system inside of Google it was somewhat
mesmerizing, you really could put together things pretty easily. I suspect
there is something similar inside Amazon. But I completely agree that Github
has done wonders for making many smaller projects accessible it has not
significantly increased our ability to tie them together. That seems to be
part of the motivation for the CCAN effort as well, although on a more
language specific vector. But as a proof of concept the CPAN system (the one
for perl) demonstrates just how powerful this function can be in enabling
large scale rapid development.

The flip side though, the monkey in the wrench as they say, is that the
projects that participate have to conform to the rules of participation. In
CPAN's case its a standardized dependency, build, and install models with
required unit tests. So perhaps the first step here is to provide an incentive
for clean build processes on a project.

~~~
bbq
Providing incentives seems key. I wonder why CPAN came to work so well.

Possible incentives:

    
    
      - automated testing (TravisCI does this in isolation already)
    
      - automated distribution (build .deb, .msi, and .pkg files)
    
      - usage statistics?
    

I'll have to think about this.

~~~
chris_va
If you come to any conclusions, let me know :).

~~~
bbq
Happily =)

------
srean
You might find the blog post [http://skalrant.blogspot.com/2013/05/build-
systems.html](http://skalrant.blogspot.com/2013/05/build-systems.html)
interesting. It argues that a declarative style is not suitable for many
software development projects. Coming from the author who is also the author
of an ML like language it makes for a good read. Declarative is surely
elegant, but not always easy to work with under specific and not uncommon
situations.

Note, however, it touches on an orthogonal aspect of build systems. I think
Repobuild's main thrust is in standardizing and assisting modularization
whereas the post linked in this comment has a different focus, it argues for a
non-declarative, fixed-point based style.

~~~
thejash
Those are the same link. Can you please update one of them? I'd like to read
both that you mentioned :)

------
saljam
Judging by his description of BUILD, it sounds like was a big influence on
Go's build system.

Well, no surprises there!

~~~
tibbe
More likely Go's build system is influenced by Google's build system, which
came first.

~~~
saljam
That's what I said. :-) BUILD had an influence on Go. Go is influenced by
BUILD. One sentence is in the active voice, the other in the passive.

------
bagosm
There is something I don't understand. You can add other project code to your
own project, but how do you use that other code? In the .NET world where I
live you can reference an Interface and let runtime fill that interface in
(with a proper dependency injection method).

For example you know you want an Interface of MailSender that has the method
.SendEmail() and it gets filled with SMTPMailer which has
SMTPMailer.SendEmail()

Is this thing similar to that concept but in build-time? How do you know the
interface/usage beforehand?

~~~
mrich
Most open source libraries are written in C/C++. The interface is usually
described in header files, you need them to compile your program. The
implementation itself is contained in a library, which you link into your
program. There is also the possibility to resolve the implementation at
runtime via shared libraries.

------
fiatjaf
There's a building collection of small pieces of Javascript code automatically
fetchable from the browser in
[http://www.corslit.com/](http://www.corslit.com/). Does it sound like it? I
think it should.

Explanation here: [http://williamcotton.com/another-way-to-publish-
code](http://williamcotton.com/another-way-to-publish-code)

------
nercury
This looks great! May be a reason for me to ditch CMake, but I kind of need
Windows support :/. Is it much work to add?

I wish someone made a similar way to share libraries like we can do in PHP
([https://packagist.org](https://packagist.org)) or Ruby
([http://bundler.io/](http://bundler.io/)), but for C++.

------
stass
I'd suggest to look at the freebsd ports build system. It is essentially
declarative being built on top of Make language, and allows easy integration
of various open source components and dependencies.

If targeting multiple platforms is of an essence, NetBSD pkgsrc provides a
great model also built on top of Make.

------
skybrian
I'd be curious about how this compares to the Buck project, which is another
Google-like build system:

[http://facebook.github.io/buck/](http://facebook.github.io/buck/)

~~~
chris_va
It's very similar. Here are a few other similar ones:

[https://github.com/chrisvana/repobuild/wiki/Similar-Build-
Sy...](https://github.com/chrisvana/repobuild/wiki/Similar-Build-Systems)

A few differences:

* Buck requires installation (meaning clients need to have it installed), whereas Repobuild generates a Makefile.

* Buck has a spiffy build daemon to make things faster.

* Repobuild auto-install files from git.

* Buck covers java better, but not other languages.

* Buck went with a turning complete language, Repobuild intentionally went with only a config langauge (hotly debated religious difference)

------
chris_va
Author here (a bit late to the party).

I'm happy to answer questions if anyone has any.

~~~
nercury
How hard it would be to add Windows support?

EDIT: Just a question, not a request.

~~~
chris_va
(I have not investigated it fully, so this may be inaccurate)

Cmake:

Repobuild basically generates a Makefile. You could change it to generate a
CMakefile instead. That would not be a lot of work, probably a few (full) days
to change and test it.

Visual Studio:

Generating a visual studio project file may be trickier. A lot of the work
overlaps that of the Cmake stuff above (changing the output format).
Realistically, a 2 week project (I would do xcode projects at the same time).

~~~
nercury
Ok, good info, I might try it.

What I know for sure, CMake is not very adequate for separating work into
different small, independant parts (somewhat "declarative" attept in CMake
here: [https://github.com/key-tools/key-machine-
logs/blob/master/ke...](https://github.com/key-tools/key-machine-
logs/blob/master/key-machine-logs/CMakeLists.txt)), and then including build
targets from other CMake parts into one.

Also, generating CMakeLists, then generating a project seems like an overkill
:)

