
CNCF to Host Cloud Native Buildpacks in the Sandbox - rettori
https://www.cncf.io/blog/2018/10/03/cncf-to-host-cloud-native-buildpacks-in-the-sandbox/
======
sclevine
For those who would like to better understand the technology:

Presentation to CNCF TOC:
[https://docs.google.com/presentation/d/1RkygwZw7ILVgGhBpKnFN...](https://docs.google.com/presentation/d/1RkygwZw7ILVgGhBpKnFNgJ4BCc_9qMG8cIf0MRbuzB4/edit#slide=id.g4054009e6b_5_0)

Formal specification:
[https://github.com/buildpack/spec](https://github.com/buildpack/spec)

Sample buildpacks:
[https://github.com/buildpack/samples](https://github.com/buildpack/samples)

~~~
nailer
In other words: they're (typically) shell scripts that set up an OS to run a
language.

Here's the node one: [https://github.com/buildpack/samples/blob/master/nodejs-
buil...](https://github.com/buildpack/samples/blob/master/nodejs-
buildpack/bin/build)

------
a012
It's first time I heard of buildpacks, can I just throw in other than nodejs,
ie. Laravel source code and build an image?

~~~
dreyfiz
Yeah, do you have a composer.json in the root of your repo? Buildpack will
detect a PHP app...

~~~
dullgiulio
I have the feeling that the detect phase is absolutely orthogonal to
delivering what should basically be an executable package.

What is the rationale behind having the build step tied to the packaging
format?

~~~
jacques_chester
It's not tied to the packaging format. Detect is the step that decides which
buildpack or buildpacks will be responsible for constructing the image from
the sourcecode.

Typically this means that buildpacks look for files that correspond to the
relevant ecosystem. Maven buildpacks look for pom.xml. PHP buildpacks look for
composer.json. Etc.

Nothing in this creates a hard binding. Detect steps may use whatever logic
they need to decide on whether to signal they can work on a codebase.

Edit: in the v3 design the detect script can also provide dependency
information that later steps can pick up. So for example, a JDK buildpack can
say "yes, I can interpret this codebase, and I can contribute a JDK". A later
buildpack can then look for this contribution as a condition, eg. the Maven
buildpack can say "I will proceed if I see a pom.xml and if there is a JDK
available".

------
josegonzalez
> As one of the maintainers of Herokuish - a tool that manages the application
> of buildpacks to repositories and is used in quite a few Buildpack
> implementations - I am super happy that CNCF reached out to us and included
> us in the process. Oh wait, that didn't happen...

All jokes aside, this looks great. Super-early of course - seems like there
are quite a few issues in the `pack` repository to be implemented - but I'm
excited to see where this lands. Buildpack detection and application is not a
straightforward problem.

~~~
jacques_chester
Please join us at [https://slack.buildpacks.io/](https://slack.buildpacks.io/)
!

(ps. I know what these surprises can feel like -- for what it's worth, I am
sorry)

------
gigatexal
What am I missing? Looking at the example for java it looks a lot more
complicated and a step back from docker and its layers concept.

~~~
sclevine
Dockerfiles require you to rebuild lower layers when any upper layers change,
even though the OCI image format doesn't care about this. Cloud Native
Buildpacks can intelligently choose which layers to rebuild and replace.
Additionally, certain layers can be updated en mass for many images on a
registry (using cross-repo blob mounting, with no real data transfer!), which
is useful for patching CVEs quickly at scale.

The samples take advantage of this (as well as a separate transparent cache)
in order to demonstrate the different aspects of the formal spec. A simple
buildpack is not necessarily much more complicated than a simple Dockerfile.

~~~
gigatexal
Yes, if the underlying layer’s hash changes then it has to be rebuilt. But if
you just change index.html it caches the other layers and builds are very
quick.

My issue with Buildpacks is that it looks like a glorified bash script (which
is a skill I am not bashing — pun not intended) whereas a dockerfile is much
more human readable and the idea of layers, for a guy coming from a systems
background, is much more intuitive for me. The analogy of a very lightweight
VM makes perfect sense to me which means I’m much more productive with it.

------
mosselman
I just tried this on my rails project and it is detecting a nodejs project. Is
there a way to have it use ruby instead while still taking care of my yarn
dependencies?

~~~
justinsaccount
This can be a problem with buildpacks, since you'd need one buildpack that has
both the ruby and nodejs runtimes.

With Docker it is easy to make this work using multi-stage builds, I'm not
sure if buildpacks account for this at all.

~~~
jacques_chester
> _This can be a problem with buildpacks, since you 'd need one buildpack that
> has both the ruby and nodejs runtimes._

This isn't strictly true of v2 designs (both Heroku and Cloud Foundry have
schemes for multi-buildpack support) and in the v3 design explicit
consideration is given to making mix-and-match a triviality. Buildpacks can
cooperate quite easily and in multiple groups.

Where this shines is in updates. If I have an OS base layer, a JDK buildpack
and a Maven buildpack, the layers they generate can be independently updated
without needing a complete rebuild. So far as I am aware, this is not
currently possible with a Dockerfile, multibuild or not. If you invalidate the
OS layer, everything else gets rebuilt whether it needed to be or not.

~~~
justinsaccount
That's cool, I'll need to read up on the v3 design.

I think you are right about the rebuilds, but maybe not in all cases.. If you
had.

    
    
      FROM nodejs:whatever as js
      RUN  npm build
    
      FROM golang:whatever as go
      RUN  go build
    
      FROM base
      COPY --from js app.js
      COPY --from go app
    

If only 'base' was updated rebuilding the image would just need to re-run the
COPY commands. The only way everything would get rebuilt is if you also
updated the go and js images.

I think for more complicated applications [https://grahamc.com/blog/nix-and-
layered-docker-images](https://grahamc.com/blog/nix-and-layered-docker-images)
is ultimately the best way to do things... ensuring that every step is
isolated with explicit dependency information.

~~~
jacques_chester
I agree, the Nix approach makes lots of problems go away, but you have to be
bought in first. I wrote something distinct a few months back which overlaps
on the "we can be smarter about layers" thing[0].

Google folks worked on "FTL", which is a technique for determining layering by
reasoning about packaging system information. Jib[1] is one such system. There
is a view that it will be possible to use FTL implementations or derivatives
as buildpack components in the future.

[0]
[https://docs.google.com/document/d/1M2PJ_h6GzviUNHMPt7x-5POU...](https://docs.google.com/document/d/1M2PJ_h6GzviUNHMPt7x-5POUaadcvK3ZNT9QEGDZhPk/edit#)

[1]
[https://github.com/GoogleContainerTools/jib](https://github.com/GoogleContainerTools/jib)

------
EdSchouten
I currently use Bazel in combination with rules_docker to construct container
images of applications, often written in Go. As I don't depend on cgo, my
container images may be bare (i.e., only containing a Go executable and some
static artifacts).

Though I understand that there are likely only dozens of us, my question is:
what would the use of Buildpacks buy us for this specific use case?

~~~
justinsaccount
one case is that you could have a bazel buildpack that contains bazel and java
and all that stuff requires in order to run the build.

Unfortunately the resulting image would be larger than just a single binary,
but building it would be a lot easier and repeatable for other people working
on the project. The base layers would be cached though so in practice it might
not be that much larger on disk.

I looked into using bazel for doing similar things, and the biggest stumbling
block is that bazel itself is a PITA to install on all platforms. May end up
trying [https://please.build/](https://please.build/) at some point.

~~~
jbott
The bazel installation process has gotten a lot better lately - they now
bundle OpenJDK inside the binary and it is only a single download.

~~~
justinsaccount
Unfortunately that's a non starter for things like debian.. but that's good to
know, I'll give it another look.

One thing I really like about bazel is the pkg rules.. I currently use
fpm/goreleaser(nfpm) to build rpms for things, and it's nice having a single
build tool that can build the app and spit out an rpm.

------
jacques_chester
Hi, in my role as professional gadfly, I have been involved with this
tangentially for a few months and directly for a few weeks (on behalf of
Pivotal).

I will make mistakes, but ... AMA.

~~~
spenczar5
I know almost nothing about this topic. I have only really built container
images with Dockerfiles. How would you introduce buildpacks to me?

~~~
bryanlarsen
Step 1: replace your Dockerfile with one that consists of the single line
"FROM cloud-gov-registry.app.cloud.gov/python-buildpack". The buildpack
contains magic that knows how to turn a standard python program into a usable
docker image. (AKA, it inspects requirements.txt, et cetera).

Step 2: Now that your Dockerfiles no longer contain any real information,
retool your orchestration system to use source tarballs rather than docker
images.

~~~
jacques_chester
I'm not familiar with this flow for using buildpacks, myself.

Here's how I would do it:

    
    
        rm Dockerfile
        cf push
    

aaand I'm done.

Someone using Heroku might do it _wildly_ differently. It's super complicated:

    
    
        git rm Dockerfile
        git commit -m "Switch to Cloud Native Buildpacks"
        git push

~~~
bryanlarsen
Sure, you're welcome to skip straight to step 2. :)

------
CameronNemo
I wonder how this relates to LinuxKit.

~~~
codefinger
LinuxKit (as I understand it) is for building the OS itself. Buildpacks are a
higher level abstract for building a complete application image (with the
emphasis on application). They take app source code as input, and output a
docker image that's ready for prod.

------
m-arnold
What is the difference between buildpacks and packer?

