
GitLab Container Registry - martgnz
https://about.gitlab.com/2016/05/23/gitlab-container-registry/
======
connorshea
Kind of unrelated, but I'm super happy to be working at GitLab.

Nearly everything we do is out in the open, and most features are available in
CE, which is fully open source ([https://gitlab.com/gitlab-org/gitlab-
ce](https://gitlab.com/gitlab-org/gitlab-ce)). I've wanted great, open source
tools that don't look like dung for a long long time, and GitLab is definitely
reaching that goal. Can't help but gush about the product we're building. :D

Here are some of my favorite upcoming features: [https://gitlab.com/gitlab-
org/gitlab-ce/issues/17575](https://gitlab.com/gitlab-org/gitlab-
ce/issues/17575), [https://gitlab.com/gitlab-org/gitlab-
ce/issues/14661](https://gitlab.com/gitlab-org/gitlab-ce/issues/14661),
[https://gitlab.com/gitlab-org/gitlab-
ce/issues/15337](https://gitlab.com/gitlab-org/gitlab-ce/issues/15337)

~~~
Cyph0n
Damn, you are one lucky person! Getting paid to work on OSS is a dream come
true.

Keep up the awesome work everyone :)

~~~
sytse
Thanks! And I think Connor created his own luck. People were assuming he
already worked at GitLab since he was so active on social media and on our
issue trackers. If you consistently contribute to GitLab it is very hard for
us not to hire you.

~~~
Cyph0n
That's a pretty effective hiring method in my opinion. You guys should publish
a blog post or something on your hiring practices and results - I'm sure the
HN crowd would enjoy it.

~~~
sytse
Thanks, our hiring practices are detailed on
[https://about.gitlab.com/handbook/hiring/](https://about.gitlab.com/handbook/hiring/)
what kinds of results should we include?

I'm also thinking about writing about an open organization, where the
handbooks and issue trackers are public. Would that be interesting?

~~~
broodbucket
Yeah, definitely. It's an interesting concept, I didn't know about the
handbooks being public, seeing employees interacting on issues in the open and
seeing the company's strategy be posted publicly is very cool.

------
lukebennett
Fantastic, Gitlab are absolutely nailing it at the moment. Don't think there's
any other hosted service offering so many CI/CD features for free under one
roof.

It feels like what Github did for SCM, Gitlab are doing for CI.

~~~
sytse
Thanks Luke! We'll try to keep this going. We're working on deployment
environments for our next release so you can see what is deployed in each
environment. After that we can do deployments that need to be confirm
manually.

~~~
jstsch
Wow, that's great news. Exactly what we are looking for. I was investigating
to use GitLab CI to build, and then only deploy commits with a certain tag,
but a dedicated workflow for deployments is much more convenient and less
error-prone.

~~~
sytse
Glad to hear that. By the way, the 'only deploy tags' strategy is already
possible today if you want to do it that way.

------
pritambarhate
These days it seems that Docker is everywhere. I am new to docker and I seem
to find it as one more additional complicated system that the developer now
needs to learn in order to deploy his or her application.

How useful do you find docker for application which can be deployed on Heroku
or Beanstalk? I can understand using docker for a Language ecosystem which is
not supported on Public PaaS. Or for people for whom public PaaS is not an
option.

I would like to know about the experience of using Docker in day to day
development from people who have used Docker in team environments. How was
your experience of converting the team to use Docker instead of their regular
development/deployment? For example, at our company, For LAMP or MEAN or Java
Stacks, we have a defined a procedure of setting up the dev machines to
include all these tools and developers know how to manage these on their own
machine. Once the code is pushed to Jenkins, there are deploy scripts written
by tech leads which take care of deployment to various servers or to Heroku.

In your Docker driven development process, did everybody started using docker
on their development machine too? Or everybody just kept using their local
setup, and only one/two people handled the task of creating final docker
images? Or do you just use you CI server to handle the docker image creation
and deployment from that point onwards?

~~~
trey-jones
I am also pretty new to docker and it took a few days to get my head into it,
but I am using it mainly as a contained dev environment, similar to the way we
use vagrant. I am finding that docker is much better suited to complex
environments, and a lot easier to update.

Contrary to your fears of everyone having to learn docker to develop, I see it
as a tool that allows every developer on a team to be brought into new
technology without having to learn all about a new toolchain. We are using
docker-compose for this. Everyone has to install docker, and have some cursory
knowledge of docker-compose commands, but that's it.

As an example, I recently shoehorned an angular2 front end into an existing
project. Without docker, this means every developer needs node and all the
dependencies before they begin to work. With docker-compose, the next time
they start up the environment, the npm container installs all of their
dependencies, and the typescript compiler container listens and compiles all
changes to the webroot. The other developers don't even need to know what node
is, or that it's being used!

Docker is a very powerful tool with a lot of untapped potential I'm sure. I've
no experience with docker in production, however.

------
Tiksi
It seems like every time I set up some supporting infrastructure it ends up
rolled into the next gitlab release a week later, not that I'm complaining
(much).

If you haven't checked out gitlab in a while, you definitely should. It's been
moving fast and come a long way lately.

Thank you Gitlab team for making an open source, self hosted platform and all
the recent improvements you've made.

~~~
sytse
Thanks for your kind words. May I ask what supporting infrastructure you'll
set in 3 weeks from now? We'll have to add it to the schedule for 8.9

~~~
Tiksi
Well since you asked, I'm working on a system that writes perfect bug-free
code for me, but if you can implement that I'll save myself the effort.

But really, the next thing on the roadmap is to get tighter integration
between our CFM (saltstack/salt-api) and gitlab. Tie it into deploys, auto-
open issues on failed states, etc., though that's gonna be more on the salt
side than the gitlab side and likely wouldn't make sense for you guys to
implement anyways.

Im sure I'll end up setting up whatever does make it into 8.9 a week before
it's released though.

~~~
sytse
It would be nice to upload Salt modules directly from GitLab.

Since you can deploy from GitLab too, auto-opening issues for failed builds
makes sense. I opened an issue on [https://gitlab.com/gitlab-org/gitlab-
ce/issues/17771](https://gitlab.com/gitlab-org/gitlab-ce/issues/17771)

I predict you'll be setting up different environments a week before 8.9 since
that is what we'll ship.

------
nikolay
At this pace of innovation, GitHub will soon be a part of the history! GitHub
has done a tremendous job with forking and pull requests, but, at least
recently, they've acted disorientedly! I hope they get back into shape!

~~~
r3bl
I really don't know what to hope for.

All that I do now that I really enjoy watching what looks like a full on
feature battle between GitLab and GitHub going on at this moment. As a guy who
uses both (GitLab for my job, GitHub personally), I feel like I'm getting the
best of both worlds.

 _Although_ , to be perfectly honest, it feels to me like, at this point of
the "feature battle", GitLab started embracing and actually improving every
single thing I liked about GitHub to the point where I'm questioning my stay
on GitHub for personal projects.

~~~
andromeduck
now if only they could get gitlab.com page loads consistently under 1
second...

x_x

~~~
nikolay
Yup! It supposedly is written in a much more efficient language, but it
severely slower than GitHub and Bitbucket.

~~~
ymse
It's written in Ruby like Github, but the latter have considerably more
experience running high-traffic systems.

Just perused their job page and will apply this week. Sound like they could
use some Varnish and Ceph talent :)

~~~
sytse
For sure we're interested in people that have experience with Ceph.

------
andrewmunsell
Everyone at GitLab has been doing an amazing job shipping some really cool
features.

I'm really looking forward to trying out the new container registry and move
away from my hack-y solution that I use right now to build Docker images on my
own VM, and move back to the shared runners :)

~~~
sytse
Thanks Andrew, I hope your move goes smoothly. We want to allow everyone to go
frictionless from idea to production. We're extending GitLab so this can be
done simply without having to script things together yourself.

------
diggan
I'm excited for this, seems like GitLab is moving more into a all-in-one
solution, compared to Github that focuses on "social coding", whatever that
now means.

So to try out this new feature (together with the pipelines), I tried setting
up a simple project that uses a docker image to serve a simple html page.

However, it seems like it's not possible to build/push from the CI system
(unless you setup a self-hosted runner) which kind of leaves this "Container
Registry" without value, because I still need to manually build/push my images
from one machine...

~~~
diggan
Ah, thanks to sytse, Snappy and tmaczukin for the replis. I missed that.

My configuration looks like this currently (and I'm guessing I'll hold of for
a few days for the shared workers to get updated):

    
    
      image: docker:latest
      services:
      - docker:dind
      stages:
        - build
        - deploy
      build:
        stage: build
        script:
          - docker build -t registry.gitlab.com/victorbjelkholm/deploy-html-test:latest .
        only:
          - master
      deploy:
        stage: deploy
        script:
          - docker push registry.gitlab.com/victorbjelkholm/deploy-html-test:latest
        only:
          - master

~~~
Snappy
If you're going to use the shared runners, I don't think that configuration is
going to work. The build step will build it locally, and even though you named
it to use the registry, it won't actually push to the registry automatically.
By breaking up the push to another step, you're actually going to run that
deploy stage on a fresh Digital Ocean instance, so your previously-built image
will be lost.

That's why the example pushed the image right after the build step.

Having said that, if you're literally using that script, you don't even have
tests, so you may as well just put both steps into a single build stage like
the simplified example. :)

For now, you'll need the explicit `docker login` as well. We'll work to remove
that.

------
cridenour
After GitHub moved to unlimited repositories, I started to question if our
small team should stay on GitLab - and this just triple-confirmed that we
will.

Awesome feature for an already awesome product. Great job!

~~~
sytse
Thank you very much much. I hope your switch goes smoothly.

------
yoavm
Up until recently I never even considered using CI in my projects because I
just didn't have the time. The way GitLab are implementing all these tools
makes it not only easy to use, but fun.

~~~
sytse
Glad to hear that! This was our intention. CI and CD is something everyone
knows they should do but it isn't always done. By making it easier to set up
we try to get more people to do it and feel good about themselves.

------
Fenntrek
Interesting move.

Seems Gitlab are moving fairly fast (compared to competition) and going for
that all in one ecosystem (If you want it) from SCM to deployment with the
range of products they've introduced.

~~~
ayufan
We are thinking big. The integrated Container Registry is part of our plan. We
will soon have an integrated deployments with easy to access environments. We
also plan to introduce manual actions on pipelines allowing you to execute
arbitary things, ex. promote to production, but also merge it back to the
master.

[https://gitlab.com/gitlab-org/gitlab-
ce/issues/17009](https://gitlab.com/gitlab-org/gitlab-ce/issues/17009)
[https://gitlab.com/gitlab-org/gitlab-
ce/issues/17010](https://gitlab.com/gitlab-org/gitlab-ce/issues/17010)

------
sotojuan
Other good news for GitLab: Zach Holman just joined them as an advisor!

[https://mobile.twitter.com/holman/status/734842346278244352](https://mobile.twitter.com/holman/status/734842346278244352)

~~~
sytse
HN discussion of that tweet in
[https://news.ycombinator.com/item?id=11756674](https://news.ycombinator.com/item?id=11756674)
BTW that post and this post are 1 and 2 on HN right now

------
avitzurel
Gitlab is what Github was 3 years ago.

They are on a roll, rolling out features left and right and seem to nail the
need for every engineer.

Kudos to the team! Keep rocking!

~~~
sytse
Thank you, we're very determined to keep shipping. Please let us know if there
is something you can use that we're not doing yet.

------
iyn
I'm really impressed by the hard work of GitLab team, keep it up!

If anybody from GitLab is reading: any plans for rkt support?

~~~
Snappy
Thanks!

No current plans for rkt, but feel free to create an issue to discuss it.

~~~
hackbinary
For those who don't know what rkt is, I am presuming that it is the App
container runtime: [https://coreos.com/rkt/](https://coreos.com/rkt/)
[https://github.com/coreos/rkt](https://github.com/coreos/rkt)

------
nhumrich
Can I use gitlab container registry with gitlab.com? Can I use private images
in said registry with public gitlab.com ci builders? I haven't found a way to
use private containers with gitlab.com ci yet without spinning up a worker for
every project. Maybe a way to register a "team wide" builder?

~~~
ayufan
Currently not. The limitations can be found here:
[http://docs.gitlab.com/ce/container_registry/README.html#lim...](http://docs.gitlab.com/ce/container_registry/README.html#limitations).
We plan to improve that in one of upcoming releases.

------
alfg
So awesome! I've been invested into the Docker ecosystem lately and one of the
next things I've wanted to setup is a decent CI workflow.

I wish Github would also ship more features at the pace Gitlab is. The only
issue I've had with Gitlab are the response times on their UI. Hopefully they
sort that out soon.

~~~
jobvandervoort
I suppose you are using GitLab.com. We're working on improving the speed [0].
If you are willing to run your own instance, it should be fast.

[0]: [https://gitlab.com/gitlab-
com/operations/issues/42](https://gitlab.com/gitlab-com/operations/issues/42)

------
znpy
It's happening: GitLab has basically filled the feature gap with GitHub and it
is surpassing it.

~~~
voltagex_
Once the performance gap closes it'll be even more exciting.

I just worry about 10, 15, 20 years in the future when ArchiveTeam has to back
up GitHub and GitLab and whatever else... there's so much data!

~~~
znpy
There will also be bigger storage :)

------
educar
I wish Gitlab had an easy clean way to import GitHub. The current method
involves setting up oauth information in the config file and restarting
gitlab. Not nice :/ I fail to see why they cannot just work with GitHub API
tokens for import?

------
mikewhy
This is great! But I'm curious if anyone at Gitlab can comment on what this
gives over running the existing docker registry? Can we configure this to use
S3 to store the resulting images?

Our runner image has docker, we run our tests with docker-compose, if they
pass we push them to our existing registry. In fact our .gitlab-ci.yml looks
very similar to the example under "elaborate example" in the blog post.

Just wondering what we'd be missing out on if we didn't switch to Gitlab
Container Registry.

~~~
ayufan
The biggest plus of using integrated registry is that you have integrated
authentication and authorization of GitLab that follow your groups and members
assigned to your GitLab projects, making it really easy to have private
container repositories stored on registry.

Second the built-in registry is really easy to configure and maintain. You
have to specify the address and provide a certificate to start using it. You
also use the same backups mechanism as you use for your GitLab installation.

In the future we may make it possible to use external storage for container
images, like S3. This is something that is already supported by
docker/distribution.

~~~
fapjacks
I am _super_ stoked to see this and will be using the crap out of it and
pointing others to it (the registry is kind of a pain to get going)! It looks
like this is just the v2 registry (from Distribution) integrated into Gitlab,
so I'm wondering what's stopping me from backing this registry with S3? Is it
just not supported by the Gitlab config yaml? I back my private registry with
S3 and it's just a couple of config options to enable it. Or am I
misunderstanding some fundamental concept here? Thanks for the _awesome_ work!

~~~
sytse
Glad to hear you're super stoked! I think you're on the money regarding the s3
backup. I think it is making the configuration accessible. I expect you can
work around that by doing it yourself.

------
allan_s
In the same idea, has someone already worked integration with a debian
repository (with aptly or similar) and has some linked to share on how to do
it the smartway?

We're thinking about using FPM to create the debian package Package which is
the retrieve as the artifacts of the gitlab-ci build stage

And then to have a separate service that will receive a webhook at the end of
the (successful) build, to retrieve the artifacts and update a repository
using aptly.

Does it seems the right way or is there some much simpler solution ?

~~~
tostaki
At my work, we have a setup close to this. On a successful build of jenkins
(after build action), a deb is generated with FPM (awesome piece of software
btw) and uploaded to an s3 bucket. Then on another machine, a cron that run
every minutes sync the s3 bucket to a local aptly repository and publish it to
another s3 bucket which is the real debian repository.

This works well but the s3+cron part is probably not the smartest way and
publication can take some time (like 5-10min).

~~~
voltagex_
I know it probably doesn't matter too much for internal use, but what does
Lintian say about your packages?

~~~
tostaki
Many bad things! I didn't check until now but it's a bunch of "non-standard-
dir-perm" and "dir-or-file-in-var-www" and things like "maintainer-name-
missing". I think it's possible to build correct debian package with FPM but
for our use, we just want to build easily native deb package.

~~~
dozzie
> I think it's possible to build correct debian package with FPM [...]

Of course it is possible, it's just ridiculous amount of work. It's much
easier to build package with proper tools in the first place.

------
thebouv
I've tried several times to figure out how to get Docker working in a
situation like mine. And we've been considering GitLab as well. So this is
likely a good time to experiment.

Doing all the research on how to integrate Docker into my particular situation
has been daunting. I really need to track down some online courses or
something. The articles just aren't cutting it. Or need to find a mentor just
to ask stupid questions to.

------
Tomdarkness
Looks great. One thing that does not appear to be clear though is if you build
multiple docker images from the same project.

Our deployment workflow at the moment builds two docker images, one for the
web app and another for the background services. Both share the same code.

It would appear you can only have one docker image per project?

~~~
hackerboos
AFAIK you can just have 2 stages to build each image and push both under
different tags.

~~~
sytse
You can do this but currently you can have only one image per project. We did
this to keep it simple. You can use a dummy project to work around it. Or
consider making web app and the background services separate projects.

------
netcraft
am I correct that this is only for on-prem deployments, that they aren't going
to be offering a public container registry a la docker hub or quay?

~~~
tmaczukin
You can enable Container Registry for your projects on GitLab.com :)

And in a few days our shared runners should be updated to support the full
docker workflow with the Container Registry.

~~~
netcraft
thats really amazing - are there size limits?

~~~
sytse
Our soft limit is 10GB per project for GitLab.com
[https://about.gitlab.com/gitlab-ci/](https://about.gitlab.com/gitlab-ci/)

------
SEJeff
This might sound like a silly question, but does this require gitlab, or can
it work standalone?

~~~
sytse
You install it by installing the GitLab Omnibus package. But you push and pull
to it from other application too, no need to use GitLab.

------
btashton
When would we expect to see this on githost.io?

~~~
dblessing
We are investigating what it will take to enable this on GitHost.io. At first
glance, it doesn't seem to be too difficult. I created
[https://gitlab.com/gitlab-com/githost/issues/12](https://gitlab.com/gitlab-
com/githost/issues/12) to track this feature request.

------
skylan_q
These guys are f*cking killing it. Way to go!

------
davexunit
Awesome, a new place to host your unpatched, opaque disk images!

~~~
sytse
I think the patching of Docker container images is a big problem. We try to
provide a completely automated flow in GitLab using CI, CD, and the container
registry. I would love to have a big button that says 'update all my
containers'. I've made an issue [https://gitlab.com/gitlab-org/gitlab-
ee/issues/592](https://gitlab.com/gitlab-org/gitlab-ee/issues/592)

~~~
davexunit
The problem is that Docker, and more generally using stacks of binary disk
images, is fundamentally flawed with respect to security and reproducibility.
It's nontrivial to inspect these images for vulnerabilities because there is
nothing that specifies the _precise_ set of software that is in that image.
Some stuff is compiled from source, some stuff is installed via one or more
package managers, each of which may bundle additional software that one may
not know about if they didn't inspect each piece of software carefully.
Furthermore, one cannot even verify the image reasonably because the results
are different depending on when it was built.

In short, container images as popularized by Docker are insecure by design.

~~~
moondev
Don't you think that gitlab adding container registry support will encourage
more people to build their own images, as opposed to trusting "black-box"
images from elsewhere? Combining a base image with the deployable artifact is
much more efficient than baking amis or other images. You then know exactly
what your image contains and nothing more, because you built it yourself.

Of course I'm looking at this from the perspective of deploying micro-services
and immutable infrastructure. Your use-case may be different.

~~~
davexunit
Any nontrivial image relies on a large number of other images as a base, and
just because you built it yourself doesn't mean that you didn't just download
and install software with known security vulnerabilities.

~~~
snuxoll
Yes, but you will ALWAYS have that issue with software you build yourself.
This is primarily why I maintain an in-house yum repository for anything that
we use out of our distribution repositories, it takes a little more effort to
build RPM's but it's worth it from a maintainability and security aspect (as I
type this I'm working on a salt-based package management tool somewhat akin to
Katello/Sattelite that I can use to manage upgrades that Puppet doesn't).

