
Jenkins X: a CI/CD solution for cloud applications on Kubernetes - Garbage
https://jenkins.io/blog/2018/03/19/introducing-jenkins-x/
======
kosta_self
Does anyone have Jenkins in production and found it to be reliable and
pleasant to use?

I had to do lots of Jenkins integrations some time ago, and even though I
tried to minimize the number of plugins and make things as simple as possible,
things would randomly break from time to time or exhibit weird behaviour etc.

I had the impression that Jenkins is deeply conceptually confused about some
of its concepts, e.g. how builds are triggered. Also, it is a huge pile of
untestable spaghetti code which explains the weird bugs.

I modified an open-source plugin only to find out that it's almost impossible
to write meaningful tests for it: you can't even mock Jenkins API without
using the darkest Java mock magic. Jenkins classes are just written in an old
style that makes testing _really_ hard but probably can't be changed without
breaking all of Jenkins.

I tried Jenkinsfile which was only even more unreliable (at the time at least,
this was > 1y ago). The whole idea of using groovy, modifying the hell out of
it to make it even more weird and surprising and edge-casy just didn't go well
for me.

I ended up with generating a _lot_ of very simple jobs for each project and
connecting them via triggers instead. It was not very pretty, but it was the
most reliable that I could get out of Jenkins.

So the thought of integrating Jenkins deeply into you deployments, talking to
Kubernetes and sitting in the middle of a huge pile of complexity
(Jenkinsfile, Dockerfile, Helm, ...) and magic "that you don't need to worry
about" scares the hell out of me.

But then again, if you want to do CI/CD with Jenkins that's what you might
want, right? (I would prefer more simple approaches if forced to use Jenkins,
though)

~~~
nobleach
Pleasant? Never. I absolutely hate every time I have to launch a webpage to
dig around and find out why my build failed. Automating jobs is a nightmare. I
can cURL a request to kick off a build and do you know what it returns?
NOTHING a 201 response with NO information to link to the build in progress.
Oh there's a JSON API to see running jobs, but without some sort of ID, it's
useless. Scripting is in Groovy. Want to use another language? Too bad! "It
was hard enough to get Groovy" is the response from the team. If I'm forced to
use a web interface, does it still have to look like it was designed by a team
of Java devs from 2008? The only thing that's changed is the Hudson to Jenkins
clipart. Yes, I'm ranting.

~~~
vorg
> "It was hard enough to get Groovy"

To make it work, they had to cripple Apache Groovy so you can't use its
functional collections-based functions. Not sure if you can really call it
"Groovy" with that handicap.

------
cfontes
This looks like an improvement over what Jenkins 2.0 provides and I wish you
guys good luck.

I have used and vouched for Jenkins in several companies and some decent sized
licenses were bought mainly because of my input.

But to me, Cloudbees has done a major dick move making the stages not
restartable in Jenkins 2.0, among other things. E.g: Dropping stage view out
of nowhere and focusing only in Blue Ocean. I complained about in the channels
that I had at the time and the response was, it's going to be unsupported from
now on.

It's a super squechy thing to have such a useful feature bundled with a bunch
of support and useless stuff that I don't care, and then charge me per node. I
am migrating away from Jenkins into GoCD after close to 10 years using it, and
don't get me wrong I don't feel happy doing this, but it's hard to justify it.

Fortunately the future looks bright, there are several interesting solutions
available, Argo is super interesting to me, looking forward for Argo CD!

~~~
theptip
How are you liking GoCD? I've been looking for an alternative to GitLab for a
while.

~~~
sytse
What can we improve in GitLab that would make you stay?

~~~
theptip
Thanks for taking the time to ask.

My primary concern is that instead of polishing the features that have already
been released, the platform is trying to do too many new things. Some of that
stuff is cool (k8s monitoring integration, though EEP is too expensive for me,
and my Grafana dashboard does basically the same thing), while some of it
seems a bit bloated (SAST/DAST for example, which was a few lines of code to
implement ourself).

I really want the core Github replacement use-case to be as ergonomic as
Github is. And the CI/CD piece is also great, but still has plenty of rough
edges (e.g. Environments are a great feature, but I still can't clean up stale
ones, which makes the environment list basically useless).

My experience with support has been a bit lackluster; e.g. see
[https://news.ycombinator.com/item?id=16897644](https://news.ycombinator.com/item?id=16897644).

General reliability in CI/CD is not great; I'd say something like 0.25-0.5% of
my build jobs fail from intermittent infrastructure failures (mostly gitlab
runner/API issues from what I can tell), which wasn't a problem when I was
using Jenkins.

Ops is still a significant concern; site reliability has improved in the last
year, but that's not saying a lot; it's still a fairly frequent occurrence to
get errors during/after a deploy. I'm not sure if this problem would be better
or worse if I self-hosted, as I don't know how hard it is to run a GL instance
(seems like it's hard, given how often the gitlab.com site has issues).

Performance has also improved in the last year, but the site is still on the
slow side (e.g. compared to contenders like gitea).

Oh, and the pricing model is a bit broken -- all of the other SaaS platforms
that I use let me pay monthly (at a higher rate); when I was evaluating paying
for gitlab.com vs. doing self-hosted EE, I really wanted to pay for my team to
use the hosted offering for a few months to see how things went, but I wasn't
prepared to lock in for a year, so I didn't end up trying out the hosted paid
offering.

None of these points in isolation is enough to make me leave the platform and
go back to Jenkins, but they are enough to make me pay close attention to the
alternatives.

~~~
sytse
Thanks for voicing your valid concerns.

Regarding polish we're improving existing features all the time, we're very
close to a big refactor of the merge request view.

Regarding CI/CD environment deletion I can find it in the API
[https://docs.gitlab.com/ee/api/environments.html#delete-
an-e...](https://docs.gitlab.com/ee/api/environments.html#delete-an-
environment) but I'm not sure about the interface.

Support has had trouble scaling, we just hired a director to make sure we get
on track, sorry about that.

Having you builds fail intermittently is bad, this should be a problem only on
GitLab.com Reliability of that is not where it should be and we're taking
drastic actions to improve it. If anyone reading this is up for the challenge
please see
[https://jobs.lever.co/gitlab/a9ec2996-b7b6-4d87-aed0-1fc2ce3...](https://jobs.lever.co/gitlab/a9ec2996-b7b6-4d87-aed0-1fc2ce3f8faa)

Regarding the yearly pricing this is a tradeoff we made, see point 7 of
[https://about.gitlab.com/handbook/product/pricing/#when-
is-a...](https://about.gitlab.com/handbook/product/pricing/#when-is-a-dollar-
not-a-dollar) I thought we offered a 30 day money back guarantee but I can't
find it
on[https://about.gitlab.com/pricing/](https://about.gitlab.com/pricing/) so
I'll ask product marketing what is up with that.

~~~
jhurewitz
It is actually 45 days and in our subscription terms, Section 5.2: "If
Customer terminates this Agreement pursuant to Section 6.2 within 45 calendar
days from receipt of the initial invoice for the Licensed Materials, GitLab
will refund all Fees paid hereunder."

------
fredsted
So we already have a very custom Jenkins setup that builds containers, runs
tests and creates test environments out of our pull requests in a k8s
namespace. This seems to come with so many things that we already have.

We have a few issues with it, like Jenkins suddenly deciding to build & test
all branches/PRs in all repos, killing the server.

• What is Jenkins X exactly, and how does it relate to Jenkins? Is it just a
CLI utility that generates git repos, k8s clusters and Jenkinsfiles for us? Is
it a fork of Jenkins?

• What would we gain from switching to Jenkins X?

• How does it work with our existing setup?

~~~
jstrachan
The things you gain from switching to Jenkins X:

* automated CI/CD for your kubernetes based applications using Helm Charts & GitOps to manage promotions (manual or automated)

* a single command to create a kubernetes cluster, install Jenkins X and all the associated software all configured for you OOTB (including Jenkins, Nexus, Monocular etc): [http://jenkins-x.io/getting-started/create-cluster/](http://jenkins-x.io/getting-started/create-cluster/) \- ditto for upgrading

* a single command to create new apps or import them via build packs to create docker images, pipelines and helm charts with GitOps promotion: [http://jenkins-x.io/developing/create-spring/](http://jenkins-x.io/developing/create-spring/)

* automated release notes + change logs with links to github/JIRA issues etc

* feedback on issues as they move from Staging -> Production

i.e. more automation around CI/CD and kubernetes so you can spend more time
focussing on building your apps and less time installing/configuring/managing
Jenkins + Pipelines

~~~
Rotareti
Does this work on DigitalOcean as well?

~~~
jstrachan
It should do - in theory - it needs testing though ;).

I notice that kops recently added support for Digital Ocean
[https://github.com/kubernetes/kops/blob/master/docs/tutorial...](https://github.com/kubernetes/kops/blob/master/docs/tutorial/digitalocean.md)

So we should be able to add a command `jx create cluster do` for using kops on
DO - the current `jx create cluster aws` uses kops under the covers to spin up
the kubernetes cluster.

I’ve raised this issue to track it:
[https://github.com/jenkins-x/jx/issues/705](https://github.com/jenkins-x/jx/issues/705)

------
zippie
I find this solution compared to the gitlab Auto Devops, frankly,
underwhelming.

We recently deployed AD in our self hosted gitlab instance and combined the
SAST container checks with our production policies, it’s been rock solid.

Add to this the fact we are able to manage all the production policies via the
pipeline API’s and AD templates, the whole Jenkinsfiles deal seems far less
scalable and difficult.

I have no affiliation with gitlab.

~~~
jstrachan
Sorry to hear you're underwhelmed! Did you check out the GitOps features for
versioning all changes to all Environments in git with human approval?
[http://jenkins-x.io/about/features/#promotion](http://jenkins-x.io/about/features/#promotion)

Or the automated feedback on releases to all your issues as they move through
Environments:
[http://jenkins-x.io/about/features/#feedback](http://jenkins-x.io/about/features/#feedback)

Or the automatic publishing of Helm charts to the bundled Monocular for all
versions of your apps for your colleagues to easily be able to run via helm?

Or that it works great with GitHub, GitHub Enterprise & JIRA and has awesome
integration with Skaffold?

Or easy setup a kubernetes cluster with Jenkins X on any public cloud in one
command: [http://jenkins-x.io/getting-started/create-
cluster/](http://jenkins-x.io/getting-started/create-cluster/)

------
jillesvangurp
This solves part of a real problem. There are a lot of tools out there but end
to end integration is left as an exercise to the reader. I'd love some out of
the box sanity when it comes to devops, CI/CD, and cloudhosting. Instead I'm
stuck building nw architectures, deployment scripting, dealing with archaic
broken by default configs of misc bits and pieces, host configurations, build
servers & CI/CD pipelines, etc. from scratch. Add log aggregation, security
auditing, and other nice to haves that are actually not optional these days
(what are you running blind and ignoring failed ssh attempts?) and you have a
full explanation why so many companies make a mess of all of the above.

IMHO a lot of stuff in this space are either focusing on making life harder
through added complexity to up-sell support or more services or on solving a
narrow problem in such a way that you have to still take care of a lot of
other stuff.

------
dankohn1
Lots of innovation occurring in the CI/CD space around Kubernetes. We just
added JenkinsX to the cloud native interactive landscape:

[https://landscape.cncf.io/grouping=landscape&landscape=ci-
cd...](https://landscape.cncf.io/grouping=landscape&landscape=ci-
cd&selected=jenkins-x)

------
jdc0589
Looks pretty cool.

I really don't like the idea of my ci/cd tooling being responsible for
provisioning its own k8s cluster....there are a lot of other more mature
projects out there for doing this.

Is the idea that the ONLY thing running on this cluster is jenkins-x and
review/preview environments or something?

~~~
jstrachan
Thanks! You are free to use any kubernetes cluster and install Jenkins X
there: [http://jenkins-x.io/getting-started/install-on-
cluster/](http://jenkins-x.io/getting-started/install-on-cluster/)

The default is to use separate namespaces in kubernetes for each teams
developer tools & pipelines, Staging & Production environments (plus Preview
Environments). Multiple teams can obviously use the same cluster with
different namespaces.

We’d expect ultimately folks may want to use a separate cluster for
development & testing to Production. GitOps makes that kind of decoupling
pretty easy but we’ve still work to do to automate easily setting up a multi-
cluster approach:
[https://github.com/jenkins-x/jx/issues/479](https://github.com/jenkins-x/jx/issues/479)

------
dcosson
It's not completely clear to me from reading the site - does this run a non-
dockerized app build in kubernetes, or does it also work for building and
deploying my app as a docker container itself? This usually requires things
like being able to spin up a cluster of containers per build - one with my
app, one with a database to run against, maybe one with memcached or
elasticsearch for integration tests, etc. And does it work out of the box for
complicated cases like partitioning a large test suite to run in parallel,
where each parallel part of the build needs its own mini cluster of a couple
of containers talking to each other?

I haven't looked into the current state of this recently but I ran into a lot
of problems with this with a bunch of hosted CI services in the past. Somewhat
ironically, as of a couple years ago if you needed to build your own docker
container as part of a build you had to specifically stay clear of CI services
that mentioned docker at all because that meant they were running their builds
inside of containers and it was a pain to figure out how to run my own docker
build, much less spin up a cluster per build with something like docker-
compose, inside of a running container.

Curious if and how Jenkins X solves this. Or have things changed and it's now
easy to build and run docker containers inside of a container?

(Aside from that, I'm not sure how I feel about Jenkins coordinating with a
Kubernetes cluster. I've always found their monolithic approach to be a pain
to work with, and always wished that, for example, I could just have Jenkins
trigger jobs by pushing them onto an ActiveMQ queue or something and read back
the results on another queue. Then I could just set up an autoscaling group of
build servers, and provision them with whatever tools I'm already using to
just start up and listen on this queue. Instead, jenkins wants me to duplicate
a lot of this work I already have CM tools doing, and set it up manually
through the UI, using community plugins that are often out of date).

~~~
dankohn1
Some useful work on building Docker containers without root:

[https://cloudplatform.googleblog.com/2018/04/introducing-
kan...](https://cloudplatform.googleblog.com/2018/04/introducing-kaniko-Build-
container-images-in-Kubernetes-and-Google-Container-Builder-even-without-root-
access.html)

~~~
jstrachan
Agreed! Jenkins X uses Skaffold to do the docker image building so can be
configured to use kaniko or Google Container Builder etc
[https://github.com/GoogleContainerTools/skaffold](https://github.com/GoogleContainerTools/skaffold)

------
theptip
Looks interesting,

* I use one namespace per env (staging, prod, etc), is this supported or must I go with the default (slightly wacky) staging and prod releases side-by-side in the same namespace?

* How are bugfix releases handled? If I pushed 1.2.0 to staging, and want to hotfix the prod release 1.1.0 with 1.1.1 (a common bugfix flow), can I promote releases from the hotfix branch?

* Is there a permission model? Does it bottom out to GitHub permissions for each env repository? E.g. can I have a smaller set of users approved to promote releases to production?

~~~
jstrachan
Each environment is in a separate namespaces. You can add/edit/delete the
Environments to use whatever namespaces you wish to use.

Promotion is either automatic or manual. By default Staging is automatic and
production is manual. You can manually promote any version to any environment
whenever you wish:
[http://jenkins-x.io/developing/promote/](http://jenkins-x.io/developing/promote/)

For promotions we're delegating to the git repository for RBAC; so you can
setup whatever roles you want for who is allowed to approve promotions & if
you need code reviews etc

------
djabatt
Automation is never easy. If you zoom your focus out watching the innovation
in the ci/cd space is fascinating. Not to sound like an old fart but doing
this all by hand back in the day sucked. Jenkins for us has made a lot easier.
Curious to try GitLab and Jenkins X on a new projects

------
tra3
I can't find this in the docs; but what happens when a promotion fails? Do
things get rolled back to previous known state? The reason I ask is because
I"m trying to replicate similar functionality in our much simpler environment.

~~~
jstrachan
we're using helm for installing/upgrading apps; it takes care of reverting bad
versions/releases etc

~~~
tra3
Thanks, that's helpful, but we're not mature enough for k8s yet. What else can
I look at?

~~~
jstrachan
if you can't use kubernetes or containers then I guess Ansible is a fallback?

------
C4stor
Interesting annoucement, I'd like it better if there was a clear comparison
between the current possibilities offered by Jenkins 2.0 and this version of
Jenkins.

I'm not huge fan of the demo video, since it doesn't really handle what I can
only imagine is a very common use case : I already have a Jenkins 2.0 instance
with Jenkinsfiles, how easy would it be to migrate to Jenkins X ? Is it
isofunctional with added capabilities ? How much will I lose ?

Bootstrapping a java spring app from scratch is fun, but I suspect most people
have an already existing codebase with already existing CI/CD tools.

~~~
tootie
I'm leery of projects whose goal is make bootstrapping easier. Bootstrapping
projects has generally gotten easier and easier and was never a real
bottleneck. Projects spend 99% of their lifetime in development and
maintenance so those are the parts that need the most help.

------
tlynchpin
Is it appropriate to compare this with OpenShift, and if so then what are the
highlights of this comparison?

I use Jenkins and k8s and the objective is generally Spring services, so it
sounds like this is for me.

~~~
jstrachan
OpenShift is Red Hat's supported fork & distribution of Kubernetes - so its
another platform we can install and use Jenkins X on.

OpenShift also includes some Jenkins support; e.g. you can add BuildConfig
resources via a YAML file in the OpenShift CLI which will create a Jenkins
server and a pipeline. But Jenkins X isn't yet integrated into OpenShift - but
its easy to add yourself for now :)

If you are pondering which kubernetes cluster to try for developing Spring
services: OpenShift is a good option if you are on premise. If you can use the
public cloud then GKE on Google is super easy to use; AKS on Azure is getting
there & EKS is looking like it will be good if you use AWS.

On the public clouds the managed kubernetes services are looking effectively
free; you just pay for your compute + storage etc. So its hard to argue with
free + managed + easy to use kubernetes - if you are allowed to use the public
cloud!

~~~
dankohn1
OpenShift is Certified Kubernetes, so not a fork.

See: cncf.io/ck

(Disclosure: I run the Certified Kubernetes program at CNCF.)

~~~
jstrachan
OpenShift is a fork of the Kubernetes code base:
[https://github.com/openshift/origin](https://github.com/openshift/origin)

I.e. it’s not using the upstream distribution of Kubernetes like the public
cloud vendors or Heptio etc.

It’s great it’s a Certified Kubernetes though!

~~~
dankohn1
Fork has a perjorative meaning, as in taking it and going their own direction.
I would describe OpenShift as a distribution.

------
amq
When I started using Travis I immediately got a fan of it. Not having worked
with Jenkins before, when I first tried it (pre-blue ocean), I was shocked by
the unnecessary complexity. Since then, I've settled with a self-hosted
drone.io for private projects, it offers a very similar experience to Travis,
while I don't feel like I'm lacking anything compared to Jenkins.

~~~
greenhatman
Is the self-hosted version the same as the hosted enterprise version? Or are
there features missing? Is it easy to self-host?

------
tirumaraiselvan
If I understand correctly, this creates an environment for each PR. How does
it accomplish that exactly? It would require all Kubernetes manifests for
resources somewhere in the repo? What if the environment has some stateful
dependencies, etc?

~~~
jstrachan
Jenkins X creates a Preview Environment per Pull Request yeah; which can be as
much or as little as you want it to be. e.g. it could be just 1 pod only or
could be a suit of related apps (you may want to test multiple things
together).

You can define what a Preview Environment is in the source code of your
application - its just a different Helm chart really. You can of course opt
out of Preview Environments completely if you wish.
[http://jenkins-x.io/about/features/#preview-
environments](http://jenkins-x.io/about/features/#preview-environments)

Though I've personally found them to be super useful - especially if you are
working on web consoles - it lets you try out changes visually as part of the
Pull Request review process before you merge to master.

------
mekazu
Just remember the name XKCD is already taken.

------
adam-_-
Presumedly this is only of interest for those running Kubernetes?

~~~
jstrachan
Btw I’m the author of the above blog post & committer on Jenkins X.

So our focus is currently anyone looking to automate CI/CD on kubernetes, the
cloud or any modern platform like OpenShift, Mesos or CloudFoundry which all
come with kubernetes support baked in.

You can use just the CI part and do CI & releasing of non-cloud native apps if
you want - we use Jenkins X to release jars, plugins & docker images using it
- but doing so does miss out all the benefits of automated Continuous Delivery
& GitOps promotion across environments

------
nodesocket
No TLS on jenkins-x.com, tisk tisk.

~~~
tuananh
it's using github pages for the main website i think so i'm not sure if it's
possible to setup custom domain + custom ssl cert for it?

~~~
nodesocket
Should be able to use CloudFlare and "flexible" SSL setup. CNAME at the apex
pointing to GitHub pages.

~~~
LethargicStud
Please don't do this. It gives the user the illusion that their connection is
secure, but the connection between Cloudflare and the site is not secure.
Arguably it's better to encrypt some of the route rather than none of it, but
also giving people a false sense of security comes with its own drawbacks.

~~~
nodesocket
Actually "flexible" might not be needed, "full" without strict should work.
Traffic is still over TLS, but a valid named certificate is not required.

------
ku6pk
In no attempt to bash OP at all, but what is the HN policy on links that have
been posted before?

Edit: Previous posts for reference

[https://news.ycombinator.com/item?id=16622215](https://news.ycombinator.com/item?id=16622215)

[https://news.ycombinator.com/item?id=16626765](https://news.ycombinator.com/item?id=16626765)

[https://news.ycombinator.com/item?id=16628313](https://news.ycombinator.com/item?id=16628313)

[https://news.ycombinator.com/item?id=16658640](https://news.ycombinator.com/item?id=16658640)

~~~
tomhoward
It's only considered a dupe if the story/topic/product release has had
significant attention (front page presence and discussion) in the past 12
months. None of these posts got much attention.

See dang's comments on the issue for the official position:

[https://hn.algolia.com/?query=dang%20dupe%20porous&sort=byPo...](https://hn.algolia.com/?query=dang%20dupe%20porous&sort=byPopularity&prefix&page=0&dateRange=all&type=comment)

~~~
ku6pk
Thanks for the informative post and link.

~~~
tomhoward
No worries! By the way, it's frowned upon if one person posts the same link
repeatedly without getting any traction - eventually it just gets annoying,
especially for people who watch the 'new' feed and keep seeing the same thing
coming up. But the above posts seem to be submitted by different people, so
it's all good.

