
Shifting Gears - twic
https://jenkins.io/blog/2018/08/31/shifting-gears/
======
nrclark
I love Jenkins, and use it professionally.

With that said, I'd really like to see better documentation on the Jenkinsfile
Pipeline format. I've tried to get started with it a few times, and haven't
had tons of success. Stuff like "How do I pull in secrets", and "How do I
control a plugin". I appreciate that it's Groovy-based, but that's not
particularly helpful information (for a hack like me, at least).

The snippet-generator is nice, but it doesn't necessarily produce working
code. Especially for things like getting secrets into a build. And it doesn't
give me a broader picture for "How do I even write one of these from an empty
text-box".

I recently tried the job-to-Pipeline exporter plugin, and that didn't work on
my jobs - it generated stuff that didn't match the input job, and also wasn't
structured like the example snippets Jenkins provides natively.

Maybe some kind of a sandbox I could experiment in? Or a REPL or something? It
would really help to have something that gave great discoverability, with fast
feedback. Faster than I can get by editing a job, saving it, running it,
waiting, then realizing I still don't have the syntax right.

~~~
shoo
After working in a team that was heavily using Jenkins files & scripted
pipelines I started to believe that writing Jenkins scripted pipelines is a
bit of an anti pattern, as you end up with lots of build script code that can
only run inside of a Jenkins, perhaps coupled to plugins, which hampers your
ability to locally develop and test changes.

Perhaps sometimes using Jenkins scripted pipeline is a good idea, but if
you've got the choice of implementing something as a Jenkins pipeline script
or some other script that isn't coupled to Jenkins, prefer the latter.

~~~
auxym
I work with Jenkins day in, day out.

Doing anything build-script related in Jenkins, whether Pipeline or freestyle
jobs, is definitely an anti-pattern. All build-related scripts should
definitely be in standalone scripts / build tool config files (make or
whatnot), for reasons you describe.

Jenkins should be there to handle the "side effects", as I view them. In our
case that's stuff like integration with git PRs (posting results of linting,
building, unit tests), sending emails when new builds are available,
integration with JIRA (we automate some workflows), publishing artifacts to an
internal server, etc.

Conversely, putting any of those side-effects or stateful steps inside build
scripts is a bad idea, and it leads to not being able to run build scripts
locally without worry of messing up a JIRA workflow or spamming people with
build emails. Thus, they should be stored only in Jenkins.

These are all mistakes of my predecessors that I am still living with to this
day.

~~~
michaelneale
I think thats a great rule of thumb. Declarative pipeline came after the
script was "invented", which is slightly unfortunate, had it came before it
would have encouraged the practices you describe (declarative is just for
orchestration), and script would have been mainly an escape hatch (I think
many people get the idea now though).

------
jonthepirate
I worked on Jenkins at Lyft and completely set it up for DoorDash. If anybody
needs help with their Jenkins setup, hit me up I give free advice and have a
few blog posts on the matter.

If you happen to be using AWS, GitHub, and Slack, we at DoorDash have
developed lots of goodies for streamlining things. We have secured our Jenkins
behind our VPN, created load balanced Jenkins clusters, built a shared Groovy
library for all of the Jenkins behaviors that are useful for each of our
microservices, implemented a Flask app that receives each of the GitHub
webhooks which starts pipelines instantly (rather than git polling), setup
Okta integration, interfaced with our internal secrets store, and implemented
a way to map GitHub users to Slack users allowing us to Slack message people
when they are mentioned in GitHub (when their PR's receive LGTM's etc.) When
new microservices launch, Folders automatically appear in Jenkins configured
correctly for the service's pipelines.

If any of this sounds good let me know, maybe we open source some of our work.
I love working on Jenkins and am happy to help advise you on how to scale,
secure, demystify your own Jenkins setup. Links on my HN profile page.

~~~
kohsuke
You should be presenting in a future Jenkins World event!

------
chinhodado
Jenkins's biggest strength is also its biggest weakness: plugins. Any
development shop that has been using Jenkins for a while is using at least a
bunch of plugins. Plugins are not stable, they break every now and then. They
require constant update with new Jenkins versions. They get abandoned by their
creators (hell, many plugins still don't support pipeline).

It's a fundamental issue with how Jenkins is set up that I don't know how they
can get away with unless they abandon the whole plugin architecture all
together. But obviously that's not a solution.

~~~
oblio
They kind of made several plugins "blessed": Pipeline, Blue Ocean, Git, etc.

The core package plus these "blessed" plugins is a lot more stable than
throwing every random plugin on top of a base installation. Just write a bit
of glue script code and you're golden.

~~~
isbvhodnvemrwvn
They still have their own issues. Blue Ocean requires a lot of stuff I have no
need for (like github support) which in some cases conflict with stuff I do
need (like bitbucket support)

~~~
chinhodado
Same. Every time there's a Blue Ocean update it requires you to update two
dozens other plugins, many of which I don't use and can't get rid of (like the
github one). And more annoying is the fact that you can't "select all" to
update all plugins, you have to select them one by one.

~~~
geerlingguy
There’s a select all link at the bottom of the plugin updates page...

------
mothsonasloth
I think this is too much too late for Jenkins.

I can't speak for other countries but in London a lot of companies are now
using Gitlab or Circle CI.

I migrated all my builds (12 projects) to Gitlab CI. After figuring out the
first CI pipeline using DockerInDocker, it was easy to then setup the
remaining pipelines.

Self hosting Gitlab was perfect for our needs (private docker registry). I use
Gitlab for personal use too.

I wonder if they will get rid of Ruby in the future though and go Java to make
it more performant, as it does slow down sometimes.

The Jenkins box is still running though, more out of sentimental value :)

~~~
sytse
Thanks for using GitLab! I'm glad to hear you found it easy to set up.

I just wrote an article in response to the OP
[https://about.gitlab.com/2018/09/03/how-gitlab-ci-
compares-w...](https://about.gitlab.com/2018/09/03/how-gitlab-ci-compares-
with-the-three-variants-of-jenkins/)

We're working on making GitLab more performant. It is mostly fixes to our
code, the parts where ruby is a problem are already rewritten in Go. GitLab
self-hosted should be fast if it has enough memory, so make sure you check on
its memory consumption.

~~~
yebyen
> The current legacy version of Jenkins needs to be restarted once a day by an
> administrator

Is this true? Do you have a source that says this? We have a Jenkins instance
that Kubernetes is configured to scale down from 1 replica to 0 at night, and
up to 1 again in the morning, so if it is true we never would have noticed.
(It hasn't always run on this cron cycle, which is why I'm a little
incredulous at this claim, but if it's given in the OP or somewhere else easy
to find this, I'll concede it... ah... found it: > It’s not unheard of that
somebody restarts Jenkins every day.)

Honestly I don't understand this about "making a version of Jenkins that runs
well on Kubernetes" – this is the _only way_ I have ever run Jenkins, and I
think it runs already extraordinarily well for our purposes. I'm thrilled that
they are making it their focus, and I'll concede also that our use of it is
pretty narrow, but I haven't had these issues.

We installed it from the stable helm chart nearly 2 years ago and have hardly
needed to make any tweaks. We are not tracking every K8S release, so maybe
that's why I haven't noticed Jenkins falling behind, and we also haven't tried
GitLab seriously (heard great things, but my work is very risk-averse when it
comes to new technologies, and to be honest we rarely try new things on a
short cycle once a given problem has been solved adequately for us... we are
also not primarily a development shop, so maybe it makes sense.)

> The article doesn't mention how Cloud Native Jenkins addresses the problem,
> maybe it doesn't allow plugins.

Like I've been saying, we've always used the stable helm chart for Jenkins and
started maintaining our own values.yaml about a year and a half ago. Over time
we have had less need to change the templates as more configuration got moved
into values.yaml. When I have needed a plugin or other configuration that is
able to be set in values.yaml, that's easy and almost makes maintaining
Jenkins fun. It is a little obtuse that I have to maintain my list of plugins
and their latest versions there manually, but this could be something that
gets resolved in Cloud-native Jenkins if they are ultimately providing an
operator or something like that.

(Breaking a rule by commenting before I've read all of the content, but I
liked your article and wanted to give you some feedback since you posted it.)

For configuration that doesn't live in values.yaml, Jenkins chart maintains a
Persistent Volume where configuration and build artifacts/history are stored.
It is easy enough to take backups of that with the ThinBackup plugin, and the
storage costs of that are sure not breaking the bank.

> Services interacting through Kubernetes CRDs in order to promote better
> reuse and composability

And there it is! That's the big announcement from today. Knative is still
early but this news from Jenkins sounds supportive and I should really read
the whole article / watch the video now.

~~~
jacques_chester
> _Honestly I don 't understand this about "making a version of Jenkins that
> runs well on Kubernetes" – this is the _only way_ I have ever run Jenkins,
> and I think it runs already extraordinarily well for our purposes._

I think the idea is not that Jenkins runs _on_ Kubernetes, which as you note
can already be done. It's rather that Jenkins uses Kubernetes as a replacement
for the worker infrastructure.

~~~
michaelneale
yes that is right I believe.

~~~
yebyen
This can be done with plugins, kubernetes-plugin for example. I'm looking
forward to seeing what they come up with. It was good to see knative on their
roadmap! This could be something else majorly.

------
rosshemsley
I was seduced by BlueOcean and tried using Jenkins for CI, using Github and
AWS ECS builders (which felt like a common enough use-case).

Unfortunately it ended up costing an astonishing amount of engineering time to
get working and maintain, with builds frequently stalled or failing.

Since moving to CircleCI 2.0 enterprise (admittedly far from perfect) and
Airflow, we have _dramatically_ reduced eng. time spent managing our job
scheduling.

The core of our problem was how fragile and complex the Jenkins ecosystem
seems to be: any change to the config or settings and it would easily burn a
day of engineering, due to random bugs and hard to understand error messages.
In the end, no one wanted to touch it!

I think there's a great project hidden somewhere here, but just getting the
basic "everyday" stuff done with it can be a real PITA.

~~~
kohsuke
I'm sorry to hear the bad experience.

I recognize those challenges in my pitch, we have various efforts already
under way to address them, and with this gear shifting, I think we'll be
combining those in a compelling way.

For example, defining Jenkins config in YAML in Git is a key piece to solve a
fear of config change, and this is called "Jenkins Configuration as Code" and
is under way for a while now.

Cloud Native Jenkins will also split single process "master" into many build-
as-a-function kind of processes, so it isolates builds and allows changes to
be rolled out more incrementally.

There's more focus on us owning a bigger responsibilities around "basic every
day stuff," too.

------
ak217
Having used Hudson/Jenkins for many years, I recently considered setting it up
for a new project, and backed away mostly due to the issues Kohsuke describes.
We ended up choosing GitLab instead.

GitLab has been pulling ahead in features and usability, compared to other
things I've tried. Right now, different projects I'm involved with use a
combination of GitLab Enterprise, Travis, Circle, and Google Cloud Build. Of
those, GitLab accommodates the heaviest and most sophisticated workloads,
without having to go through too much trouble to set up, maintain, and
instruct developers how to use it (certainly less trouble than Jenkins). I
highly recommend taking a critical look at all of these services, to see which
best fits your needs.

~~~
Sir_Cmpwn
If you don't mind, I'll shill my service as another option: builds.sr.ht. It's
still in closed alpha, but it's being used seriously by several open-source
projects for complex build automation. It also deploys itself, here's the
build manifest which does it:

[https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/.build.yml](https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/.build.yml)

And an example of a build which used it:

[https://builds.sr.ht/~sircmpwn/job/6974](https://builds.sr.ht/~sircmpwn/job/6974)

If you or anyone else would like to try it, please let me know. I used Jenkins
for a long time (and still do at work), Travis for a while as well, also tried
Drone and Circle, but none of them were exactly right. I think builds.sr.ht
does it very well.

------
cfontes
Nice to know there is a plan and it's refreshing to see that they can
understand most of the problems from the customer perspective now.

I hope they address the constant shifts in focus with this plan and Jenkins
can secure it's market spot, it really deserves it from a historical point of
view in the least. It should not be a Nokia or a Xerox, it's better than that
and has been a major tool for the industry.

The whole CRD is a great way to move forward, but Argo is looking great right
now and it's way ahead, if they manage to finish it soon and make it
production ready it will be hard to beat it.

The problem is that every segment has it's player now and there are some big
ones, GiLab, GoCD, Spinnaker, Concourse... So many tools and the difference is
that most of them have more focus than Jenkins does, they also have newer code
and more speed, each has a niche but Jenkins has the market share, it will be
an interesting match.

Jenkins is fighting a bit of a uphill battle but with a huge army.

I hope they keep it simple, focus in being the best in one or 2 things and
then scale to other areas, that is my 2 cents.

~~~
twic
First i've heard of Argo CI. Why would i want that rather than Concourse?
Which can now be deployed on Kubernetes, it seems:

[https://github.com/helm/charts/tree/master/stable/concourse](https://github.com/helm/charts/tree/master/stable/concourse)

~~~
jacques_chester
Concourse's _components_ (ATC, DB, Workers) can run on Kubernetes, but it is
still handling the scheduling of containers it creates.

Delegating container scheduling to Kubernetes is the next major epic on the
Core track for Concourse.

As for Argo: I am not particularly in favour of Turing-complete YAML.

Disclosure: I really like Concourse. I work for Pivotal, which sponsors
Concourse development.

~~~
cfontes
Valid point.

Don't fly you end up creating a turing-complete config file as well? I
remember it being stored as some kind of Yaml like syntax.

~~~
jacques_chester
Fly does substitutions, but that's it: no loops, no if-thens. It's almost
certain that something can be tickled into being Turing-complete; it's
actually devilishly difficult to avoid doing it by accident. My hunch is
pipelines are Turing complete, but I haven't gotten around to proving it.

But there's a difference between trying not to introduce it and going out of
your way to build a programming language in YAML. An actual, honest-to-god
programmable YAML.

------
Aissen
I'm surprised they have such a good understanding of Jenkins' shortcomings.
It's a good first step in fixing them. Although to be fair, this has been
coming a long time as the post says; but having Cloudbees' CTO publicly
acknowledging those is even better.

~~~
oblio
He's not only Cloudbees CTO, he's the original developer and architect.

And before someone lambasts him for the Jenkins architecture, Jenkins/Hudson
was created in 2005, when things were a lot different, and Jenkins managed to
create an entire subgenre of software and lead it to the current day. Jenkins
hasn't aged gracefully but how many software products from any category have
even survived 5 or 10 years? :)

~~~
mooreds
I can think of a number (Linux, Eclipse, vim, mysql, etc), but as a percentage
of total software produced, it's very small.

------
lima
OpenShift's Jenkins <-> Kubernetes integration plugin is pretty neat.

Authentication, SSH secrets and - most importantly - running each build in an
ephemeral pod works out of the box.

[https://docs.openshift.com/container-
platform/3.9/dev_guide/...](https://docs.openshift.com/container-
platform/3.9/dev_guide/dev_tutorials/openshift_pipeline.html)

~~~
jstrachan
OpenShift's Jenkins integration is good.

Though its even cooler to use Jenkins X on OpenShift as you get automated
CI/CD pipelines + Environments, Preview Environments on Pull Requests and
GitOps based Promotion between environments.

------
curtis
The biggest deficiency I found in Jenkins is that GUI-based job configuration
is great for simple setups and one-off jobs, but the moment you throw in any
sort of parameterization it becomes a real headache. At that point you really
need to be able to configure your jobs in code.

~~~
dharmab
And the Jenkinsfile documentation is relatively bare and reliant on examples.

------
tannhaeuser
I'm wishing Jenkins all the best. I know it since the Hudson times as _the_
de-facto CI system for Java (and Cruise Control before that as my first
encounter with CI).

OT: does anybody know a CI system based on plain Makefiles, convention-over-
configured for autotools-like default targets, and supporting file-suffix
based build and test rules for C + JS + custom compilers and such?

~~~
jstrachan
invoke `make` from inside your `Jenkinfile`? :)

------
mbubb
I love working with Jenkins - I know it is a pain to keep up to date but for
me it has become a way as a sole or small team syseng to manage all kinds of
stuff. "Jenkins-Ansible-Github" where you have a Jenkinsfile sitting in the
git repo you are bulding/ deploying etc., has been a pretty good set of tools
to manage heterogeneous environments.

~~~
oblio
Agreed. Now if only they could generalize Configuration as Code:
[https://github.com/jenkinsci/configuration-as-code-
plugin](https://github.com/jenkinsci/configuration-as-code-plugin). It's the
missing piece.

~~~
csanchez
That's definitely the goal, part of Kohsuke announcement.

------
Thaxll
I'm trying Gitlab atm, it's great to see something simpler than Jenkins to do
CI/CD.

~~~
dsumenkovic
Glad to hear that. We'd love to hear your feedback about GitLab CI/CD.

~~~
Rowern
I've been working with gitlab CI for the last year. Here are some of my
feedbacks:

\- 6 months ago we seriously considered moving away because it was really
unstable (even when running on private runners) but now its a lot smoother

\- with private runners you can have a very powerful CI without having to
manage a master (as Jenkins) for a fraction of the costs (runner with docker-
machine on spot instances)

\- beware that if your CI flow is more complex than just a simple pipeline to
build and deploy your project (we have a project for our code, that then
trigger a project for end-to-end tests, that then trigger a deploy to our env)
you will need to do a lot of boilerplate code (you will need to manually
manage artifacts if they need to be shared between jobs)

\- variables from a triggered pipeline should be available through the API and
made more visible in the UI

\- we do not use kubernetes so eveything CD is off the plate for us
(environment and monitoring tab are useless)

\- DO NOT USE THE BUILT IN CACHE, it's super slow and will fail unexpectedly
(simply do cp to s3 and it will never fail)

\- IF YOU USE THE BUILT IN CACHE, parallelism will be hard (you cannot
populate part of the cache from a job, another part from another job and in
the next step use the result of both cache)

\- triggers are weird, its a curl to an API endpoint but it does not use the
normal auth mechanism and it will answer with a useless json (please add the
project id, variables etc to the result of the trigger it's a must have for
anyone that needs to parse the output)

\- the gitlab API is top notch except on the CI part...

\- be ready to restart some jobs 2-3 times if gitlab is deploying a new
version ;)

\- be ready to have some random errors that can be fixed by a retry

\- it will seem a good idea to run gitlab-runner on every laptop of your team
to reduce cost. DO NOT DO THAT, if you are more than 2 in your team the guy in
charge of making the CI run (me) will make you restart you docker, delete a
specific image, restart gitlab-runner, etc... invest 1 day to setup the docker
machine on spot

\- please show in some way when a job triggered another one (maybe a section
in the YML, or even better check make us populate an env var with a link to
the triggered pipeline or anything)

\- design your pipeline so that if a part fails you can restart it without
breaking everything (I'm looking at you terraform)

This list seem really long but, I have worked with Jenkins and even if more
stable the steady improvements and addition to gitlab CI still make it my
first choice for my needs.

~~~
orf
> it will seem a good idea to run gitlab-runner on every laptop of your team
> to reduce cost.

Will it?!

~~~
artursapek
Agreed, that's a crazy way to try to reduce cost

~~~
crazysim
Reminds me of the Xcode built-in distcc thing they had back then.

------
CamouflagedKiwi
I'm impressed about how honest the author is about the shortcomings of Jenkins
as it is now. Very appropriate that he mentions being in a local optimum -
that is where most organisations end up with Jenkins. The server nearly
immediately becomes a snowflake, most stuff is configured through the GUI
rather than code, probably some people know it's not ideal but getting to
something better requires changing everything and people know how it works
now.

Having said that, I think the conclusion is wrong. The next-generation CI
already exists (CircleCI, Gitlab, etc), attempting to evolve Jenkins into that
seems like a punishing task given the huge legacy and relatively little
strategic advantage. Don't want to take anything away from them blazing the
trail, but in the same way RCS and CVS did that and eventually bowed out of
the game. Jenkins should gracefully do the same.

~~~
kohsuke
Thanks for your thought. I took your main question to be "why bother?"

I think a part of it is that I fundamentally believe in an extensible system.
The world of software development is so diverse, and we have smart people
everywhere. So I always felt that the best thing a geek like me can do to
other geeks is to give them a shoulder to build on. I don't think that's a
solved problem, and to me, that'll always be an important value of the Jenkins
project, more so than any code.

I think a part of it is the responsibility to users. Jenkins is very widely
used software, and it's an incredibly important part of the software
development process for many. I appreciate that kind of trust, and I want to
deliver better software for them. I think people in the community shares the
same passion.

As CTO of CloudBees, serving our users and customers, and broadening the
adoption base are an obviously important business goal. So the interests are
aligned there as well.

And finally, I think this kind of "reinvention of the brand" happens all the
time. Windows got reinvented from 95 to NT, Firefox got reinvented a few
times. There are many other examples less famous but closer to my part of the
universe, like Maven 2, GlassFish 3, ...

------
honkycat
The worst part of my job is configuring our Jenkins server and managing builds
in their dumbass groovy based DSL.

I'm willing to bet that most people just want to build GitHub repos. Then why
do we have to do this mess to get a decently repeatable deployment strategy:
[https://coderanger.net/jenkins/](https://coderanger.net/jenkins/) I should
not have to crack open plugin source code in order to configure the plugin
programmatically. It's dumb and bad.

Also groovy is a bag language. Managing Jenkins pipeline library deps is a
pain.

Also yeah, plugins break constantly and upgrading them is always a nightmare.

~~~
zmmmmm
> Also groovy is a bag language.

This seems to be a repeated pattern that is really giving Groovy a bad
reputation: it keeps getting embedded as an extension point / scripting
solution inside other products. It is sold as "it's almost the same as Java,
so we don't need any documentation for it" \- and the result is that people
with little to no Groovy knowledge end up trying to use it and get incredibly
frustrated with it.

I'm curious if your conclusion above is based only on encountering it inside
other things (Gradle,Jenkins, etc) or if it's actually from analysing its
characteristics as a language more generically?

(FWIW, Groovy is probably my favorite language, but I use it as a full stack
language for application development, quite a different mode to how most other
people encounter it).

------
baylisscg
At ${DayJob} Jenkins is our default of yore. Returning to refresh a 1.x
install for one group's product we're faced with the poster child for a
Jenkins install gone bad. Looking at you Chuck Norris plugin. We can't upgrade
and we can't migrate to a fresh install due to how Jenkins handles plugins. So
we're left with a critical chunk of infrastructure that's a time bomb.

Ultimately instead of making the jump to 2.x and Jenkinsfiles we're trialing
Buildkite with great success so far and the confidence that we can jump ship
to CloudBuild, TravisCI, Concourse, CircleCI, ect should we need to.

------
bdcravens
The focus on cloud-first Jenkins is interesting considering Codebees's
acquisition of Codeship earlier this year. Obviously Kohsuke would be biased
to Jenkins, but as CTO, I'd imagine the corporate goals take precedence.

~~~
jstrachan
even from an OSS goals perspective I'm looking forward to seeing better
alignment, reuse and interoperability between Jenkins, Jenkins X & things like
CodeShip & Knative Build

------
toredash
I'm surprised that I haven't seen more of this announcement in my media
streams. I'm not a big fan of Jenkins, I find it overly complex and a Swiss
army knife.

When you can do anything, you often end up with poor implementations (IMO). If
the tools you have are restrictive but useful enough, I find it easier to
adopt my workflow for the tools instead of demanding that my complex workflow
fit into this one tool.

------
batbomb
I wish we could get encrypted credentials a lá travis in a Jenkinsfile. I’ve
found most configuration for a job can be in a git repo but you have to manage
some things through the web interface, and it’s not that easy securely
managing credentials for a Jenkins installation, even with Folders and Roles

~~~
jstrachan
BTW we use the kubernetes credentials provider plugin in Jenkins X which
exposes Kubernetes Secrets as Jenkins Credentials; then the `credentials` step
in the `Jenkinsfile` encrypts them from any build logs

------
slavik81
This is nice to see. I wish them luck on their project, because it is not
going to be easy.

------
misterbowfinger
Hate to be that person, but what's "Cloud Native" even mean? Is there a
glossary for buzzwords somewhere?

