
Untangling Jenkins - mooreds
https://www.conjur.org/blog/2017/12/19/untangling-jenkins.html
======
oblio
I've been a build/release engineer for quite a few years now, most spent using
Jenkins.

It can be quite a heavyweight beast, especially since it's written in Java (so
heavy in terms of resource usage) and since it's old (so lots of legacy in
terms of architecture).

Still, in terms of costs and scalability (up to the level where you have
thousands of developers and you probably want to roll your own), I think it is
one of the best tools out there for its role as a prettier cron/queuing
system.

A few tips:

* if you can afford it, stop ASAP using the UI for configurating jobs; look into Pipelines or the JobDSL, based on your needs

* once you have more than a few jobs, stop running anything on the master node, for increased stability

* use the minimum number of plugins and tie as few things to Jenkins as possible; script as many things and put the scripts into version control

* backup the global configuration; you can try to version control it but it's tricky; at least back it up periodically and most of the time it is enough

That's it for now, for more details, $200/h :p

~~~
grogenaut
Going off of this I wouldn't even use job dsl or pipelines as they allow you
to execute arbitrary groovy. I would consider emulating Travis ci or Amazon
pipelines where it is all declarative not imperitive. This way you can re-
impliment and consolidate without doing a Turing complete parse of tons of
custom groovy scripts

~~~
oblio
There's a Declarative Pipeline these days plus you can turn on the Groovy
sandbox. There's also a YAML based thing but I forgot its name.

~~~
grogenaut
Can you provide links? I'm on a plane so very hard to Google.

Seriously? Downvotes? On a holiday on a plane and I get down votes asking for
a little help? I did try and google and didn't find the stuff.

~~~
RyJones
We use JJB, which is YAML based.

[https://github.com/hyperledger/ci-
management/tree/master/jjb](https://github.com/hyperledger/ci-
management/tree/master/jjb)

for instance

~~~
harlowja
Just fyi :)

Docs:

[https://docs.openstack.org/infra/jenkins-job-
builder/](https://docs.openstack.org/infra/jenkins-job-builder/)

Reviews:

[https://review.openstack.org/#/q/project:openstack-
infra/jen...](https://review.openstack.org/#/q/project:openstack-
infra/jenkins-job-builder)

Code:

[https://git.openstack.org/cgit/openstack-infra/jenkins-
job-b...](https://git.openstack.org/cgit/openstack-infra/jenkins-job-builder)

------
chrisanthropic
The best Jenkins advice I ever received was from here:
[https://medium.com/mindera/jenkins-a-la-
travis-6c5a8debbb5b](https://medium.com/mindera/jenkins-a-la-
travis-6c5a8debbb5b)

On setting up Jenkins Job Builder using a 'seed job'. This setup allows you to
add a .jenkins directory to you project, define a yaml based jenkins job
inside that directory, push to github, and Jenkins will create the job you
defined - NO web UI required.

In other words you make Jenkins behave like Travis.

Set this up with your first job being a job to backup Jenkins to S3 and things
aren't as bad to manage.

~~~
finspin
I'm currently looking into how JJB could be used for simple scheduled jobs.
But isn't the setup described in the article basically the same what you can
do with Jenkinsfile?

------
correlation
I have come up with a slogan for Jenkins: "Jenkins - The butler that does not
hesitate to do a disservice".

It is currently the most brittle piece of infrastructure we run, with jobs
spiralling out of control and slaves crashing with cryptic exceptions. Add to
that a horrible plugin ecosystem (tried to install the Slack plugin, it
crashed all builds and our webhook endpoint). Add to that groovy (I don't feel
like becoming an archeologist).

Gosh, can't someone just build a lightweight ci/cd system ontop of e.g. k8s
jobs?

Or should we just give up and do managed ci/cd?

~~~
CodeAlong
My workplace has primarily switched all net new ci/cd workflow from Jenkins to
Drone ([https://github.com/drone/drone](https://github.com/drone/drone) \-
Drone is a Continuous Delivery system built on container technology.). Having
managed masters with 20,000 + configured jobs I agree that Jenkins is a very
sore spot for our entire pipeline. We’ve had greater success with increasing
the number of masters (one per team or application) and using the kubernetes
plugin for slaves. However, most of our pipeline complaints have been
mitigated by getting teams to migrate from Jenkins to Drone.

While I don’t think Drone currently supports k8s job, it is very lightweight
in comparison to Jenkins. Most of our developers would agree that it isn’t a
perfect product, but it has reduced friction points by a large amount.

~~~
SOLAR_FIELDS
Interesting war story. We have a couple of master instances that go down all
the time because of several thousand jobs running. Gonna ask our devops what
they are cooking up to resolve and compare to this solution.

------
jonthepirate
I am spearheading an effort to release a CloudFormation template that will
launch Jenkins clusters into your AWS account. It represents everything
learned while working with Jenkins for over 6 years including 2.5 years of
Jenkins at Lyft and 1 year at DoorDash. If you would like to collaborate,
please contact block.jon@gmail.com. The project was written with an eye
towards making the launching of a Jenkins cluster very very easy. It's
designed with Docker workloads in mind. Some of the gems include autoscaling,
instrumentation & metrics, automatic Let's Encrypt certificate registration,
GitHub webhook integration and retries during pipeline slave disconnects which
makes spot instances viable even when they abruptly go away... something that
saves you 75%!! on your AWS instance costs.

~~~
mooreds
I don't have cycles to help, but please post this on HN when you are ready!

~~~
jonthepirate
Here's a preview of what I'm working on that I just published to YouTube:

[https://www.youtube.com/watch?v=GkflqV6KenM](https://www.youtube.com/watch?v=GkflqV6KenM)

I'd love to hear more about what problems any of you are facing with Jenkins
so I can be sure to address them in the work I'm doing.

Jon

------
Sir_Cmpwn
If I may suggest my own alternative to Jenkins: builds.sr.ht. It runs build
manifests submitted through an API and does not support job configuration
itself; you must use an external integration to automate your builds. Build
manifests look something like this:

[https://git.sr.ht/~sircmpwn/wlroots/tree/.build.yml](https://git.sr.ht/~sircmpwn/wlroots/tree/.build.yml)

More complex example:

[https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/.build.yml](https://git.sr.ht/~sircmpwn/builds.sr.ht/tree/.build.yml)

Builds run with full virtualization (KVM) on any number of distributed
(optional trust) build runners. The whole thing is open source:

[https://git.sr.ht/~sircmpwn/builds.sr.ht](https://git.sr.ht/~sircmpwn/builds.sr.ht)

It's also self-hosting, it builds and deploys itself and the rest of the
*.sr.ht network:

[https://builds.sr.ht/job/602](https://builds.sr.ht/job/602)

If anyone wants to give it a try, please get in touch (email in my profile)
and I'll hook you up with an account on my hosted instance. I could use more
people to kick the wheels and help me find the pain points.

------
BlanketApple
Man, I'm having this problem right now at work and it's been a nightmare.
Admittedly the problem is like 90% cultural.

People (on my team) don't understand pipeline and they absolutely don't
understand Groovy. Before, they would just write a job with a shell script
that e.g. ran valgrind, etc. Lots of small repos with similar steps, so they'd
write one job and apply it to 20 different repos. It worked pretty well.

That's still an option with pipelines, but it feels much more discouraged (to
e.g. write a shared pipeline job that runs a multi-line shell script).

It's also just a bit of an organizational nightmare to see 200+ jobs on the
main screen (as opposed to 20 jobs that did the same thing for 200 repos).

Unfortunately, it also seems like declarative pipeline is limited enough that
we end up writing a lot of Groovy. I sorta get why Groovy became the official
scripting language, but it's like pulling teeth getting people to learn even
basic Groovy.

I realize, again, that a lot of this is a cultural problem. Mostly just trying
to give a counterpoint that pipelines don't always look as clean as the
article implies, especially for orgs with a lot of repos (where committing the
same 'test.sh' file to each repo doesn't make a lot of sense).

~~~
Someone
_”It 's also just a bit of an organizational nightmare to see 200+ jobs on the
main screen”_

I never look at the main screen. You can create tabs that segment those 200
jobs. If your job names are consistent (as they should), you can do the
segmenting by regular expression (I’m not sure whether creating tabs requires
a plug-in)

As to doing (almost) the same thing for different projects: you can POST XML
job definitions to the web interface, so if you have the rights to create jobs
and to run code on your local system, you can script job creation. In my
experience, that’s the way to keep job definitions consistent. To figure out
what the XML to POST should look like, grab it from your browser by GETting
job/ _jobname_ /config.xml ([https://support.cloudbees.com/hc/en-
us/articles/218353308-Ho...](https://support.cloudbees.com/hc/en-
us/articles/218353308-How-to-update-job-config-files-using-the-REST-API-and-
cURL-))

Yes, that duplicates lots of stuff in the Jenkins job definitions, but you
shouldn’t treat that as source, but as the output of your job creation
scripts.

~~~
tokenizerrr
If you get to that point, check out the Job DSL plugin.

~~~
Someone
I would have to convince the system administrator to install that plug-in.

I also don’t like having to use yet another language (or two, counting Groovy
and the Job DSL separately), but that, I could live with.

------
nunez
Jenkins and any other CI/CD tool is a prime candidate for immutable
infrastructure; it’s the one thing that nobody cares about too much until it
no longer works, and it is relatively easy to get going with scripts or Docker

------
emmelaich
Has anyone here used BlueOcean and have anything to say about it good or bad?

[https://jenkins.io/projects/blueocean/](https://jenkins.io/projects/blueocean/)

~~~
tokenizerrr
Meh. The interface for viewing logs lags my browser.

~~~
emmelaich
It is beta-ish still, so that may go away. I was more looking for comments
about the features, pipeline etc.

Also, regarding logs the site says this:

 _" Diagnose problems instantly and say good-riddance to endlessly scanning
through logs."_

Won't believe till I try it but it's a good goal!

