
Can we use Jenkins for that? - brensudol
http://engineering.simondata.com/can-we-use-jenkins-for-that/
======
gkop
Eh, I'm sure there are many things Jenkins is OK at, but it's not that great
of build server.

The Multi-Configuration Project abstraction (IE build matrices) is clunky and
the plugin ecosystem doesn't respect it well (eg. the Gerrit plugin is
extremely popular but very brittle here). So you wind up with O(n) projects
anyway and still needing to copy and paste configuration among them.

Also Jenkins configuration itself is pretty nuts - settings splattered _all
over_ the web UI, backed by XML - compared to the simplicity of modern tools
like Travis (which uses YAML).

And Jenkins' UI I would definitely categorize as typically-poor open source
UI, having evolved and grown more complex over many years with no strong
guiding vision.

~~~
JulianWasTaken
I agree with all this. Unfortunately I haven't found anything less bad :(

~~~
VoiceOfWisdom
Jenkins + the Job DSL Plugin [0] has been amazing. We don't do any
configuration in the web interface, instead everything lives in a groovy file
on BitBucket. Every time we push a change Jenkins grabs it and auto runs the
groovy file, regenerating all of our jobs. We have it set up so that a job is
created for every branch in our repo, so much easier to work with. All a
developer has to do to get his branch building is push it to the repo.

[0] [https://wiki.jenkins-
ci.org/display/JENKINS/Job+DSL+Plugin](https://wiki.jenkins-
ci.org/display/JENKINS/Job+DSL+Plugin)

~~~
vorg
You only need recognition-memory to use Web and GUI interfaces, whereas you
need recall-memory to use command line interfaces and scripts, which is far
more difficult. That's why clicking in a Web GUI or graphical XML-generator
replaced scripting. You need a lot of previous information in your memory for
recall to write stuff like the following example taken from your linked page:

    
    
      def project = 'quidryan/aws-sdk-test'
      def branchApi = new URL("https://api.github.com/repos/${project}/branches")
      def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
      branches.each {
        def branchName = it.name
        job {
            name "${project}-${branchName}".replaceAll('/','-')
            scm {
                git("git://github.com/${project}.git", branchName)
            }
            steps {
                maven("test -Dproject.name=${project}/${branchName}")
            }
        }
      }
    

The vocabulary and grammar in that script is quite complicated and people
would rather be presented with the options for clicking on a web page. But
perhaps you're promoting a particular software product?

~~~
dripton
GUIs are more discoverable than command lines. Text files are easier to save
in version control. So it's common to have a system where the GUI hooks new
users, but they eventually have enough configuration in it that the GUI is no
longer a reasonable way to control it all, and they need a way to extract the
information in a reasonable format. Hence the reason several Jenkins plugins
have evolved to do that.

------
sheraz
Cool ideas here even though a lot of comments are pouring water on it.

Reminds me of this post by Ted dzubia where he uses makefiles for data
processing [1].

I like reading about novel uses of tools other than their original intent.

1- [http://widgetsandshit.com/teddziuba/2011/02/stupid-unix-
tric...](http://widgetsandshit.com/teddziuba/2011/02/stupid-unix-tricks-
workflow-control-with-gnu-make.html)

~~~
jasondavis
Thanks sheraz - we certainly don't expect Jenkins to serve as core
infrastructure for many of these problems long term.

------
suprgeek
The flexibility of Jenkins continually impresses me. Given :

1) Simple UI (for simple usecases) 2) Easy setup on single node or multi-node
scenarios 3)Automation capabilities

Unfortunately there are some gaps that make it just enough of a pain to really
take up in a Production 24x7 env.

1) In a distributed setup, there are very minimal node management capabilities
unless you manually integrate with say Zookeeper or something.

2) the plugins for backing up and restoring configurations are "lacking" to
put it politely

3) Very hard to change the Master machine in a master slave set-up

4)etc etc

So while Jenkins is a like the Swiss army knife for CI be careful that you
don't take it to a (multi-node production) Gunfight - to stretch the analogy.

------
forgottenpass
This is actually my problem with Jenkins. It's great for running arbitrary
jobs when triggered to do so. But it's super frustrating for managing many
permutations of of the same class of jobs, with component tasks, permutations
of different input, and orderly storage/display of output categorized on
input. Or in other words: it sucks at building and testing software.

I'm almost to the point of using it just to manage generic "do X sort of
stuff" tasks across many nodes, but the jobs that rely on an external system
to run parameterized builds, and then the job then stores the results into the
external system.

------
lmm
This is part of the problem I have with Jenkins. It does far too many things.
In many organizations it seems to occupy an awkward position between dev and
production, where it's used for both and so ends up as the most fragile part
of a production pipeline. Its config is very awkward, and partly as a result
of that I've _never_ seen it properly backed up or even a staging instance for
testing changes, yet alone blue/green instances.

So I prefer to have clear segregation. Jenkins as a build tool only. Rundeck
for deployment. If I had a big need for scheduling, I'd want a dedicated
system for doing that too.

------
thebeardisred
While not the best answer, I've used Jenkins extensively in the past for some
rather creative purposes. Besides the standard "watch a SCM repo and react"
type functions I've used it in combination with libvirt to spawn VMs self
service for users (as well as coordinate the accounting and access control
around this), watch an etag for a file to change in object storage and react,
etc. I agree with some of the other commentary in that it's most useful
aspects are around _not_ using it with a traditional CI pipeline.

------
herge
We use jenkins for CI, but only on our QA server. I haven't found a way to do
CI without giving Jenkins full credentials on our server.

Jenkins is very powerful, but I would not trust it (or any of the myriad
plugins we have installed) to not have security holes.

~~~
ktt
Yep. Not long ago a remote code execution hole in Java was found. Fortunately
Jenkins was quickly patched: [https://jenkins-ci.org/content/mitigating-
unauthenticated-re...](https://jenkins-ci.org/content/mitigating-
unauthenticated-remote-code-execution-0-day-jenkins-cli)

------
rrdharan
At my previous employer I used Jenkins with build matrices quite heavily.
Lately I've been using TeamCity and we're paying for the enterprise variant.
TeamCity has its share of flaws but FWIW it seems to handle more complex
workflow tasks better.

In particular I like the flexibility around snapshot versus artifact
dependencies, the APIs are decent (and you can do a lot of troublingly clever
things if you invoke the API from within a build), and the metarunner concept
seems strictly more powerful than the Jenkins equivalents, albeit with a
somewhat steeper learning curve.

~~~
codereflection
We've been using TeamCity for many years, and recently evaluated Jenkins as a
possible replacement. Stability was a huge concern and decided to stick with
TC. I agree, TC does have its share of flaws, but it's way ahead of Jenkins in
many areas.

------
anthonybsd
I would be extremely wary of doing something like this considering that
Jenkins stability is not stellar and bugs are aplenty. The codebase is fairly
old and is littered with outdated code, failed refactorings etc. In the course
of daily operation Jenkins is extremely susceptible to race conditions,
potential memory leaks, and that is just core - without even running any
plugins. Entrusting your company's critical infrastructure to it seems to me a
bit dangerous.

------
chizzl
Does anyone consider Jenkins as a replacement to CRON?

~~~
pilom
We do at my work. It has much more robust options. You can keep multiple jobs
from running at the same time if one takes too long. You can auto kill it
after a time limit. You get a web UI for all logging from the job. You get
great built in tools for auto deleting logs after a certain number of builds
or time period. You get a very robust email solution for sending out alerts
when jobs run or don't run or fail. It adds some complexity vs CRON but it has
many more features.

~~~
feld
it uses a huge amount of resources compared to cron, though, which is my
biggest problem with this approach.

------
stuff4ben
The question appears to be "how can we abuse Jenkins today"? For small teams
and orgs, Jenkins is fine, but scales horribly.

~~~
falsedan
Totally agree. When you have critical systems driven from Jenkins jobs, and
they're called all hours of the day, how do you manage to upgrade Jenkins
without trashing all the running jobs? (A: pay CloudBees for the Multi-Master
plugin).

Jenkins skimps on so many basic problems (cleaning workspaces for deleted
jobs, resource requirements for jobs, API which can handle 10 requests a
second, config store that doesn't require parsing hundreds of XML files on
startup). Plus it has its own git implementation, which has unconfigurable
checkout behavior (i.e. fetch all branches then checkout), which makes it a
huge disk hog on busy repos.

The plugin system is half-baked: we have to check every new plugin (& version)
on a test instance to verify that it doesn't interact horribly with any
existing plugin.

When Jenkins triggers a downstream job & blocks on it before continuing, it
logs the job it triggered, but not the build (until after the job completes).
This makes watching a build in progress extremely tedious, and you have to
hunt through the builds of the triggered job to find the relevant build. None
of this is exposed directly by the API, your client has to derive it itself.

------
muyfine
This article feels like it's from years ago. Using Jenkins today is an
anachronism - poor UI, poor configuration, poor scalability, poor
distribution. My current employer has poured so much well intentioned effort
into Jenkins' black hole. Never again.

~~~
Lewisham
What are you going to use instead?

~~~
muyfine
Migrating to Travis and a SWF-based solution, all driven by versioned
configuration and code.

~~~
fierarul
From a quick look Travis doesn't seem to be open-source or even usable on-
premise.

~~~
muyfine
Open source is not a requirement. Neither is the ability to run locally, which
is something we'd like to very much move away from.

------
pilom
In addition to CI, we use it for all of our Selenium testing and for jobs that
would otherwise just be cron jobs. The UI for logging and built in email
alerting when jobs fail are invaluable.

------
falcolas
I hope your Jenkins server is well protected, if you're running so many things
through it. One [JVM] to rule them all.

