
Jenkins Is Getting Old - zdw
https://itnext.io/jenkins-is-getting-old-2c98b3422f79
======
hardwaresofton
Disclaimer: I'm pretty biased towards Gitlab -- write about the things you can
do it from time to time and they gave me some free swag once.

Best CI I've ever used is Gitlab CI[0]. The runner is completely open
source[1] and you can use your own runner with your gitlab.com (or local
instance) projects -- set it up in an autoscaling group[2] for savings.

I run [https://runnerrental.club](https://runnerrental.club) but Gitlab also
recently released the ability to pay for minutes in 11.8 [3], so my product is
more-or-less dead in the water but I don't mind since Gitlab is such an
excellent tool, I'm glad to see them fill the need.

But back to Gitlab CI -- the YAML configuration documentation[4] is pretty
fantastic -- Most easy things are easy and hard things are possible. I suspect
that one could run an entire startup like circleci/travis/drone based on
_just_ the software that Gitlab has open sourced and made available already.

[0]: [https://docs.gitlab.com/ee/ci/](https://docs.gitlab.com/ee/ci/)

[1]: [https://gitlab.com/gitlab-org/gitlab-runner](https://gitlab.com/gitlab-
org/gitlab-runner)

[2]:
[https://docs.gitlab.com/runner/configuration/runner_autoscal...](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws)

[3]:
[https://about.gitlab.com/2019/04/22/gitlab-11-10-released/#p...](https://about.gitlab.com/2019/04/22/gitlab-11-10-released/#purchase-
add-on-ci-runner-minutes)

[4]:
[https://docs.gitlab.com/ee/ci/yaml/](https://docs.gitlab.com/ee/ci/yaml/)

~~~
chupasaurus
Multirepo pipelines behind a paywall is a good reason to say "No, thanks".

~~~
majkinetor
TBH, its easily done in shell via REST API. We use custom pipeline runner from
simple Powershell scripts that works even better for us then default style.

We set all jobs to manual and then our script triggers them depending on
commit message, person, moon phase etc.

But really, this should be in core. Its very hard to do multirepository stuff.
Its not that easy to do mono repo stuff too - I really need a pipeline withint
sub-project, a mother pipeline, option to run whatever one I want etc... Gitlb
pipelines could be a lot better.

~~~
hardwaresofton
I find it very easy to make multi-repo stuff, I've also wrote about it:
[https://vadosware.io/post/fun-with-gitlab-ci](https://vadosware.io/post/fun-
with-gitlab-ci)

Also, running pipelines in different projects and whenever you want is
accessible from the web interface... It could be cleaner, but IMO adding even
more syntax to the YAML file can also be a rabbit hole -- CURL from the script
section seems like not a bad middle ground.

Am I misunderstanding what is being referenced?

~~~
Robin_f
This post seems very focused on projects with just a single developer. The
"recognizing and tagging" versions for example completely breaks down when you
have multiple people commit and merging things to master and multiple
pipelines running that are trying to recognize and tag versions.

~~~
hardwaresofton
The post is very much not complete -- there is much more a team would have to
standardize to fully take a set up like this into production, bu I'm not sure
that it "completely breaks down" with more people contributing.

As long as there is _some_ release coordination, a system like this can work.
In particular, I've found that the lightest way to get a system like this
working for x > 1 developers is to have a release-vX.X.X branch for releases
that are going out and vX.X.X tags for releases once they have actually
landed.

You mention the recognizing and tagging versions being an issue -- are you
imagining a world where two people are releasing something at the same time?
I'm not exactly sure

If we assume that what that posts suggests _isn 't_ feasible, regardless of
how you deploy and do your CI, if your release process can be done by a human,
it can likely be automated. Gitlab CI is most robust and yet easiest to
understand automated CI system I've seen for making that happen (at least that
was my intended point).

------
jennbriden
Hello, I see a lot of great feedback in this post. I am a product manager
working at CloudBees, the primary corporate sponsor of Jenkins. Jenkins is now
in the Continuous Delivery Foundation as well.

While it is easy to bash on an inanimate object, there are some very dedicated
and empathetic people who care deeply about the project. Some of those people
do this work in their off-hours and some to this work as part of their daily
work activities AND also in their off hours.

In that spirit, we want to make Jenkins better and created a separate group at
CloudBees in the Product and Engineering teams late last year. They focus on
open source work for Jenkins and on some proprietary things for CloudBees Core
(built on Jenkins). We also have a dedicated user experience/product designer
who started working on the project a few months ago. One of the first things
he and I worked on was creating a curated, tailored version of Jenkins via the
CloudBees Jenkins Distribution. This distribution will focus more and more
over time on a guided workflow for continuous integration and delivery with
Jenkins. These patterns will also be shared with open source Jenkins - some
through direct contributions and others through suggestions (better plugin
categorization, documentation, etc.).

Please use this comment thread to share your constructive, honest feedback
about how we can improve Jenkins.

~~~
mechastorm
I think its already mentioned about Jenkins Configuration as Code. But in
general stick to a configuration method, I have had to migrate from JJB >
groovy scripts > init.d groovy > Jenkinsfile.

And can we PLEASE open up the issues tab on the Github repos especially for
plugins. There is currently no way to provide any feedback/report issues on
these plugins because the "issues" tab is disabled. Our current options are \-
Put a comment on the plugin wiki page \- Hunt down the the relevant support
forum for the plugin \- Find the original source repo and post an issue

All of which are not ideal and does nothing to directly help the development
of the plugins.

~~~
owenmehegan
Jenkins has used Jira for bug tracking since before GitHub existed, so
although the project's code and all plugins are now managed on GH, bug
tracking continues to live in Jira. Anyone can create an account and add
issues here: [https://issues.jenkins-
ci.org/secure/Dashboard.jspa](https://issues.jenkins-
ci.org/secure/Dashboard.jspa) All plugins track issues there, so you just put
the appropriate plugin, or "core" for Jenkins itself, in the 'component' field
and the issue should be assigned to the right person.

------
FrenchyJiby
I sincerely wish I could move away from Jenkins for the reasons stated in TFA
(GUI-oriented, slow, hard to backup/config, test-in-production mentality and
boundless plugins) but I've never found something that fits the bill.

The much-touted repo integrations (travis, circle...) all have an exclusive
focus on build-test-deploy CI of single repos.

But when you have many similar repos (modules) with similar build steps you
want to manage, and want to have a couple of pipelines around those, and
manage the odd Windows build target, these just give up (it's docker or bust).
Sadly, only Jenkins is generic enough, much as it pains me to admit.

Anyone got a sane alternative to jenkins for us poor souls?

~~~
user5994461
TeamCity from JetBrains is the same thing as jenkins, except the core features
are working core features instead of broken plugins. It's paid software
though, you get what you pay for.
[https://www.jetbrains.com/teamcity/](https://www.jetbrains.com/teamcity/)

On the other hand there is Bamboo from Atlassian.
[https://www.atlassian.com/software/bamboo](https://www.atlassian.com/software/bamboo)

I really don't understand this mentality of there is no better tools when
there are better tools than jenkins and they've been around for a while.

~~~
orlovs
Fine, when you are one specific vendor shop, like Jetbrains or Atlassian stack
and you have got plenty of financial power, then there is always cool
features, what can bring benefit. But in the end CI and CD systems are
glorious semi-smart cron runners. Are these tools 10x better than Jenkins. Not
so much, CI/CD is from one of the standpoint most important and in the same
time less important tool, delivery should suck very much to migrate to new
platform just because. Jenkins shines here, it's not perfect, but it works.
More or less it's for free from licensing standpoint, you don't have to go
thru Corporate procurement hell. It's not for free from workforce perspective,
but none of these tools are with zero configuration. Just x,y,z, still some
yaml or some other crazy configuration needs to be done (like Bamboo dsl).

~~~
user5994461
Of course it can be 10 times better. It's so trivial to be 10 times better.

First you checkout the project from the repo and it just works, doesn't matter
GIT, SVN or whatever. How many plugins does it take to checkout a project in
jenkins? Is there even a git plugin working nowadays?

Then, you build the project. If it's any of C# or Java for example, the
ant/maven/sln/nuget files are detected automatically, just click next and it's
built. Does jenkins even understand what is a requirements.txt? Hint: It's not
a bash script.

The JVM and the Visual Studio are detected automatically on all the build
slaves and the project is already building in the right places. If you want to
aim for specific tool versions, there are presets variables on all hosts to
filter where to build and to use in build scripts so paths are always right.
How is the build matrix plugin in Jenkins lately? Broken as usual?

The project is built, or is it building? It's easy to tell because there is a
clear colored status icon and the estimated time to completion is displayed.
Teamcity offers that out of the box for maybe 15 years now. Well, jenkins
finally got a progress bar too a couple years ago. I guess I'm defeated,
Jenkins caught up on basic core functionality only a decade late, I can't
justify to pay for working and polished tools anymore. Well, I hope our
sysadmin will install the Extra Status Icon Plugin or we'll have to live
without the big colored circles next to the build.

~~~
blattimwind
TeamCity is indeed quite good.

------
solatic
OP really needs to try Concourse. Same container-based workflow as Drone that
is touted as a solution, but more mature, and much more testable than Drone.

Concourse really hits his requirements for ops-friendliness and testability.
It's easy to upgrade because the web host and CI workers are completely
stateless, and the work you do with Concourse is easy to test because the jobs
themselves are all completely self-contained. Because Concourse forces you to
factor your builds into inputs, tasks, and outputs, it becomes somewhat
straightforward to test your outputs by replacing them with mock outputs.

The main issue with Concourse is that it has a huge learning curve to use
effectively. When you have many pipelines, you end up learning the
undocumented meta pipeline pattern to help manage them. You end up
implementing stuff like mock outputs by yourself, since it's not really a
first-class concept. Concepts like per-branch versioning that have been in
products like Jenkins for years are only now entering development as "spaces".
All of the configuration can be version controlled, but it's all YAML, so to
wrangle everything together, you end up using something which will compile
down to YAML instead. RBAC much improved in v5 but still leaves much to be
desired. There are no manual steps, only manual jobs. Job triggering is
relatively slow due to a fundamental design tradeoff where webhooks trigger
resource checks which trigger jobs instead of triggering jobs directly, to
make resources easier to maintain. Nobody really tells you this on the front
page.

It has its flaws. But if you delve into it you see very quickly that it's an
incredibly solid product. The ability to run one-off jobs from developer
workstations on CI workers, and to easily SSH into those CI worker containers
from developer workstations, is incredibly helpful for debugging issues.
Because everything is stateless, everything tends to "just work". If you go in
with your eyes open about its limitations, you'll have a good time.

~~~
jniedrauer
The thing that turned me off concourse last time I checked it out is that
their documentation assumes (assumed?) you're going to use BOSH. I don't want
to have to learn and maintain yet another infrastructure as code tool, just
for my build server. I know you can run concourse without it, but all their
examples seemed to use it and I didn't want to hit edge cases that they didn't
account for. So I gave up before too long.

~~~
_whiteCaps_
There's a helm chart available for Concourse, I haven't tried it yet though.
Definitely agree about BOSH making it harder to get started.

[https://github.com/helm/charts/tree/master/stable/concourse](https://github.com/helm/charts/tree/master/stable/concourse)

------
asgeirn
Our setup runs Jenkins master and slaves as Kubernetes pods, with plugins
limited to only the very few required to get GitHub integration and slaves
working.

Jobs are configured by adding an entire GitHub organization. All repositories
with corresponding branches, pull requests and tags are automatically
discovered and built based on the existence of a Jenkinsfile.

Everything is built by slaves using Docker, either with Dockerfile or using
builder images.

Job history and artifacts are purged after a few weeks, since everything of
importance is deployed to Bintray or Docker repositories.

By keeping Jenkins constrained in this fashion, we have no performance issues.

~~~
asey
That is exactly how we're doing it as well, though I am interested in checking
out Cloudbees'Jenkins. We've recently incorporated Zalenium (selenium grid
which autoscales nicely natively in kubernetes) - just had to work a little
magic with automatic service creation during builds.

------
tinix
I receive vulnerability notifications for Jenkins, pretty much regularly...
mostly XSS and RCE.

[https://www.cvedetails.com/vulnerability-
list/vendor_id-1586...](https://www.cvedetails.com/vulnerability-
list/vendor_id-15865/product_id-34004/Jenkins-Jenkins.html)

I'm just waiting for Apache to adopt it, and then it'll sit and fester like
everything else in the Apache graveyard, full of vulnerabilities and slowly
decaying.

Those are just Jenkins core exploits too... there are so many many more for
Jenkins plugins.... [https://www.cvedetails.com/vulnerability-
list/vendor_id-1586...](https://www.cvedetails.com/vulnerability-
list/vendor_id-15865/Jenkins.html)

~~~
mfer
Jenkins is now part of the CD Foundation
([https://cd.foundation/](https://cd.foundation/)) which is one of the linux
foundation sub-foundations. Don't expect it to show up in the apache
foundation.

~~~
1ris
I don't think tinix was excpeting it to literally become a apache project - he
was just saying its in a state of decay that apache is infamous for.

------
stabbles
Jenkins's plugin manager is absolutely terrible.

If you're stuck on an older version of Jenkins, you better not click the
"refresh" button in the plugin management page, cause otherwise the page is
just filled with red warnings saying that the latest version of each plugin is
incompatible or has dependencies that are incompatible with your current
Jenkins version.

There is afaik no way to install the last plugin that was compatible with your
version of Jenkins.

~~~
buzer
> There is afaik no way to install the last plugin that was compatible with
> your version of Jenkins.

Probably not directly, but if you know the version, you can download the HPI
and install it manually. Jenkin's Docker build also contains install-
plugins.sh
([https://github.com/jenkinsci/docker/blob/7b4153f20a61d9c579b...](https://github.com/jenkinsci/docker/blob/7b4153f20a61d9c579b5f653a4a99fb543f26c41/install-
plugins.sh)) that you can use to install specific version of plugin via
command line.

~~~
stabbles
Thanks! That script definitely gives some pointers on how to update things.

------
mrferos
We use Jenkins at work and have found a pretty damned sweet spot.

Let me start by saying that we used to use GitLab, a lot of it was because of
the CI but I didn’t have a great experience trying to manage it on top of
Kubernetes, they’ve since introduced a Kube native package and I’ve been told
it’s much easier, but with the deployed omnibus we ran into a lot of issues
with runners randomly disconnecting, it became frustrating to the point where
I had developers not wanting to use GitLab and finding interesting ways to
work around it.

So I set up Jenkins on a dedicated EC2 instance with a large EBS volume for
workplace storage and installed the Jenkins plugin then I wrote a Jenkins
library package that exposes a function to read a gitlab yaml file and
generates the appropriate stages with parallel steps that execute as pods in
khbernetes - took about a week to get the things we actually used from Gitlab
CI and their YAML DSL working correctly.

Now we very happily use Jenkins, mostly through YAML but in the occasions
where things are much easier going directly to Groovy to use interface with
plugins, developers can.

~~~
mrferos
I want to specify this is my own experience and I think a lot of our own
issues may have been from mismanaging our self-deployed setup. I’ve had a lot
more experience managing Jenkins.

GitLab and their approach to CI (easy to use yaml) really facilitated
developers writing CI, which increased our software quality overall.

~~~
yebyen
Have you used the Jenkins Job DSL plugin?

I'm just getting started using it, but it seems like the solution to scaling
up to a lot of Jenkins jobs. There's a good talk about it, and since you're
one of only two people in the thread who used the word DSL and you are having
a good experience with Jenkins, I thought I'd ask.

My config is similar except my single EC2 node is actually running Kubernetes
via kubeadm, it's a single node Kubernetes cluster and has enough room for
about 3 of my worker pods to execute concurrently before the rest have to
wait.

(But that's just my setup and has nothing to do with Job DSL.)

For me, managing Jenkins via the helm chart has been the best part of the
deal, but I'm a pretty big fan of Helm already...

~~~
mrferos
I haven’t used it, we use the GitHub Org Folder scan to automatically add a
job for every Repo that has a Jenkinsfile- so I don’t have to do anything,
it’s pretty great.

~~~
yebyen
Awesome, that also sounds good

The big advantage of the job DSL plugin is that if you have many similar
repos, you don't just treat them all the same with multiple copies of the same
Jenkinsfile, you actually can build the jobs from the same source,
inheritance-style.

There could be some reasons not to do this, but if you have 100 jobs and 20 or
more of them are mostly identical in form except for some parameter, it's a
better strategy than multiple Jenkinsfile.

But then again if your Jenkinsfile isn't changing, it might not matter.

------
epage
Jenkins is terrible but I've not found something better yet.

For me, the great thing is that it is an automation platform where we can
integrate builds, scheduled tasks (cron), and very basic end-user UIs with
Plugins that let me customize the whole experience.

Azure Pipelines comes close in capabilities. If they made it easier to
manually start a job and for them to accept their parameters from the user,
then it'd probably have everything i need.

------
willcodeforfoo
There are lots of alternatives for Jenkins as a CI builder but I haven't found
many for Jenkins as a web-based cron with a nice UI that keeps track of
history, has tons of options for notifications on failures, and quickly allows
re-building (even with user-supplied parameters!)

We use Jenkins for running backup jobs, periodically updating data, and
building quick little jobs for support staff to run--infrequently enough that
they don't warrant adding to our admin app but frequently enough that
bothering at developer adds up.

~~~
cpitman
For periodic admin tasks, you might consider giving Ansible AWX a try
([https://github.com/ansible/awx](https://github.com/ansible/awx)). It is a
web interface for managing Ansible playbooks, and lets you configure jobs that
another user can then execute and provide parameters. It keeps track of all
run results, has a fair number of supported integrations for notifications,
and can schedule runs.

I work for Red Hat, and AWX is the upstream community project for Ansible
Tower, which we provide support for. AWX is one of our newer open source
projects (we open sourced it after acquiring Ansible), so you'll sometimes
have better luck searching for information on "Ansible Tower".

------
bcoughlan
I spent quite a while grappling with Jenkins pipelines, and while it
definitely has its warts, once you get over the hump and get one pipeline the
way you want it, it's quite easy to use it as a starter for other projects.

Personally I go to great lengths to minimise the plugin dependencies and push
the logic into bash scripts or Maven/Gradle. The Jenkinsfile just calls the
bash scripts, so I can run it before I change it. Where I use plugins in the
pipeline I document what they are and what they do in case we ever need to go
from scratch.

It is hard to test but the linter picks up quite a lot of errors before push,
and if you're doing a multi-branch pipeline you can just test your changes by
pushing to your own branch. It's a long cycle, but the variation should go
down the more you have them.

It has also been indispensable as an ad-hoc task scheduler. For example, I
have a job that cleans up old branch artifacts every Saturday night. It was
easy to alert on failures and see the results and history of past runs. I
don't know anything else that would have fit the bill.

When you make software over 10 years old in such a fast-changing environment,
the legacy of the software can weigh you down. I've worked on similar products
that were tied to their legacy counterparts (sharing a database), and every
change meant carrying the baggage of backwards compatibility with the legacy
platform. If I could do it all again, I'd have done it the Basecamp way: Get a
clean break from the legacy version, and have a migration path to the new
system.

I would like to see CloudBees take all they've learned over the years, take
the great work they've done on Blue Ocean and their core job scheduling system
and put it all together without carrying the history with it.

------
dvtrn
I'm not at all ashamed, nor a single bit remorseful to comment on the fact
that it took a catastrophic data loss for a team I once worked with to finally
sit down and look at our CI/CD pipeline before deciding "Maybe jenkins is
overkill for what we need".

Which was something I had been kvetching about for _months_ and expressly
warned, multiple times to our release manager as a point of concern given how
quickly plugin vulnerabilities are reported (as someone comment on elsewhere
in the thread).

The day finally came when someone from one of our other offices went to update
some infrastructure as code repos, poof. Jenkins server gone. They didn't have
a roll back plan, and to complete the trifecta, they somehow also killed all
of the instance volume backups. An entire sprint was summarily dedicated to
creating a new build pipeline, I had resumes out the door the next day.

This article hits so many of our pain points I joked to a current coworker who
followed me out of that place that I wanted to print it and mail it to our
former RM.

~~~
Iwan-Zotow
WHat do they/you use now?

~~~
dvtrn
No clue. Didn't stick around long enough to find out, that moment was
officially the last straw for me. There was a lot of hemming and hawing, I
left and product, project and dev still hadn't made their minds up. Ops (where
I lived) would give the team ideas and proposals, dev gatekept _everything_
from tickets to the VCS.

This was one of those shops that hired people and gave them devops job titles,
but demanded they maintain very monolithic status quos with everything from
tickets and stand-ups to release management.

------
js2
Another container based option is Screwdriver, which started as a Yahoo in-
house solution built on top of Jenkins. But it is now fully open source and no
longer makes use of Jenkins in any way:

[http://screwdriver.cd/](http://screwdriver.cd/)

~~~
dpedu
This looks really good! For a long time Jenkins has seemed like the only
option for flexible jobs. Tools like Drone and Travis and the many other that
use a "config file" approach just aren't able to scale against your needs.
They're fine for a run-of-the-mill build but that generally just isn't enough.

~~~
bradrydzewski
Drone is working on support for Starlark, a python-inspired configuration
language used by the Bazel Build system [1]. For complex pipelines, this
should be more analogous to Jenkins scripting and may improve scaling Drone
for larger projects (time will tell).

[1]
[https://github.com/bazelbuild/starlark](https://github.com/bazelbuild/starlark)

------
jdhzzz
I work at a big enterprise that uses Jenkins. We support a large group of
legacy applications and the waterfall type development processes that lately
have been papered over with agile-lite. I work about 80% of the time
administering (Cloudbees) Jenkins. I have lived through Freestyle projects,
migrating to Job DSL, and now we are moving to declarative pipelines in shared
libraries. I agree with 95% of the article's pain points. Still, I'm a fan,
perhaps because each iteration is significantly less painful from the previous
one.

Still Jenkins works well for us in our environment. Our waterfall style
development processes make for ops-friendly, cookie-cutter type builds: one
deployable asset per repository, one deploy target, all projects laid out the
same way... Jenkins shared libraries provide powerful ways to capture the
commonalities among the projects and provide a way for us to manage them
centrally, while providing some (small) flexibility for customizations by the
developement teams. Shared libraries also allow the ability to test changes
using selected versions (branches) of the pipeline for specific builds.
Looking forward to what's next (I still have a few mortgage payments to
make...).

------
llamataboot
Jenkins also routinely has massive security holes.

Last major exploit I heard about - the matrix.org exploit, was from privilege
escalation through a Jenkins vulnerability [1]

[1] [https://www.zdnet.com/article/matrix-hack-forces-servers-
off...](https://www.zdnet.com/article/matrix-hack-forces-servers-offline-user-
credentials-leaked/)

\--

This isn't nec a knock against Jenkins itself, but certainly a knock against
thousands of orgs running their own unpatched Jenkins servers, often on the
same machine as their other apps

------
cjbprime
It's true that Jenkins is old, and has systemic architecture problems which
mean that I'm sure we haven't seen the last RCE for it. RCEs in build systems
are a nightmare because build systems necessarily end up knowing secret
tokens.

But drone.io is no kind of replacement. It only works with Docker containers!
Even small Jenkins installs can quickly end up with Windows targets, mobile
builds, etc.

~~~
bradrydzewski
> It only works with Docker containers

This will only be true for another week or two. We have devised a framework
for alternate runtimes, including running pipelines directly on the host
machine or in traditional virtual machines, that will be released end of week.
Reference issue
[https://github.com/drone/drone/issues/2680](https://github.com/drone/drone/issues/2680)

~~~
rmoriz
Does it support running on macOS and Windows, too? CI for native apps is IMHO
the most common lock-in for Jenkins at the moment. Anything web or container
related can almost always be implemented using pretty much any other CI/CD
solution out there.

~~~
bradrydzewski
Yes, it supports windows, macos, and linux. I expect it will also support bsd
although I have not tested this yet. This should land in master some time this
week.

------
GeneticGenesis
Unrelated to the content, but the stock photos throughout this article felt
completely needless to me. We're not 5 year olds, we don't need pictures
unless they add to the content.

------
bryanlarsen
Jenkins-X is really cool. I love how you can group services together into an
environment and deploy that to a single Kubernetes namespace, and how well it
uses github releases.

I really hope Gitlab's Auto-Devops team is looking at it closely and stealing
all the great ideas it has because that's what our team is using.

~~~
kkapelon
Jenkins X requires Kubernetes and Helm. Not all companies are deploying like
this.

(I work for Codefresh, a competitor of Jenkins X)

------
bovermyer
The most interesting part of this is the very last paragraph.

There really _isn 't_ a good self-hosted solution for build metadata/metrics,
unless you write something yourself. It would be interesting to see a GUI
dashboard for Drone/whatever builds and pipelines.

~~~
qznc
I never understood the purpose of dashboards. In the context of CI, I see
three use cases and there is always a better solution than a dashboard.

1\. Detect broken builds. It is better to actively notify somebody by mail,
chat, whatever.

2\. Prevent broken master. It is better to reject the pull request/patch
before it becomes a problem.

3\. Analyse the system. It is better to download the data and enable the use
of whatever analysis tools are suitable.

~~~
jayd16
Direct emails can't replace the collaborative "hey why are these still red"
moments without spamming the whole team.

They can show non-event information such as build time trends at a glance.

------
RealSpamapS
Anyone who agrees with the OP should give Zuul-ci.org a look.

I run Zuul in production with GitHub and AWS, and it's been really useful for
us. It scales to the moon, parallelizes everything, and having _all_
job/pipeline configuration in git means you can test things like CI job
changes _before_ they're committed.

Cross-repo pre-merge dependency management means I can build whole speculative
futures in PR's _before_ it lands and breaks the build.

Funny story, the Zuul build status badge is just a static image, because Zuul
does not allow code to merge if it fails tests: [https://zuul-
ci.org/docs/zuul/user/badges.html](https://zuul-
ci.org/docs/zuul/user/badges.html)

Full disclosure: I worked on a defunct zuul based service for a little while,
and still am a core developer with lots of code in Zuul. A few presentations
I've given are here:

[http://fewbar.com/the-build-is-never-broken/](http://fewbar.com/the-build-is-
never-broken/) [http://fewbar.com/zuul-ci-crossing-
streams/](http://fewbar.com/zuul-ci-crossing-streams/)

------
deboflo
AWS CodePipeline with CodeBuild supports GitHub webhooks, export artifacts
between stages, many types of caching (S3 artifact caching, Docker caching,
file/volume caching), Secrets Manager environment variable integration,
queueing between stages, pipeline retries, stage retries, manual approval
steps, integration with CloudWatch and many of services like Lambda, and
provisioning all of the above via API, CLI, SDK, or CloudFormation.

~~~
robrtsql
> queueing between stages

Doesn't it only support a queue depth of 1 at each stage? Last I checked, if
you have a change that is queued up waiting to get into a stage, and a newer
change comes along, that newer change will supersede the old one. That makes
CodePipeline only good for workflows where you only care about deploying the
'newest' of something.

I think the worst thing about CodePipeline is how hard they make it to run
custom code from your pipeline. Your options are Lambda (limited to 15 minutes
per run, or you have to rewrite your Lambda function to be called once every
30 seconds, essentially using the first run to start your deployment and every
subsequent run to check to see if it's done yet or not) or CodePipeline Custom
Actions (where you have to write the AWS SDK code for interacting with
CodePipeline).

The AWS Developer Tools team could learn a thing or two from Azure Pipelines.
They did it "right", IMO: you can create a 'bash' stage in your pipeline which
runs whatever script you want from the build agent (which can either be hosted
my Microsoft or hosted yourself). That's all I really want. CodePipeline could
support that with a custom action but it's more stuff that I would have to set
up.

------
rjeli
I’d never used Jenkins before, a few weeks ago I needed to do some CI that was
more complicated than just pytest or whatever. Installed jenkins, put together
a dockerfile, connected the github hooks and It Just Works. I’m a happy
customer

~~~
bluGill
The problem is people want to do really complex things with Jenkins. A few
hundred repos, each with dozens of builds; Multiple operating systems... There
is a reason a team of 10 works full time to administer our CI system: it is
complex (the CI system is more than Jenkins)

------
acd
It is good to support open source software vs closed source. Gitlab is a
modern good open source alternative to Jenkins which is also open source.
Gitlab has its configuration files default in yaml files which is easy to read
for humans. Jenkins uses XML which is hard for humans to read.

One should not use close source solutions just because it is easy to use. Just
because it is easy does not mean it is good for you in the long run.

------
ethagnawl
"cripple-wear" made me cringe. Shouldn't it be "-ware", anyways?

------
sanitz
I recommend [https://concourse-ci.org/](https://concourse-ci.org/)

~~~
agentx3r
I really love the ideas/architecture behind Concourse, but there's a few
things that disqualified it in favour of Jenkins for a new CI pipeline during
prototyping:

\- no hooks to manage worker scaling
([https://github.com/concourse/concourse/issues/993](https://github.com/concourse/concourse/issues/993)).
Our builds are _heavy_ and we'd run up an enormous AWS bill without something
like
([https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin](https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin))

\- no way to restart a build, resorting to 'empty commits', which is a huge
red flag for useability
([https://github.com/concourse/concourse/issues/413](https://github.com/concourse/concourse/issues/413))

\- limited documentation/examples (network effect)

~~~
tedk-42
Wow I never knew Jenkins could scale its workers like that. We definitely have
performance issues with our Concourse cluster running in K8s. It's got it's
own set of dedicated nodes but we need to scale the workers better as they're
ofter under heavy load throughout the day when the devs are pushing code and a
bunch of tests are running in Concourse (and PR checks as well)!

Concourse definitely feels more refined than Jenkins. Like other commenters
have said, it's a steep learning curve to grasp how things move between
Tasks/Jobs.

------
michaelneale
Co-founder of cloudbees here, just some corrections:

1) there are (since 1 year) restartable builds/stages in open source (a bit
over a year IIRC) - when the article mentions that they are not in open
source.

2) Jenkins X is something VERY DIFFERENT from Jenkins (despite the name), it
is "master less", yes requires a kube cluster to power it, but it has no
Jenkins instances as you know it, in a lot of ways quite a bit different as it
uses Tekton as the engine for running pipelines (which has a bunch of
advantages). So I wouldn't group it in with the same challenges and
constraints. It is something new that shares a name and some of the ideas.

Jenkins, along with Spinnaker and Tekton and Jenkins X are now part of
[https://cd.foundation](https://cd.foundation) \- worth a look to see how they
are evolving (expect some changes).

~~~
manigandham
Unfortunately people are looking for simplicity and this is going in the
opposite direction. A kubernetes cluster is just offloading a large part to
yet another component you have to run. Spinnaker is another completely
separate system.

This is the opposite of what teams want. Drone, Gitlab, Teamcity as mentioned
in these comments is a far better approach for 99% of companies who want a
solid working solution.

~~~
wsy
Soon, "running on Kubernetes" will be like "running on Linux", i.e., it won't
add any operational complexity because you anyway have a Kubernetes cluster
running.

So maybe you are not there yet, but for a future-oriented CI/CD platform with
self-hosting option, using Kubernetes as basis is a good approach.

------
2StepsOutOfLine
Tangentally I've had great success with Drone. Holding similar frustrations
that other commenters are expressing here it was a breath of fresh air to see
how dead simple making custom steps in Drone was compared to actually needing
to write code for Jenkins.

Every step in Drone is just a container, so if I want a Golang build container
I can just set the step image to Golang:latest and start running build
commands. And if you're wanting to encapsulate some logic into a "plugin" you
can just pre-make your own docker image and from there any settings you pass
in with the steps are converted to environment variables that can be accessed
inside the container. Many of the default drone plugins like docker registry
build, etc. Are just a couple line bash scripts on GitHub.

------
littleweep
I think a better headline would have been something more like "Jenkins doesn't
have the features I require". There are plenty of examples of older pieces of
software or older languages that still work perfectly fine today.

~~~
kissgyorgy
You should really read the article before you comment on it.

------
sudofail
What I'd like to see is a CI/CD engine backend. Most CI/CD solutions have
relatively the same feature set:

\- Code builds

\- Code deploys

\- Git Hooks

\- Parallel builds

\- Matrix builds

\- Build as configuration

\- Etc

If there were an engine that supported these features, the community could
create a number of frontends that target their specific needs.

Jenkins was held back, in my opinion, for a number of years by its antiquated
UI. And while Jenkins does have an API of sorts, writing a new UI for it is
not a simple task.

I've used Drone for a while, and it gets a lot of these things right, but at
the end of the day it is a holistic, integrated solution.

I'd like to see a composable, modular CI/CD framework.

~~~
bogomipz
What is a "matrix build" in the context of CI/CD?

~~~
sudofail
When your software needs to build on, or run on multiple targets, it's useful
to define it in a matrix. Here's an example:
[https://0-8-0.docs.drone.io/matrix-
builds/](https://0-8-0.docs.drone.io/matrix-builds/)

------
Cedricgc
[https://outline.com/VvVhg6](https://outline.com/VvVhg6)

~~~
rglullis
Yours is not the first post that contains nothing more than an outline version
of the link. Why is this becoming a thing? The page seems perfectly readable
as it is...

~~~
lultimouomo
The page is actually a medium post which immediately pops up an annoying modal
trying to get you to give them your personal information and phones home who
knows what kind of analytics.

I'm not sure about the ethics of outline.com, but I can totally see why people
are using it.

~~~
kristianc
I'm not sure about the ethics of taking other people's content and putting it
behind a paywall.

~~~
pilif
But it's Medium's content. You gave it away when you posted it here. In
exchange, you got exposure[1]

[1]:
[https://theoatmeal.com/comics/exposure](https://theoatmeal.com/comics/exposure)

~~~
detaro
As far as I know, Medium writers opt-in to this, in exchange for money if a
paying user reads the article.

------
0xdeadbeefbabe
Is buildbot old too? I sure enjoyed the flexibility.
[https://buildbot.net/](https://buildbot.net/)

~~~
herge
One complaint about buildbot is that if you get too creative, your
buildmaster.cfg gets very hard to maintain, but if you stay very diligent,
just having python (and being able to print or log whatevery is happening)
makes debugging and having complex setups be very easy.

Also, if you ever need to schedule jobs/tasks (not just ci builds) across
multiple machines, buildbot is great because all you need is a master, and
slave python processes which just need a network connection to the master.

------
vladletter
They should definitely dig more into Jenkins X: it evolves a lot and it's
pretty new but they're gonna have a stable version soon!

------
robax
I'm still pretty new to devops and I personally love Jenkins but man is it a
PITA at times. My biggest gripe is that every plug-in seems to be no longer
actively maintained. We really on a huge list of plug-ins and in some cases
upgrading Jenkins will break plug-ins. It's gotten to the point where I'm
afraid to update to even minor or patch versions.

------
EngineerBetter
I'm a very strong Concourse proponent. We've been using it exclusively for
three years, and it's better than anything we've experienced that our
customers use (I run a consultancy).

We run Concourse London User Group, and the first few instances were like some
kind of Jenkins survivor's support group.

------
linuxftw
Trying to integrate Jenkins w/ Github Enterprise proved to be a real pain for
me. It wasn't clear what git* plugin is the best to use for how I wanted to
setup builds. It's literally impossible to deploy Jenkins in a repeatable
manner with just configuration files, you must get it online and use the api
to configure such simple things as API keys (eg, specify the ID of the key so
your jobs actually reference the key and don't break).

Another big pain point, for me was, no obvious openstack integration. I wanted
my jobs to run on ephemeral instances so all my dependencies could be defined
in the job themselves, not having to rely that the Jenkins worker is setup
_just right_.

I think Gitlab runners get this right, but I have not investigated too far.

~~~
mathewpeterson
You definitely can configure Jenkins in a repeatable manor. See
[https://wiki.jenkins.io/display/JENKINS/Groovy+Hook+Script](https://wiki.jenkins.io/display/JENKINS/Groovy+Hook+Script)

You can write all the configuration in groovy and it will execute when the
Jenkins process starts up.

There is also "Jenkins Configuration as Code" plugin which allows you to have
a single yaml file to configure most of the Jenkins system.

[https://github.com/jenkinsci/configuration-as-code-
plugin/bl...](https://github.com/jenkinsci/configuration-as-code-
plugin/blob/master/README.md)

~~~
linuxftw
I managed to accomplish the task at hand ~2.5 years ago, so yes, it's possible
to deploy jenkins in a repeatable manner.

I guess what I'm driving at is that it's not very declarative and it's
cumbersome.

------
chrisswanda
Jenkins is to CI,

as Nagios is to monitoring.

~~~
geggam
Both work ?

~~~
mschuster91
No, the key difference is that Nagios has perfectly viable alternatives
(Zabbix admin here and it's not the only one). Jenkins however has no
alternatives.

Bamboo or whatever it is called now is typical Atlassian crapware. Expensive
as fuck, eats more resources than the stuff it builds, and did I mention it is
yet another half assed product that got shoddily integrated into the usual
Atlassian lineup?

Gitlab CI is great for anything that is _code_ (think build, test, deploy),
but it is not suited for abstracting "non-development" jobs which can
perfectly be automated in Jenkins (e.g. creation of a dev environment with
fresh data from production). Plus it is Docker and the runners are polling -
which means at minimum 10s _startup time_ compared to milliseconds for a
Jenkins shellscript job running on a ssh connected slave!

Github and friends are cloud which is a big no-no. We're placing too much
power in the hands of AWS, GCE and Azure already, no way in hell it is a good
idea to put private source code to a cloud provider.

~~~
loopbit
GoCD?

------
sethammons
Our org is happy with buildkite. We moved to that after fighting Jenkins for a
number of years.

------
solarengineer
See [https://build.gocd.org/](https://build.gocd.org/) for the live instance
of GoCD used to build Gocd (apache licensed CD orchestration tool backed by
Thoughtworks).

------
killjoywashere
</read-article>

Um, so doesn't his workflow buy an essentially infinite maintenance problem
with all his containers? This sounds like tomorrow's Node left-pad debacle in
a different form.

------
closeparen
Of all the undifferentiated heavy lifting you can offload to your cloud
provider, CI makes a lot of sense:

\- You probably want to store artifacts in its object store anyway.

\- Rent exactly the right amount of compute for the concurrent builds running
in any given minute.

\- If your projects have sufficiently simple build-and-test entrypoints i.e.
"make" then you're highly portable and not too worried about lock-in. You
don't even need to be deploying to the cloud provider you're building on.

------
eeeeeeeeeeeee
We ran Jenkins in-house and it was a miserable experience. Always breaking.
Any time we needed to update it we knew it was going to break, and it did.
Debugging startup failures was always a pain if you had a moderately complex
setup. There were times where it was quicker to wipe the entire configuration
and start over than trying to debug it.

We switched to CircleCI and it did everything we needed and we never had to
lose half a day to CI problems again.

------
outworlder
I agree with this article on all points. Currently we have a massive jenkins
pipeline sprawl that's difficult to maintain. It is also difficult to create
new jobs in Jenkins itself, specially if you are using pipelines. My average
is around 100 test builds before I can get a full pipeline success for
anything of modest complexity.

If all you are doing is using jenkins to run simple bash scripts, you may be
able to get away with it. The problems start when you want to add some logic
to the pipeline – which you are doing, otherwise why bother with a pipeline?

First things first: are you going to use the scripting pipeline, or the
declarative pipeline? The declarative pipeline is a bit better, but it lacks
examples, has lots of bugs (I've littered my code with references to JENKINS-
XXXXX) and is very restrictive (arguably by design). Of course, you can have
'script' blocks inside your pipeline->stages->stage->steps blocks.

Then you want to take advantage of parallelization or conditional steps, and
to visualize that you want Blue Ocean. Problem is, not all plugins are
compatible with Blue Ocean, it also doesn't have all features, so you drop
down to 'old' jenkins often.

People will want to have a whole bunch of tools with incompatible versions in
their builders. Not all are supported natively, so you need to figure out your
versioning.

Once you figure all that, congratulations. Next guy to automate something will
either find a similar pipeline to copy from, or will endure all the pain
again. At this point you may want to use Groovy.

Groovy was absolutely the wrong tool for the job. Yes, I get it that it works
with Java, which Jenkins is based on. Still it is the wrong choice. You see,
the kind of things you want to automate often involve passing commands around,
be them bash, ansible, SQL statements, what have you. Groovy's string escaping
rules will ensure your life will be pretty miserable
([https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e412...](https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4))

You could get around most of these by perhaps moving most of the logic to
containers and then running those. There again you'll run into problems with
declarative pipelines, random things won't work and you'll be scratching your
head.

However, if you are going to do that anyway, you're better off using a more
modern system for CI, any system. Drone was already mentioned, there's also
Concourse and a bunch of others. For CD, you can use Spinnaker as well.

Or maybe keep jenkins around but forget all the fancy stuff. Delegate all the
'thinking' to scripts and pretend the more recent development has never
happened. You'll be saner that way.

~~~
humbleMouse
If it takes you 100 tries for a moderately complex jenkins pipeline, you have
other problems that are not jenkin's fault.

~~~
outworlder
I wish I could bring you here to see you do better.

Or do you mean systemic corporate problems? In that case, I agree.

It still doesn't change the fact that Jenkins does not make my job any easier.
I'll spend a day worrying about Jenkins idiosyncrasies ("why can't I use a
pipe in sh", "why did my bash escaping disappear completely", "why 'dir'
doesn't work with a container build agent?! (JENKINS-33510)", "why this input
plugin won't work with blue ocean", "why can't I use a for loop in this piece
of code in particular but it works elsewhere" (JENKINS-27421)).

Whereas with concourse or other newer build systems I can write a simple YAML
description, which is modular and uses an existing standard, and test that in
isolation. And then provide it as a building block for other tasks.

~~~
humbleMouse
I feel you, but why are your jenkins pipelines so complicated? I feel like
your workplace's deployable artifacts should follow a familiar pattern and
there should not be much guessing/re-inventing the wheel with jenkins scripts.
I feel like complicated builds are usually the result of an application that
is not very well thought out in the first place.

~~~
outworlder
> I feel you, but why are your jenkins pipelines so complicated?

You have an excellent point. Individual microservice containers are not
complicated (then again, all they do is call a standardized script). The
script will run a Dockerfile and push it to the registry. I would classify it
as a 'trivial' Jenkins job, not even pipelines are used.

The pain starts when you want to do more than CI and try to get into CD. Or
even worse, automate 'devops' tasks. That's where you run into all those
warts.

A job could call Terraform, or spin up VMs, or run vacuum on a database, or
any number of tasks. Or it may perform tasks on K8s to deploy a complex app.
It may need to call APIs to figure out where to run things. And so on.

Since Jenkins is not only a CI/CD system, it can do anything, so people will
try to make it do increasingly complicated stuff. And I'm arguing that this is
wrong. If you have complex logic, it should be moved out of Jenkins so it can
be more easily maintained and tested, and dependencies isolated. One of the
easiest ways to do that is with containers. At which point, Jenkins loses most
of its usefulness and other, newer tools shine.

Alternatively, use more specialized tools. If it is for CD, and Spinnaker
works for you, please use that instead.

~~~
humbleMouse
I agree with you that bash heavy complex things are best suited for something
like ansible.

However, deploying containers to environments like openshift and kubernetes is
extremely simple with jenkins. I don't think that's complicated at all. As a
rule of thumb, you should be able to hide all the complexity in your
deployment in the dockerfile. In addition, you can always use jenkins "build
with container" functionality to build your application in a dedicated
container on the fly. Many ways to hide complexity with jenkins.

I do agree with you that jenkins is abused because it is more than a CI/CD
tool. I think that you need some experience using it to know what works well
and what doesnt. Unfortunately in the new age "sprint agile" world some random
guy has to pingeonhole crap into jenkins in 2 week time windows that shouldnt
be there in the first place.

I also think that many devs underestimate what you can do running local
jenkins as a tar file on your macbook. I like using jenkins to automate
tedious tasks for myself. As an example, it is trivial to write yourself a
custom github code scanner that will scan all files and folders in as many
repos as you want. I like using jenkins for outside the box things like that.

------
zhynn
Interesting, not a single mention of Vespene, or more relevantly:
[https://medium.com/@michaeldehaan/discontinuing-
vespene-17fd...](https://medium.com/@michaeldehaan/discontinuing-
vespene-17fd9c0f3079)

Michael DeHaan worked on it after Ansible, but he got zero traction, and so
shuttered the project. I think he should have stuck it out longer, because
people hate Jenkins.

------
stan_123
No mention of Jenkins Shared Libraries? When Jenkinsfile pipelines start
getting complex, or need to make them more generic, you can create groovy
lib-s in a separate repository. Single line to load the lib dynamically at
start of Jenkinsfile. Then you can call library functions/tasks passing
appropriate params within your pipeline. But as anything Jenkins,
documentation lacking or out of date.

------
sisu2019
Normally I have no issues getting things to work but I bounced off trying to
get modern Jenkins to build a github gradle project three times now. I even
used to have one, way back, that built maven projects from SVN and deployed
into artifactory but I can't seem to manage the same now. I thought it was me
but maybe jenkins just became really awful in the meantime.

------
infamousjoeg
This completely fails to mention the security issues with Jenkins, as well.
[https://www.cyberark.com/threat-research-blog/tripping-
the-j...](https://www.cyberark.com/threat-research-blog/tripping-the-jenkins-
main-security-circuit-breaker-an-inside-look-at-two-jenkins-security-
vulnerabilities/)

------
pjmlp
Not at all.

In what concerns Java and .NET projects, Jenkins is just a mature tool that
gets the job done.

We have no reason to consider anything else.

------
ogtifs
Anyone looking for a more back-to-basics self-hosted CI might be interested in
Laminar[1] - this article reminds me why I was inspired to write it

[1] [https://laminar.ohwg.net/](https://laminar.ohwg.net/)

~~~
oblio
It's not distributed, from what I see. I don't see stuff about agents.

Oh, it does, via DYI SSH, sort of. Could work, I guess.

~~~
ogtifs
Right, the philosophy is more about making it easy to DIY stuff and getting
out of your way, rather than endlessly tacking on features and integrations.

------
keithnz
We currently use Appveyor (good) and TeamCity (okish) and feed into Octopus
Deploy. We have ~50ish projects (web, backend systems, embedded firmware) that
all go through these systems. Works really well!

------
peterwwillis
_Getting_ old? It's ancient. Nobody should use Jenkins for new projects. I
even started writing my own piece about it on a plane back from a recent
vacation.

------
breatheoften
I want pause on breakpoint <x> in cloud worker <y> from my IDE — tooling which
does not assist me to bring that about is working against my needs.

------
KaiserPro
I have been with jenkins since about 2010.

At its peak I had to shepherd ~50 masters of varying ages, plugins and levels
of customisations.

Yes, it allowed repeatable builds, and it can be, in certain circumstances a
very good replacement for cron jobs where you need to administer >5000
machines.

However, it has an offensive UI, the plugin system appears to be designed to
deliberately cause things to break, versioning the flows is difficult, unless
you rely on plugins (which break) or want to learn groovy.

Automating and making jenkins it's self repeatable is a soul sapping task.

In short, jenkins needs to disappear.

Yes, cirle is crap, bamboo suck massive genitals.

However I have had great success with gitlab runners. Now that gitlab is
actually reliable, they are a brilliant first class integration into a git
workflow.

~~~
badloginagain
GitLab CI is pretty good.

I hate, on a fundamental level this whole "bash in YAML" trend. Let's take a
powerful, turing complete language and put it in a _config file_.

What? I don't even...

And the fact that you only get _one_ .gitlab-ci.yml file. That is legitimate
madness. Want modularized CI/CD configuration? Not in these parts, buster.
Have submodule build dependencies? We laugh at your use case.

Sure as shit better than Jenkins though.

~~~
core-questions
> I hate, on a fundamental level this whole "bash in YAML" trend.

This is why I hate Ansible. Looks simple until you need to do something
complex, and then it becomes horrible. Chef's use of an actual language is far
better, though on the flip side it's much harder to provide any kind of
interface to configure it outside of a text editor.

If only there were some language where config data and code were the same data
structure. (hmmmmmm)

~~~
Jedd
> If only there were some language where config data and code were the same
> data structure. (hmmmmmm)

Do you perhaps mean syntax rather than structure? (Or a subset of the syntax.)

If I've guessed correctly, that language has already been wedged into lots of
places that it's a poor fit, so I guess there's precedent there.

~~~
mynegation
I am pretty sure Lisp was wedged into some places where it is a poor fit, but
“lots”? Did you guess Javascript by any chance?

~~~
Jedd
I did guess Javascript, as it happens.

Was I wrong? If it was meant to be a reference to Lisp, I take much of it
back.

~~~
ISO-morphism
The parent was most likely referring to lisp, but note that there are other
homoiconic [1] languages.

[1]
[https://en.m.wikipedia.org/wiki/Homoiconicity](https://en.m.wikipedia.org/wiki/Homoiconicity)

------
RickJWagner
It seems like KNative wants to move into that space.

Regular turnover in the Programming world. You have to re-learn everything
once or twice a decade.

------
liveoneggs
groovy pipelines solve a lot but cost a lot. Finding where to draw the line is
really really hard. Anyway jenkins works, is free, is easy to setup, and easy
to use for everyone except the one writing the groovy :)

groovy sucks, jenkins works (high praise, really), but you get what you get

------
FraaJad
How does sourcehut (sr.ht)'s build service compare to the alternatives
mentioned here?

------
sabujp
drone is like google cloud build and probably and whatever amazon has

------
matsilva
Classic product promo by poo pooing on someone else.

------
fnord77
gitlab CI is awesome. too bad it is not a standalone thing.

~~~
jl-gitlab
You can actually use it standalone, for example with GitHub repos:
[https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/githu...](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html)

(Disclaimer: GitLab employee here)

