Best CI I've ever used is Gitlab CI. The runner is completely open source and you can use your own runner with your gitlab.com (or local instance) projects -- set it up in an autoscaling group for savings.
I run https://runnerrental.club but Gitlab also recently released the ability to pay for minutes in 11.8 , so my product is more-or-less dead in the water but I don't mind since Gitlab is such an excellent tool, I'm glad to see them fill the need.
But back to Gitlab CI -- the YAML configuration documentation is pretty fantastic -- Most easy things are easy and hard things are possible. I suspect that one could run an entire startup like circleci/travis/drone based on just the software that Gitlab has open sourced and made available already.
The artifacts feature is great and some artifacts, like unit test report files, can be interpreted by GitLab and used in various parts of the GitLab web UI. A lot of this just works and most of the CI features are available in the GitLab community edition which is open source.
I have not used Jenkins actively since before the Jenkins pipeline file format was common. So for me Jenkins always appeared to be this game of checking the right checkboxes and clicking the right buttons in the Jenkins UI. The new pipeline feature is probably much nicer. However, now that I use GitLab I don't really see any reason to switch back to Jenkins.
Our hope to simplify away the checkboxing and plugins is ready to go distro (free of course):
You mention docker and it is super great for CI: it’s probably one of the widest used feature of Jenkinsfiles (you can specify what image or dockerfile you want a stage to run in - simple but powerful, as you probably know)
- Can't customize your git checkout process (e.g. shallow clone with depth, or merging source branch with target branch with certain strategy)
- Can't make job run/not run based on filter on source branch/target branch/etc. of a merge request
- Can't dynamically make certain jobs only run on certain agent machines
So I'm still stuck with Jenkins for now. I know we love bashing Jenkins but I have yet to come across anything that offers the same amount of flexibility.
1) you can customise the git checkout depth and style: https://docs.gitlab.com/ee/ci/yaml/#shallow-cloning https://docs.gitlab.com/ee/ci/yaml/#git-strategy
I'm not associated with gitlab at all but happy to give pointers if anyone wants to contact me direct
So it would look something like this:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"
We have an overall CI/CD direction page up at https://about.gitlab.com/direction/cicd which you can drill down into the individual stages plans from. Feedback is always welcome, we love building things in partnership with real users. You can reach me at firstname.lastname@example.org any time.
They are displayed in merge requests but not anywhere in the normal CI pipeline UI and I'd live to be able to see those and see what works and what doesn't.
Yes - that's something we want to get to this year. We call it "CI Views" right now but we want to expose better all the types of test results that get collected by GitLab. Today for XML (JUnit) results, you can see what fails in the merge request with https://docs.gitlab.com/ee/ci/junit_test_reports.html.
But we want to make the XML, JSON or HTML output of _any_ type of test first class with CI Views: https://gitlab.com/gitlab-org/gitlab-ce/issues/35379
If you have any questions or feedback, you can email me at at email@example.com. I'd love to hear more about how you are using the docker registry today and any improvements you'd like to see made to the product.
I'm not super familiar with Circle CI's SSH debugging, but we do have a feature called "Interactive Web Terminals" https://docs.gitlab.com/ee/ci/interactive_web_terminal/.
This allows you to connect to a terminal in the running job on GitLab CI for debugging. Would love to understand how this does or does not meet your use-case here.
CircleCI supports SSH connection for max 2h, then shutdown the task.
This is not a full standalone mode, because GitLab CI is a built-in solution that can not be easily separated, but might work for you.
We set all jobs to manual and then our script triggers them depending on commit message, person, moon phase etc.
But really, this should be in core. Its very hard to do multirepository stuff. Its not that easy to do mono repo stuff too - I really need a pipeline withint sub-project, a mother pipeline, option to run whatever one I want etc... Gitlb pipelines could be a lot better.
Also, running pipelines in different projects and whenever you want is accessible from the web interface... It could be cleaner, but IMO adding even more syntax to the YAML file can also be a rabbit hole -- CURL from the script section seems like not a bad middle ground.
Am I misunderstanding what is being referenced?
As long as there is some release coordination, a system like this can work. In particular, I've found that the lightest way to get a system like this working for x > 1 developers is to have a release-vX.X.X branch for releases that are going out and vX.X.X tags for releases once they have actually landed.
You mention the recognizing and tagging versions being an issue -- are you imagining a world where two people are releasing something at the same time? I'm not exactly sure
If we assume that what that posts suggests isn't feasible, regardless of how you deploy and do your CI, if your release process can be done by a human, it can likely be automated. Gitlab CI is most robust and yet easiest to understand automated CI system I've seen for making that happen (at least that was my intended point).
I have found that there are moments that start to give chaotic behavior in CI that are influenced by the number of people working on mono repository and some other stuff like code maturity: 1 dev, 2-5 devs, 5-10 devs and 10+ devs .....
Something that works great for any number might start to break for other number. Something effective for huge team may be very bad for smaller team. Something that works great in one moment (first several months of the project when thing still get shape 0.X.Y versions) sux when we move to 1.x version and vice versa.
I am not aware of anybody giving more detailed thought about this nor examine what could be most optimal solution for any type of team (if such thing exist) or on what precise moment you start doing certain practices (for example protecting master branch).
Am I misunderstanding what you meant?
In order to pull other pipelines through Gitlab's API, you have to build another CI on top of an existing one. What I mean is that you're making the a pipeline to handle projects, but there's a catch - you have to run another pipeline to set the variables in the One pipeline to rule them all because you can't override variables from the jobs themselves! And the main pipeline would run a bunch of scripts to pass variables and handling the state.
Now let's compare that to other systems from my experience. Jenkins way: one Groovy script (might be written as a special job type from UI or through VCS) to do the job with some fancy features like pause to confirm the next step from the CI's Web UI, full control of everything. Teamcity way: either could be made from UI (with reverse dependency chain because you need to start connecting pipeline from the last jobs to first) or write yours from scratch with Kotlin DSL (there's still no direct support for pipelines), you can handle the process through dependency settings and by setting variables from the jobs themselves.  An example of Teamcity approach. (black boxes are there because of my NDA)
edit: added picture.
edit2: added link to TC blog and rewrite the link to a screenshot.
This is really valuable feedback, thank you for it. It's something we are thinking about and working on fleshing out a vision for. Two epics we have around these ideas include  Making CI lovable for monorepos and  Making CI lovable for microservices.
Both of these challenge our current assumptions around the project::pipeline relationship and will help make room for improvements around that model and provide flexibility to build more complex models that work for "real world" problems like the ones you've stated. However, I don't believe we have all the answers yet so I would love more feedback on these and the issues attached.
I.e. Project1 has branches in form feature/TICKET_NAME or fix/TICKET_NAME, Project2 uses only TICKET_NAME, TeamCity VCS settings are based upon masks of branch names so ticket 12345 would be branch 12345 in P2 but fix/12345 or feature/12345 of P1 if one exists (priorities could be tweaked too).
While it is easy to bash on an inanimate object, there are some very dedicated and empathetic people who care deeply about the project. Some of those people do this work in their off-hours and some to this work as part of their daily work activities AND also in their off hours.
In that spirit, we want to make Jenkins better and created a separate group at CloudBees in the Product and Engineering teams late last year. They focus on open source work for Jenkins and on some proprietary things for CloudBees Core (built on Jenkins). We also have a dedicated user experience/product designer who started working on the project a few months ago. One of the first things he and I worked on was creating a curated, tailored version of Jenkins via the CloudBees Jenkins Distribution. This distribution will focus more and more over time on a guided workflow for continuous integration and delivery with Jenkins. These patterns will also be shared with open source Jenkins - some through direct contributions and others through suggestions (better plugin categorization, documentation, etc.).
Please use this comment thread to share your constructive, honest feedback about how we can improve Jenkins.
And can we PLEASE open up the issues tab on the Github repos especially for plugins. There is currently no way to provide any feedback/report issues on these plugins because the "issues" tab is disabled. Our current options are
- Put a comment on the plugin wiki page
- Hunt down the the relevant support forum for the plugin
- Find the original source repo and post an issue
All of which are not ideal and does nothing to directly help the development of the plugins.
I've been using Jenkins heavily over the past year since a client of our purchased the enterprise version. It was billed to me as a mature open source product that has been refined over the years by people trying to optimize their dev operations. While I'm sure some people really care about it, the user experience is so utterly disappointing it's almost impossible to imagine how it got to this state without neglect.
Even if you ignore all the complicated stuff, the web UI is embarrassing. While I don't suggest a "pretty" UI is necessary for devops, I would think you'd have a quick win just by having some people re-style the existing UI to make it look and feel like something built this century and not require a dozen clicks to get to important information. There are also bugs that are so painfully obvious it makes me wonder how they still exist. An example is if your Github branch has a slash in it (eg. feature/something) you get a 404 error if you try to navigate to that builds' results.
There are also features that appear to have almost no value yet are in the core UI and clearly took some time to build. The weather icons representing various permutations of previous build states is one ridiculous example that comes to mind.
I would respectfully suggest you run through some real world Jenkins experiences like the ones mentioned in the article. Also setting up a new server, configuring non-trivial SCM settings, debugging Jenkinsfiles, etc. To echo the article's sentiment - it feels like I'm constantly fighting with Jenkins to do what I need instead of being guided into a mature set of features.
Conversely - Octopus Deploy is a related product I have been using alongside Jenkins which has been an absolute joy to work with. Everything from initial setup to configuring its agent software on host servers has been straightforward. It has a simple, elegant UI that provides access to important information and actions where you would hope to see them. And most importantly - everything works. I have yet to encounter a bug or experience any broken UI states.
I'm glad to hear CloudBees is making some effort to improve things and I hope PMs like you continue to be involved in the community and solicit feedback, even if it's hard to hear sometimes.
For myself, I'd prioritize -
An indepth improvement of the Config as Code system as applied to Kubernetes, both for management of the config and of relevant plugins.
Plugin compatibility and management of plugins. It's not... smooth.
Also, the documentation badly needs updates and examples.
(Also, amusingly, we chatted a bit by mail on April 25th 2018, but there was no follow up on your side, I guess priorities changed...)
The much-touted repo integrations (travis, circle...) all have an exclusive focus on build-test-deploy CI of single repos.
But when you have many similar repos (modules) with similar build steps you want to manage, and want to have a couple of pipelines around those, and manage the odd Windows build target, these just give up (it's docker or bust).
Sadly, only Jenkins is generic enough, much as it pains me to admit.
Anyone got a sane alternative to jenkins for us poor souls?
On the other hand there is Bamboo from Atlassian. https://www.atlassian.com/software/bamboo
I really don't understand this mentality of there is no better tools when there are better tools than jenkins and they've been around for a while.
Bamboo on the other hand is IMO the worst of the commercial CI tools by far and where I work has gone down for us the most. Atlassian itself doesn't appear to be investing in it much anymore judging by the slow pace of development in recent years and at their most recent conference, you can hardly find it mentioned or see much presence for it anywhere.
In all the CI systems I've used though, there has not been one that I haven't encountered some major difficulties with.
Beyond that, anything to do with build automation for a large number of users always quickly becomes a support & maintenance quagmire. Users frequently want to install a new (barely maintained) plugin to solve a problem they have, complex interactions lead to difficult to understand failure modes that require time consuming investigations ("your CI tool is the problem and broke my build" ... "No, your build is broken" ...).
First you checkout the project from the repo and it just works, doesn't matter GIT, SVN or whatever. How many plugins does it take to checkout a project in jenkins? Is there even a git plugin working nowadays?
Then, you build the project. If it's any of C# or Java for example, the ant/maven/sln/nuget files are detected automatically, just click next and it's built. Does jenkins even understand what is a requirements.txt? Hint: It's not a bash script.
The JVM and the Visual Studio are detected automatically on all the build slaves and the project is already building in the right places. If you want to aim for specific tool versions, there are presets variables on all hosts to filter where to build and to use in build scripts so paths are always right. How is the build matrix plugin in Jenkins lately? Broken as usual?
The project is built, or is it building? It's easy to tell because there is a clear colored status icon and the estimated time to completion is displayed. Teamcity offers that out of the box for maybe 15 years now. Well, jenkins finally got a progress bar too a couple years ago. I guess I'm defeated, Jenkins caught up on basic core functionality only a decade late, I can't justify to pay for working and polished tools anymore. Well, I hope our sysadmin will install the Extra Status Icon Plugin or we'll have to live without the big colored circles next to the build.
I think you’re not appreciating and misrepresenting the complexity and power that comes with these solutions.
And actually, you can get quite far with the free TeamCity license of three build agents and 100 build configs. I’m also fairly sure that Jetbrains would take kindly to license requests from open-source projects and academia.
They do: https://www.jetbrains.com/buy/opensource/
Gitlab has something called templates but it's a very different thing. In Gitlab, a template is used to bootstrap a project, but that's it. In TeamCity a template is attached to a project such that if you change the template, changes are applied to all projects that inherit from the template. Each project can override any settings or build steps it got from the template, without losing the association to other settings. A project can have multiple templates attached to control orthogonal aspects of its behavior. From a template, you can see what projects inherit from it, and you can freely detach and attach to a different template. It makes managing a large number of projects with similar configs, that all evolve at somewhat different rates really easy.
- Build results
Teamcity has very good integration with xUnit and code coverage tools to quickly see test results and coverage as part of a build. Gitlab recently got better at this (it can now at least parse xUnit results), but you can still only see test results in the merge request view. TeamCity can also do things like track a metric over time and fail a build if it drops (i.e. PR builds should fail if code coverage drops more than X %). TeamCity also supports adding custom tabs to the build page so that you can attach reports generated by the build easily viewable in the UI (vs in Gitlab where you have to download the artifact and then open it to view)
- Overall view of runner status
It's very easy in TeamCity to see the build queue, and an estimate of when your build will run, and how long it's expected to take based on past builds.
For me it's easier in TeamCity to see the overall status of deployments to a set of environments (i.e. what's on dev/stage/prod) that might span multiple source code repos. At a glance I can see what changes are pending for each environment, etc. In Gitlab things are too tied to a single repo or a single environment, and the pages tend to present either too much or too little information. Also, in TeamCity I can configure my own dashboard to see all of the stuff I care about and hide other things, all in one place.
- System wide configs
There are some settings that apply to the whole system (repository urls, etc). There's no easy way in Gitlab to have system wide settings, they have to be defined at the group or repository level. In TeamCity, you can configure things at any level, and then override at lower levels.
TeamCity supports plugins. I know this can lead to the Jenkins problem of too many plugin versions, etc, but in TeamCity you tend to use far less plugins, and the plugin APIs have been super stable (I've written plugins against TeamCity 8 which is 4 major versions old and they work fine on the latest). It's really nice to be able to write a plugin that can perform common behavior and have it easily apply across projects and be nicely integrated into the UI.
To me, overall Gitlab CI seems useful for simple things, but overall it's 70% of the way to being something that could replace TeamCity.
We actually have done a lot here recently, we've improved includes so that they have a lot more flexibility (https://docs.gitlab.com/ee/ci/yaml/#include), and have even refactored our own Auto DevOps implementation to take advantage of this: https://docs.gitlab.com/ee/topics/autodevops/#using-componen.... In this way, you can have included behaviors across your projects that can then be updated in bulk.
We are planning on adding testing results over time in our vision for this year, thank you for confirming this is important from your point of view. https://gitlab.com/gitlab-org/gitlab-ee/issues/1020
We did recently add pipeline info to the operations dashboard (https://docs.gitlab.com/ee/user/operations_dashboard/), which I know isn't exactly what you're looking for here but we are making progress in this direction and recognize the gap.
The next improvement we're making to that operations dashboard is adding environments. You can see the in-progress issue here: https://gitlab.com/gitlab-org/gitlab-ee/issues/3713
This can be achieved by using includes to set the variables, which is admittedly a workaround. We do have an open issue (https://gitlab.com/gitlab-org/gitlab-ce/issues/3897) to implement instance level variables that would solve this.
This is an interesting one because Plugins are, at least in my opinion, what makes Jenkins a mess to use in reality and believe me, I've managed plenty of Jenkins instances in my career with lots of "cool" plugins that do something great, at least while they work. It is one of our values that we play well with others, though, so I'd be curious to work with you to understand specifically what you'd like to be able to make GitLab do that can't be done through your .gitlab-ci.yml. Our goal is that you should never be blocked, or really have to jump through hoops, but still not have to be dependent on a lot of your own code or third party plugins.
I'll give you a couple of examples of use cases for plugins:
We have an artifact repo that can store NPM, Python and other artifacts (Nexus if you're interested). I wrote a plugin for TeamCity that can grab artifacts from a build and upload them to the repository. Obviously this can be done in a script, but there are a couple of things that make doing it in a plugin nice:
- You can set it up as a reusable build feature that can be inherited from templates (i.e. all builds of a particular type publish artifacts to Nexus)
- You can get nice UI support. The plugin contributes a tab to the build page that links to the artifacts in Nexus.
- The plugin can tie in to the build cleanup process, and remove the artifacts from the repository when the build is cleaned up. This is useful for snapshot/temporary artifacts that you want to publish so people can test with, but have automatically removed later.
Another example of where plugins have proved useful is influencing build triggering: we have some things that happen in the build server, and then other stuff happens outside of the build server. When all that completes, we then want to kick off another process in the build server (that sounds abstract - think an external deploy process runs, and once the deploy stabilizes you kick off QA jobs). In TeamCity you can write a plugin that keeps builds in the queue until the plugin reports that they are ready to run.
While plugins aren't the first tool I reach for when looking at how to provide reusable functionality in a build server, I have written several plugins for both Jenkins and TeamCity. Overall, I don't think Jenkins/TeamCity's model of having plugins run in-process is a good one, and it leads to most of the problems people have with them (although TeamCity is much better here: Jenkins basically exposes most of its guts to plugins which makes keeping the API stable virtually impossible, while TeamCity has APIs specifically designed for plugins that they've been able to keep stable very effectively) A model where a plugin was just a Docker container that communicated with the build server through some defined APIs, combined with some way for it to attach UI elements to a build that could then call back into the plugin would be much nicer. This seems to be more like what Drone is doing, but haven't played around a lot with that.
I think Gitlab has a strong philosophy of wanting to build out everything that anyone will ever need, all nicely integrated, and that's a great ideal. I think in practice, it's REALLY hard to be all things to all people. People have existing systems and/or weird use cases that it just doesn't make sense to handle all of, and plugins are a useful tool in addressing that.
You may want to try a C#, java, python and a go projects to see the differences, with slaves on Windows and Linux. There are some pretty tight integrations for some of these.
* Broken base Ubuntu images being recommended by Atlassian as the default for agent Image configuration, only to be fixed to a usable state months later;
* Being generally years behind other CI tools, even the traditional ones;
* Data exports corrupting themselves despite apparently succeeding, blocking migrations after upgrades or server changes;
* The official documentation somewhere recommending copying across password hashes directly for setting up new admin for a migration, but I can't find this anymore so they've hopefully improved the process and documentation for this;
* A bug in an earlier version in which a strange combination of error cases in an Elastic Bamboo image configuration could spam EC2 instances in a constant loop, which we thankfully spotted before it ate through our AWS bill;
* No clear messaging from Atlassian about how the future of Bamboo compares to Pipelines. The official line seems to be that Bamboo is for highly custom setups who want to host their CI themselves, but you get the impression from to the slow pace of development that they're slowly dropping it in favour of Pipelines. I'd rather they be honest about it if that is the case.
Those are just the problems I can think of from the top of my head, anyway.
Not with pipeline files. I am a total Jenkins noob, but I was able to (relatively) quickly setup a minimal job that automatically pulls config from the relevant GH repo.
That said, pipelines has made a HUGE difference. I still want to migrate but this fixes a large pain point.
Not extremely well, but I did a small PoC where Jenkins is running in Kubernetes without persistent storage. Plugins are installed on boot with install-plugins.sh (part of Jenkins' Docker image) and configuration is done via Groovy scripts in init.groovy.d (stuff like https://gist.github.com/Buzer/5148372464e2481a797091682fabba...). It's not perfect (e.g. I didn't have time to find out good way to store & import old builds) & it does require some digging around to find how plugins actually do their configuration.
And yes, you do end up needing to use Groovy for anything non-trivial.
Or upgrade the the Jenkins master and watch other jobs fail.
And not to mention plugin Y and plugin Z crashing Jenkins when being run together because they share the same classpath.
While in the meantime one of the developer is trying to migrate his pipeline from one master to another and he finds out that of course they won't work because the plugins and configuration are not exactly the same.
This is exactly what OP was complaining about. You don't set up plugins and configuration just once. You want them to be replicable, but Jenkins does not provide a good way to do that.
Most other CI/CD system handle this issue very simply. They just don't have plugins, and have very little (if any) global configuration. This means you can start up an entirely new cluster and chances are your pipeline files will run without a hitch.
(My company switched from JJB to pipelines in the last year and has found it pretty decent.)
I use it for build and test automation and it's been pretty solid.
Each build pipeline is just a small shell script which does some setup and runs make to build or invokes our test entry points.
My biggest gripe about gitlab is you can't schedule a job in code, and I suppose it's less then ideal to support 3rd party repos in hosted gitlab, but I don't know why you'd not use it as an SCM.
The bigger problem, would be using a group job that triggers a bunch of other jobs to do the many modules type of development you spoke about, but I'd just develop my modules seperatly, and build them in parallel steps if need be.
Or are you looking more for putting the values in the .gitlab-ci.yml itself? This is something we have thought a bit about, but it gets strange with branches and merges where it's not always clear you're doing what the user wants as the different merges happen.
To your second point, you might be interested in some of the primitives we're looking at building next here: https://about.gitlab.com/direction/cicd/#powerful-integrated.... These, in concert, will help with a lot of more complex workflows.
With that said, the product is fantastic and I'm just pointing out some flaws so the parent understands I've actually used the product, and not just a fanboy yelling. :)
They have an integrated Docker registry as well!
Personally I’ve found that for my past and present use cases generating any needed step (e.g test matrix) e.g with a script is much more flexible, predictable, and reproducible since the generated result can be versioned.
I also successfully used various custom runners including baremetal Windows ones and virtualised macOS ones inside VirtualBox.
Since my use case is integration with gerrit, I poll the updated changes over ssh, and have the regex-based triggers which cause a “job” launches. Job consists of making a database entry and calling a shell script, then updating the entry upon completion. Since job is just a shell script it can kick off other jobs either serially or in parallel simply using gnu parallel :-)
And voting/review is again just a command so of course is also flexible and can be made much saner than what I had seen done with Jenkins.
So the “job manager” is really the OS - thus killing the “daemon” doesn’t affect the already running jobs - they will update the database as they finish.
The database is SQLite with a foreseen option for Postgres. (I have made diesel optionally work with both in another two year old project which successfully provisioned and managed the event network of about 500 switches)
Since I also didn’t want the HTTP daemon, the entire web interface is just monitoring, and is purely static files, regenerated upon changes.
Templating for HTML done via mustache (again also use it in the other project, very happy).
For fun I made (if enabled in config) the daemon reexec itself if mtime of config or the executable changes.
You can look at the current state of this thing at http://s5ci-dev.myvpp.net and the associated toy gerrit instance at http://testgerrit.myvpp.net
I am doing the first demo of this thing internally this week, and hopefully should be able to open source it.
It’s about 2000 LOC of Rust and compiles using stable.
Is this something that might be of use ?
* Get triggered by a github (enterprise) webhook.
* Work out the project, and clone it into a temporary directory.
* Launch a named docker container, bind-mounting the temporary directory to "/project" inside the image.
* Once the container exits copy everything from "/output" to the host - those are the generated artifacts.
There's a bit of glue to tie commit-hashes to the appropriate output, and a bit of magic to use `rsync` to allow moving output artifactes to the next container in the pipeline, if multiple steps are run.
But in short I'd probably spend more time explaining the system than an experienced devops person would be creating their own version.
How many teams do you have? In all seriousness, if you aren't talking at least one team per repo, have you considered a monorepo setup? Aren't you burning time managing those many similar repos with many similar build steps?
That said, even in a monorepo, I still prefer Jenkins compared to cleaner seeming cloud offerings due to its limitless flexibility.
I don't understand what you mean by one team per repo?
I agree, as I keep saying at $WORK: Jenkins is the least-worst system out there.
I think the people using TFA don't know what it means... Whenever I see that I think they are angry about something or arguing, but he's instead supporting the point of the article. Doesn't make sense.
The meaning of the "TFA" part does. The meaning of the "R" doesn't, which is why it is dropped.
> I think the people using TFA don't know what it means...
They generally do. You, however, seem to be confusing "derived from" with "means the same as". TFA is derived from RTFA, but it does not mean RTFA, nor does the argumentative implication of RTFA come along with it, since the argumentative implication is associated primarily with the implicit accusation that the target has not done what is expected in a discussion and read the source material that is the subject of discussion, which is carried entirely by the "R".
(One can read anger into the "F", but that's tamed by the fact that even in the context of RTFA/RTFM, that's often reconstructed into a non-profane alternative ["fine" is the one I've most frequently encountered.])
Some of us here have been using terms like RTFA and TFA for twenty years, maybe longer.
While I am a conservative christian myself (hah, most of you didn't guess that) I try to make a point out of not getting offended for such things, and if I can do it so can most people :-)
We keep our jenkins jobs version controlled by checking in each job's config.xml into a git repo. In the past I've seen the config.xml files managed by puppet or other config management tools.
This helps us get around the "hard to backup" and "test in production" issues. We can test out jenkins changes in staging, commit those changes to our jenkins repo, and then push up the config.xml file to the production jenkins server when we're ready to deploy.
I haven't tried this yet myself but AWS CodePipeline lets you have Jenkins as a stage in the pipeline. You use Jenkins only for the bits you need without the extra plugins. The resulting Jenkins box is supposed to be very lean and avoid the problems you describe.
Cloudbuild on the gcp side has had much better performance
What do you mean by underpowered buildbot steps? Are you implementing your own step classes?
In terms of underpowered build steps, I have several fairly complicated, 1k-2k line build factories, with multiple codebases and hundreds of steps (some custom, some from the stdlib). There's many dependencies between the steps, and many different properties that can be toggled in the UI. All these steps need to be defined up-front in the master, but their actual execution often depends on runtime information, so it becomes a mess of doStepIfs. I think it would be an improvement to give a program on the worker the power to tell the service what it wants to do, rather than the other way around.
Agree with you on the waterfall UI regression. It seems console view is preferred than waterfall in the recent versions. It's slower than waterfall UI though.
Concourse really hits his requirements for ops-friendliness and testability. It's easy to upgrade because the web host and CI workers are completely stateless, and the work you do with Concourse is easy to test because the jobs themselves are all completely self-contained. Because Concourse forces you to factor your builds into inputs, tasks, and outputs, it becomes somewhat straightforward to test your outputs by replacing them with mock outputs.
The main issue with Concourse is that it has a huge learning curve to use effectively. When you have many pipelines, you end up learning the undocumented meta pipeline pattern to help manage them. You end up implementing stuff like mock outputs by yourself, since it's not really a first-class concept. Concepts like per-branch versioning that have been in products like Jenkins for years are only now entering development as "spaces". All of the configuration can be version controlled, but it's all YAML, so to wrangle everything together, you end up using something which will compile down to YAML instead. RBAC much improved in v5 but still leaves much to be desired. There are no manual steps, only manual jobs. Job triggering is relatively slow due to a fundamental design tradeoff where webhooks trigger resource checks which trigger jobs instead of triggering jobs directly, to make resources easier to maintain. Nobody really tells you this on the front page.
It has its flaws. But if you delve into it you see very quickly that it's an incredibly solid product. The ability to run one-off jobs from developer workstations on CI workers, and to easily SSH into those CI worker containers from developer workstations, is incredibly helpful for debugging issues. Because everything is stateless, everything tends to "just work". If you go in with your eyes open about its limitations, you'll have a good time.
I hope to integrate Tekton into Drone  and allow individual projects to choose their pipeline execution engine. Projects can choose to start with a more basic engine, knowing they have the option to grow into something more powerful (and complex) when they need to.
certainly I love the idea of concourse as a release engineer but the lack of a nice UI for dev feedback/monitoring makes it a hard sell as a drop-in jenkins replacement
Either way, it's not a drop-in Jenkins replacement. It really does have a high learning curve because it forces you to wrap your mind and your code to Concourse's model. Probably, a lot of your build and deployment scripts would need to be rewritten. The reason why you would do so is to get the benefits described above - everything (including environments) is version controlled, ops are stateless, running code in the CI environment from a developer machine, etc.
Jobs are configured by adding an entire GitHub organization. All repositories with corresponding branches, pull requests and tags are automatically discovered and built based on the existence of a Jenkinsfile.
Everything is built by slaves using Docker, either with Dockerfile or using builder images.
Job history and artifacts are purged after a few weeks, since everything of importance is deployed to Bintray or Docker repositories.
By keeping Jenkins constrained in this fashion, we have no performance issues.
I'm just waiting for Apache to adopt it, and then it'll sit and fester like everything else in the Apache graveyard, full of vulnerabilities and slowly decaying.
Those are just Jenkins core exploits too... there are so many many more for Jenkins plugins.... https://www.cvedetails.com/vulnerability-list/vendor_id-1586...
If you're stuck on an older version of Jenkins, you better not click the "refresh" button in the plugin management page, cause otherwise the page is just filled with red warnings saying that the latest version of each plugin is incompatible or has dependencies that are incompatible with your current Jenkins version.
There is afaik no way to install the last plugin that was compatible with your version of Jenkins.
Probably not directly, but if you know the version, you can download the HPI and install it manually. Jenkin's Docker build also contains install-plugins.sh (https://github.com/jenkinsci/docker/blob/7b4153f20a61d9c579b...) that you can use to install specific version of plugin via command line.
Let me start by saying that we used to use GitLab, a lot of it was because of the CI but I didn’t have a great experience trying to manage it on top of Kubernetes, they’ve since introduced a Kube native package and I’ve been told it’s much easier, but with the deployed omnibus we ran into a lot of issues with runners randomly disconnecting, it became frustrating to the point where I had developers not wanting to use GitLab and finding interesting ways to work around it.
So I set up Jenkins on a dedicated EC2 instance with a large EBS volume for workplace storage and installed the Jenkins plugin then I wrote a Jenkins library package that exposes a function to read a gitlab yaml file and generates the appropriate stages with parallel steps that execute as pods in khbernetes - took about a week to get the things we actually used from Gitlab CI and their YAML DSL working correctly.
Now we very happily use Jenkins, mostly through YAML but in the occasions where things are much easier going directly to Groovy to use interface with plugins, developers can.
GitLab and their approach to CI (easy to use yaml) really facilitated developers writing CI, which increased our software quality overall.
I'm just getting started using it, but it seems like the solution to scaling up to a lot of Jenkins jobs. There's a good talk about it, and since you're one of only two people in the thread who used the word DSL and you are having a good experience with Jenkins, I thought I'd ask.
My config is similar except my single EC2 node is actually running Kubernetes via kubeadm, it's a single node Kubernetes cluster and has enough room for about 3 of my worker pods to execute concurrently before the rest have to wait.
(But that's just my setup and has nothing to do with Job DSL.)
For me, managing Jenkins via the helm chart has been the best part of the deal, but I'm a pretty big fan of Helm already...
The big advantage of the job DSL plugin is that if you have many similar repos, you don't just treat them all the same with multiple copies of the same Jenkinsfile, you actually can build the jobs from the same source, inheritance-style.
There could be some reasons not to do this, but if you have 100 jobs and 20 or more of them are mostly identical in form except for some parameter, it's a better strategy than multiple Jenkinsfile.
But then again if your Jenkinsfile isn't changing, it might not matter.
Which was something I had been kvetching about for months and expressly warned, multiple times to our release manager as a point of concern given how quickly plugin vulnerabilities are reported (as someone comment on elsewhere in the thread).
The day finally came when someone from one of our other offices went to update some infrastructure as code repos, poof. Jenkins server gone. They didn't have a roll back plan, and to complete the trifecta, they somehow also killed all of the instance volume backups. An entire sprint was summarily dedicated to creating a new build pipeline, I had resumes out the door the next day.
This article hits so many of our pain points I joked to a current coworker who followed me out of that place that I wanted to print it and mail it to our former RM.
This was one of those shops that hired people and gave them devops job titles, but demanded they maintain very monolithic status quos with everything from tickets and stand-ups to release management.
For me, the great thing is that it is an automation platform where we can integrate builds, scheduled tasks (cron), and very basic end-user UIs with Plugins that let me customize the whole experience.
Azure Pipelines comes close in capabilities. If they made it easier to manually start a job and for them to accept their parameters from the user, then it'd probably have everything i need.
We use Jenkins for running backup jobs, periodically updating data, and building quick little jobs for support staff to run--infrequently enough that they don't warrant adding to our admin app but frequently enough that bothering at developer adds up.
I work for Red Hat, and AWX is the upstream community project for Ansible Tower, which we provide support for. AWX is one of our newer open source projects (we open sourced it after acquiring Ansible), so you'll sometimes have better luck searching for information on "Ansible Tower".
Personally I go to great lengths to minimise the plugin dependencies and push the logic into bash scripts or Maven/Gradle. The Jenkinsfile just calls the bash scripts, so I can run it before I change it. Where I use plugins in the pipeline I document what they are and what they do in case we ever need to go from scratch.
It is hard to test but the linter picks up quite a lot of errors before push, and if you're doing a multi-branch pipeline you can just test your changes by pushing to your own branch. It's a long cycle, but the variation should go down the more you have them.
It has also been indispensable as an ad-hoc task scheduler. For example, I have a job that cleans up old branch artifacts every Saturday night. It was easy to alert on failures and see the results and history of past runs. I don't know anything else that would have fit the bill.
When you make software over 10 years old in such a fast-changing environment, the legacy of the software can weigh you down. I've worked on similar products that were tied to their legacy counterparts (sharing a database), and every change meant carrying the baggage of backwards compatibility with the legacy platform. If I could do it all again, I'd have done it the Basecamp way: Get a clean break from the legacy version, and have a migration path to the new system.
I would like to see CloudBees take all they've learned over the years, take the great work they've done on Blue Ocean and their core job scheduling system and put it all together without carrying the history with it.
Still Jenkins works well for us in our environment. Our waterfall style development processes make for ops-friendly, cookie-cutter type builds: one deployable asset per repository, one deploy target, all projects laid out the same way... Jenkins shared libraries provide powerful ways to capture the commonalities among the projects and provide a way for us to manage them centrally, while providing some (small) flexibility for customizations by the developement teams. Shared libraries also allow the ability to test changes using selected versions (branches) of the pipeline for specific builds. Looking forward to what's next (I still have a few mortgage payments to make...).
Last major exploit I heard about - the matrix.org exploit, was from privilege escalation through a Jenkins vulnerability 
This isn't nec a knock against Jenkins itself, but certainly a knock against thousands of orgs running their own unpatched Jenkins servers, often on the same machine as their other apps
But drone.io is no kind of replacement. It only works with Docker containers! Even small Jenkins installs can quickly end up with Windows targets, mobile builds, etc.
This will only be true for another week or two. We have devised a framework for alternate runtimes, including running pipelines directly on the host machine or in traditional virtual machines, that will be released end of week. Reference issue https://github.com/drone/drone/issues/2680
I really hope Gitlab's Auto-Devops team is looking at it closely and stealing all the great ideas it has because that's what our team is using.
(I work for Codefresh, a competitor of Jenkins X)
There really isn't a good self-hosted solution for build metadata/metrics, unless you write something yourself. It would be interesting to see a GUI dashboard for Drone/whatever builds and pipelines.
1. Detect broken builds. It is better to actively notify somebody by mail, chat, whatever.
2. Prevent broken master. It is better to reject the pull request/patch before it becomes a problem.
3. Analyse the system. It is better to download the data and enable the use of whatever analysis tools are suitable.
They can show non-event information such as build time trends at a glance.
I run Zuul in production with GitHub and AWS, and it's been really useful for us. It scales to the moon, parallelizes everything, and having all job/pipeline configuration in git means you can test things like CI job changes before they're committed.
Cross-repo pre-merge dependency management means I can build whole speculative futures in PR's before it lands and breaks the build.
Funny story, the Zuul build status badge is just a static image, because Zuul does not allow code to merge if it fails tests: https://zuul-ci.org/docs/zuul/user/badges.html
Full disclosure: I worked on a defunct zuul based service for a little while, and still am a core developer with lots of code in Zuul. A few presentations I've given are here:
Doesn't it only support a queue depth of 1 at each stage? Last I checked, if you have a change that is queued up waiting to get into a stage, and a newer change comes along, that newer change will supersede the old one. That makes CodePipeline only good for workflows where you only care about deploying the 'newest' of something.
I think the worst thing about CodePipeline is how hard they make it to run custom code from your pipeline. Your options are Lambda (limited to 15 minutes per run, or you have to rewrite your Lambda function to be called once every 30 seconds, essentially using the first run to start your deployment and every subsequent run to check to see if it's done yet or not) or CodePipeline Custom Actions (where you have to write the AWS SDK code for interacting with CodePipeline).
The AWS Developer Tools team could learn a thing or two from Azure Pipelines. They did it "right", IMO: you can create a 'bash' stage in your pipeline which runs whatever script you want from the build agent (which can either be hosted my Microsoft or hosted yourself). That's all I really want. CodePipeline could support that with a custom action but it's more stuff that I would have to set up.
One should not use close source solutions just because it is easy to use. Just because it is easy does not mean it is good for you in the long run.
- no hooks to manage worker scaling (https://github.com/concourse/concourse/issues/993). Our builds are _heavy_ and we'd run up an enormous AWS bill without something like (https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin)
- no way to restart a build, resorting to 'empty commits', which is a huge red flag for useability (https://github.com/concourse/concourse/issues/413)
- limited documentation/examples (network effect)
Concourse definitely feels more refined than Jenkins. Like other commenters have said, it's a steep learning curve to grasp how things move between Tasks/Jobs.
1) there are (since 1 year) restartable builds/stages in open source (a bit over a year IIRC) - when the article mentions that they are not in open source.
2) Jenkins X is something VERY DIFFERENT from Jenkins (despite the name), it is "master less", yes requires a kube cluster to power it, but it has no Jenkins instances as you know it, in a lot of ways quite a bit different as it uses Tekton as the engine for running pipelines (which has a bunch of advantages). So I wouldn't group it in with the same challenges and constraints. It is something new that shares a name and some of the ideas.
Jenkins, along with Spinnaker and Tekton and Jenkins X are now part of https://cd.foundation - worth a look to see how they are evolving (expect some changes).
This is the opposite of what teams want. Drone, Gitlab, Teamcity as mentioned in these comments is a far better approach for 99% of companies who want a solid working solution.
So maybe you are not there yet, but for a future-oriented CI/CD platform with self-hosting option, using Kubernetes as basis is a good approach.
On kubernetes - imagine if that was already managed for you, all the powerful things that can be done on top vs your self (preview apps, owasp vulnerability testing, progressive delivery - all without writing a pipeline - this is stuff that can be done.
I note gitlab is mentioned a lot here: they noted this power as well (see auto devops) and have started building things on top of kubernetes too.
I didn't mean to imply that you mix all those things together from the CDF - was just mentioning some interesting projects (some are unrelated) in the mix.
Agree also on simplicity - myself I don't like to run anything, so CodeShip is what we have for that (but it sounds like you are referring to self hosting only solutions?)
(edit: and thanks for insightful comment)
Every step in Drone is just a container, so if I want a Golang build container I can just set the step image to Golang:latest and start running build commands. And if you're wanting to encapsulate some logic into a "plugin" you can just pre-make your own docker image and from there any settings you pass in with the steps are converted to environment variables that can be accessed inside the container. Many of the default drone plugins like docker registry build, etc. Are just a couple line bash scripts on GitHub.
Also, if you ever need to schedule jobs/tasks (not just ci builds) across multiple machines, buildbot is great because all you need is a master, and slave python processes which just need a network connection to the master.
- Code builds
- Code deploys
- Git Hooks
- Parallel builds
- Matrix builds
- Build as configuration
If there were an engine that supported these features, the community could create a number of frontends that target their specific needs.
Jenkins was held back, in my opinion, for a number of years by its antiquated UI. And while Jenkins does have an API of sorts, writing a new UI for it is not a simple task.
I've used Drone for a while, and it gets a lot of these things right, but at the end of the day it is a holistic, integrated solution.
I'd like to see a composable, modular CI/CD framework.
- job config
- agents & resources
- web UI
- system config/'plugins'
- user directory with optional permission/ACL
and default ancillaries for secrets, artifacts, alerts, etc., with the idea that they could all be replaced with a purpose-built product that's actually good at it (i.e. Vault, Artifactory, PagerDuty/Sensu/whatever)
I think deployment is a separate problem space to CI and shouldn't be included
I'm not sure about the ethics of outline.com, but I can totally see why people are using it.
Anyway, good to know there is a reason for it. One of my pet projects is a self-hosted open source paywall-buster and read-it-later service.
I wonder if the HN admins would be interested in integrating with the site?
Anyway, the code is at https://github.com/lullis/nofollow. It would be great to have more people curious enough to actually run this.
We run Concourse London User Group, and the first few instances were like some kind of Jenkins survivor's support group.
Another big pain point, for me was, no obvious openstack integration. I wanted my jobs to run on ephemeral instances so all my dependencies could be defined in the job themselves, not having to rely that the Jenkins worker is setup just right.
I think Gitlab runners get this right, but I have not investigated too far.
You can write all the configuration in groovy and it will execute when the Jenkins process starts up.
There is also "Jenkins Configuration as Code" plugin which allows you to have a single yaml file to configure most of the Jenkins system.
I guess what I'm driving at is that it's not very declarative and it's cumbersome.
as Nagios is to monitoring.
Just like foot is the best way to travel and commute to work every day, as proven by the last 10000 years of people using theirs.
The tool is wrong because the tool is limited?
Bamboo or whatever it is called now is typical Atlassian crapware. Expensive as fuck, eats more resources than the stuff it builds, and did I mention it is yet another half assed product that got shoddily integrated into the usual Atlassian lineup?
Gitlab CI is great for anything that is code (think build, test, deploy), but it is not suited for abstracting "non-development" jobs which can perfectly be automated in Jenkins (e.g. creation of a dev environment with fresh data from production). Plus it is Docker and the runners are polling - which means at minimum 10s startup time compared to milliseconds for a Jenkins shellscript job running on a ssh connected slave!
Github and friends are cloud which is a big no-no. We're placing too much power in the hands of AWS, GCE and Azure already, no way in hell it is a good idea to put private source code to a cloud provider.
If all you are doing is using jenkins to run simple bash scripts, you may be able to get away with it. The problems start when you want to add some logic to the pipeline – which you are doing, otherwise why bother with a pipeline?
First things first: are you going to use the scripting pipeline, or the declarative pipeline? The declarative pipeline is a bit better, but it lacks examples, has lots of bugs (I've littered my code with references to JENKINS-XXXXX) and is very restrictive (arguably by design). Of course, you can have 'script' blocks inside your pipeline->stages->stage->steps blocks.
Then you want to take advantage of parallelization or conditional steps, and to visualize that you want Blue Ocean. Problem is, not all plugins are compatible with Blue Ocean, it also doesn't have all features, so you drop down to 'old' jenkins often.
People will want to have a whole bunch of tools with incompatible versions in their builders. Not all are supported natively, so you need to figure out your versioning.
Once you figure all that, congratulations. Next guy to automate something will either find a similar pipeline to copy from, or will endure all the pain again. At this point you may want to use Groovy.
Groovy was absolutely the wrong tool for the job. Yes, I get it that it works with Java, which Jenkins is based on. Still it is the wrong choice. You see, the kind of things you want to automate often involve passing commands around, be them bash, ansible, SQL statements, what have you. Groovy's string escaping rules will ensure your life will be pretty miserable (https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e412...)
You could get around most of these by perhaps moving most of the logic to containers and then running those. There again you'll run into problems with declarative pipelines, random things won't work and you'll be scratching your head.
However, if you are going to do that anyway, you're better off using a more modern system for CI, any system. Drone was already mentioned, there's also Concourse and a bunch of others. For CD, you can use Spinnaker as well.
Or maybe keep jenkins around but forget all the fancy stuff. Delegate all the 'thinking' to scripts and pretend the more recent development has never happened. You'll be saner that way.
Or do you mean systemic corporate problems? In that case, I agree.
It still doesn't change the fact that Jenkins does not make my job any easier. I'll spend a day worrying about Jenkins idiosyncrasies ("why can't I use a pipe in sh", "why did my bash escaping disappear completely", "why 'dir' doesn't work with a container build agent?! (JENKINS-33510)", "why this input plugin won't work with blue ocean", "why can't I use a for loop in this piece of code in particular but it works elsewhere" (JENKINS-27421)).
Whereas with concourse or other newer build systems I can write a simple YAML description, which is modular and uses an existing standard, and test that in isolation. And then provide it as a building block for other tasks.
Welcome to the world of enterprise Java or .net programming. Loads upon loads of crap. Best served with multiple frontends (e.g. web + mobile) which need different npm versions to compile and all of it out of a single fucking pom.xml which is a nightmare in itself!
You have an excellent point. Individual microservice containers are not complicated (then again, all they do is call a standardized script). The script will run a Dockerfile and push it to the registry. I would classify it as a 'trivial' Jenkins job, not even pipelines are used.
The pain starts when you want to do more than CI and try to get into CD. Or even worse, automate 'devops' tasks. That's where you run into all those warts.
A job could call Terraform, or spin up VMs, or run vacuum on a database, or any number of tasks. Or it may perform tasks on K8s to deploy a complex app. It may need to call APIs to figure out where to run things. And so on.
Since Jenkins is not only a CI/CD system, it can do anything, so people will try to make it do increasingly complicated stuff. And I'm arguing that this is wrong. If you have complex logic, it should be moved out of Jenkins so it can be more easily maintained and tested, and dependencies isolated. One of the easiest ways to do that is with containers. At which point, Jenkins loses most of its usefulness and other, newer tools shine.
Alternatively, use more specialized tools. If it is for CD, and Spinnaker works for you, please use that instead.
However, deploying containers to environments like openshift and kubernetes is extremely simple with jenkins. I don't think that's complicated at all. As a rule of thumb, you should be able to hide all the complexity in your deployment in the dockerfile. In addition, you can always use jenkins "build with container" functionality to build your application in a dedicated container on the fly. Many ways to hide complexity with jenkins.
I do agree with you that jenkins is abused because it is more than a CI/CD tool. I think that you need some experience using it to know what works well and what doesnt. Unfortunately in the new age "sprint agile" world some random guy has to pingeonhole crap into jenkins in 2 week time windows that shouldnt be there in the first place.
I also think that many devs underestimate what you can do running local jenkins as a tar file on your macbook. I like using jenkins to automate tedious tasks for myself. As an example, it is trivial to write yourself a custom github code scanner that will scan all files and folders in as many repos as you want. I like using jenkins for outside the box things like that.
Um, so doesn't his workflow buy an essentially infinite maintenance problem with all his containers? This sounds like tomorrow's Node left-pad debacle in a different form.
- You probably want to store artifacts in its object store anyway.
- Rent exactly the right amount of compute for the concurrent builds running in any given minute.
- If your projects have sufficiently simple build-and-test entrypoints i.e. "make" then you're highly portable and not too worried about lock-in. You don't even need to be deploying to the cloud provider you're building on.