
Ask HN: Is the CI space overcrowded? - Scorpiion
Hosted CI is a hot space, and fairly recently the git hosting providers (Bitbucket and Gitlab) also entered the game with their own hosted services. Some hosting providers also have some overlap with CI functionality or specific CI services on their own. I wonder what people in the HN community thinks about the CI space as of today.<p>1. Do you think it&#x27;s overcrowded? Is there space for new companies or is CI basically a &quot;solved problem&quot;? If not what do you dislike with current offerings?<p>2. Do you prefer a third party CI as compared to the hosting provider offering their own CI at a lower price? Assuming here that the hosting provider would offer lower price because their main income is from the 24&#x2F;7 hosting fees. Or would you simply switch to the hosting providers CI if it was good?<p>Disclosure: I am the founder Sourcevoid, a cloud&#x2F;PaaS company. I&#x27;m a little split on how much CI functionality we should integrate into our platform, compared to just trying to promote third party CI providers instead. We already do a fair amount via our build system, but at the same time CI is more than just the build process of course. This question is not about our company or our offerings, I just think it&#x27;s an interesting question to discuss.
======
lomnakkus
I find that it's just like the bug tracker space: There's a _lot_ of
contenders, but _somehow_ none of them actually hits the sweet spot.

For CI, it's either: "too GUI" or "not enough programmable".

Also, CI is an area that absolutely cries out for container technology, but
the state of containers on Linux[1] is absolutely abysmal. Maybe it'll be
better in ~5 years when we'll hopefully be about 40% towards the capabilities
with Solaris Zones. (Here's a hint: If you cannot fully 'contain' root and
users/mounts/devices, then you're not being serious.)

[1] The most popular platform for this, by a far margin.

~~~
a-saleh
Which camp of "too GUI" or "not enough programmable" does jenkins fall in?

How would the sweet-spot look like?

~~~
lomnakkus
"not enough programmable", but I'll add the caveat that I haven't really
caught up with the 2.x bits that I _think_ could help quite a bit.

(However, from the documentation I got the impression that the new 2.x Jenkins
bits were quite repository-centric. It turns out that what I want is really to
define the whole CI infrastructure _separately_ from my repos with a few rules
to generate appropriate jobs, etc. Perhaps the solution to that is to just
define a repo for all-the-builds? Still need to look into it.)

~~~
a-saleh
I think currently you are supposed to use the "Multi-branch pipeline" for
that, where you have separate branch in your repo for every configuration/job
you would need.

We have decided to go different route, even though it is kinda held together
by duct-tape :)

1\. We have several shell-scripts that use the $JENKINS_URL/script to update
global configuration. A.f.a.i.k. the /script endpoint gives you access to
almost all of the jenkins internals to be operated with groovy. So thats how
we set-up i.e. plugins, slave-providers, secrets, shared config-files, e.t.c

2\. for job definitions, we use jenkins-job-builder [1] with the pipeline
plugin [2] We then store both the job-builder configs and the jenkins-files
they point to in a single repo, achieving the "define a repo for all-the-
builds" solution.

3\. to reduce repetition we used shared pipeline library, where we put all of
(groovy) functions to be shared across jobs.

Neither shell-scripts for /script endpoint or the job-builder configs are
particularly nice, but they get the job done. But we used them even before
jenkins-files/pipelines/2.x

The 2.x bits do help quite a lot :-)

[1] [https://docs.openstack.org/infra/jenkins-job-
builder/](https://docs.openstack.org/infra/jenkins-job-builder/) [2]
[https://github.com/rusty-dev/jenkins-job-builder-
pipeline](https://github.com/rusty-dev/jenkins-job-builder-pipeline)

~~~
vorg
Would be nice if we script those Jenkins pipelines with some other languages
besides Apache Groovy. And why are Groovy's functional methods inhibited when
used within Jenkins?

~~~
a-saleh
The reason why you can't use functional methods of Groovy in jenkins is, that
the intepreter that runs inside of jenkins has been rebuild so that jenkins
can suspend/resume the execution of the pipeline.

This means that everything needs to be serializable, and that in reality you
are not really executing groovy statements, but rather passing continuations.

This is a reason why you have @NonCPS annotiation, that allows you to opt-out
of that behavior.

~~~
lomnakkus
FWIW at this point, this kind of demonstrates my point: I think I kind of want
a _library_ to _build_ a CI system. Not some piece of software I have to run
somewhere (as root, probably), which _may_ enable me do to what I want.

Why hasn't anyone developed a CI-orcherstration library yet? Is it actually a
difficult problem?

------
shubhamjain
_Nathan: You’ve shown with Freckle that you can enter a saturated market like
time tracking and still do well._

Amy: “Saturation” is a load of bullshit :)

 _Nathan: Really it just shows the market exists._

Amy: It’s more than that. So much more than that. If a million people use
Harvest, there’s no way they’re all served well by the same tool. The presence
of other products doesn’t just show opportunity, it CREATES opportunity.
Because wherever there’s a big biz, there will be lots of dissatisfied
customers.

Source: [https://stackingthebricks.com/difficulties-for-nathan-
barrys...](https://stackingthebricks.com/difficulties-for-nathan-barrys-app-
experiment/?_r=uf)

------
debacle
If you could make a self-hosted Jenkins replacement that is even half as good
as Jenkins (think extensions) but written in not Java, I think it'd receive a
lot of positive reception.

The trick with CI, like task/project management software, is that people don't
look at what works. They don't try and compete on intrinsics like stablility,
extensibility, etc, and really after 12 months on any platform, any mid-large
size company is going to care more about those things than new shiny.

Jenkins had the absolute worst UI for a long time, and it still was many
people's top choice for CI. You're generally writing a tool for programmers,
after all.

------
twobyfour
It's a well-populated space. If you're going to enter it, you need to be clear
about differentiation.

We picked our CI solution because it was the only cloud offering we could find
that supported Bitbucket and also allowed us to set up our own fully-custom
environments to run against - without Docker.

Sadly, it took about a week's worth of research to figure that out. And I'm a
bit concerned that in this crowded market they'll go out of business and leave
us high and dry.

~~~
rjbwork
This sounds a lot like my requirements...can you tell me your CI provider?

~~~
twobyfour
[https://buildkite.com](https://buildkite.com)

We migrated there from Bamboo Cloud when Atlassian discontinued it. It's
nearly a direct drop-in replacement, and they even had docs specifically about
Bamboo migrations. Their customer support during onboarding was also superb,
and they've been responsive to feature requests. We've been using it for
almost a year now, and overall we're pretty pleased.

What are you using currently?

~~~
rjbwork
I run an Azure VM hosted TeamCity/Octopus Deploy combo, with bitbucket for
SCM.

~~~
twobyfour
Are you happy with that solution?

~~~
rjbwork
It's alright, I rarely have to touch it, but I'm no Sysadmin. If I had
ops/sysadmin guys to shove onto, I would have no qualms using it long into the
future.

I would, ideally, like a hosted build server, and a hosted Octopus - I really
like it. Not the biggest fan of team city, but it works, and it's the only
thing I know well.

~~~
twobyfour
Yeah, that's why we wanted a cloud solution - none of us really have
significant sysadmin expertise, nor the time to devote to tending more boxes
than we absolutely must. Only having to maintain the actual build environment
has been a win.

We've had a couple occasions where we ran out of disk space (though not since
we increased the disk size and added a cleanup job), and a few times a build
agent has randomly stopped responding and needed to be restarted, but nothing
we couldn't easily diagnose and fix.

------
twunde
I think that as evidenced by the discussion about concourse ci today
[https://news.ycombinator.com/item?id=14785254](https://news.ycombinator.com/item?id=14785254)
that there is still room for improvement and new products, especially on-prem
CI tools like Jenkins/TeamCity/Bamboo/Concourse.

~~~
jetblackio
One thing that I've always wanted was for Jenkins to get away from being a UI,
and just be an engine with a nicely documented API. People could then build
more niche CI/CD systems on top of that engine without having to reinvent the
wheel. It's nice that Jenkins is finally getting a UI facelift though.

