
Show HN: Vespene – My new Python CI/CD and automation server written in Django - mpdehaan2
http://docs.vespene.io
======
markovbot
> Unlike some other CI tools, pipelines in Vespene are easily and graphically
> configured, and there is no custom DSL (“Domain Specific Language”) to learn
> and debug.

Does this mean that the CI config cannot be checked into some sort of version
control? That was one of my biggest issues with Jenkins. Someone would change
something somewhere and suddenly my builds would fail, and even if I track
down what setting is causing the problem, I have no idea why it was changed,
who changed it, what will happen if I put it back, or even what it's previous
value is.

Additionally, it made duplicating a given workflow an extremely tedious,
manual process.

~~~
pytester
Second this. I really would rather have something I can configure fully with
YAML or JSON.

~~~
ergo14
I think [https://concourse-ci.org/](https://concourse-ci.org/) is something
you want to look at then.

~~~
Aeolun
Concourse has a really weird resources philosophy that makes it almost
impossible to easily work with.

~~~
jacques_chester
I'm a notorious Concourse fan, I'd be interested to understand why you think
it's almost impossible to work with.

------
peterwwillis
> When organizations start to have too many projects, they need an easy way to
> share values and code snippets between those projects. To support this we
> have Jinja2 templating of build scripts

I would implore everyone implementing CI/CD to do the opposite. If you have a
small project, maybe use a build script to get it off the ground quickly. If
you have a large organization with complex, disparate tech projects, do not
use build scripts.

Use CM or orchestration tools. Use pre-rolled containers/images. Use
configuration options for existing solutions/plugins/deployment tools. Write
only open source, composable, extensions/modules/plugins, with yaml/json/xml
configs to control them.

The open source bit is actually crucial. By making all your build/test/deploy
plugins open source, you force yourself not to include proprietary
information, which is always a dark pattern. Hard-coding IPs/hostnames,
credentials, product names, regions, etc just makes your work less
composable/reusable. Open source all your pipeline code except for your
configs. Then everyone in your business can find everyone else's work simply
by going to GitHub and looking up a tool. Because they won't be scripts, they
can be picked up and used immediately by any team, and not have to be forked
by every team that needs a modified build script.

To do all this properly requires discipline and training and research and
time, which big companies have. The point is not to be fast, but to be
reliable, to prevent muda from sapping your productivity.

~~~
mpdehaan2
Compiling C code with puppet or whatever seems a little weird to me, to be
honest :)

If you want to keep your data externally, there are some good options for
this. I'd read the plugin docs about variable plugins, that can easily source
variables for things like etcd or Consul.

A script for a given repo could be _nothing_ more than "make go", and that
actually is a pretty good practice, to keep that in source control.

Still, you need something to build your code, and to have a good place to see
the status.

Ultimately, code still needs to be built, and orgs do want to avoid images
they cannot easily recreate. That's the role of a build system and making sure
the _process_ to create those artifacts is in source control.

I should also mention that your build script CAN be sourced from source
control.

In your .vespene file, just say "script: <path>.

However, the variables in that file can be evaluated by Jinja2 variables, so
for instance your security team could set up a snippet everybody could use or
your feature flags could be defined from there.

Another cool feature of Vespene is in each project build root all those
variables appear as vespene.json files, so it can be a good tool to use to
launch all kinds of automation from a web console where you want to record
results.

------
mpdehaan2
Hey folks, this is my new project. You might possibly remember me from
creating Cobbler and Ansible.

I thought I'd share and if you have any questions, let me know!

~~~
df24g3t36
Id strongly recommend adding the ability to implement pipelines as code -- ie:
you check in the CI/CD configuration into the repo (or another repo,
containing CI/CD configurations). Having to configure the pipelines manually
is nasty, and often builds are caused by misconfigured of the pipeline. having
this checked it allows easy versioning.

~~~
PopeDotNinja
I also like the idea of pipelines as code. I'd love to see it go a step
further and become pipelines as testable code. I've yet to see a pipeline of
non-trivial complexity that stays easy to reason about. I predict the next
wave of CI/CD (or pipeline anything) will make pipeline testing & sanity
checking simple as heck. If this already exists, please tell me about it,
because I don't see people using it ;)

------
ianamartin
This looks really great to me, and I can't wait to try it. I've been using
buildbot for extremely light-weight CI/CD server that basically just triggers
Ansible scripts on git changes. Not at all the way buildbot was designed to
work, but very quick and easy to manage.

This looks like a tool that could be a lot closer to what I want to do and
designed to do it that way.

Also, is the repo being moved somewhere?
[https://github.com/vespene/vespene](https://github.com/vespene/vespene) is
404.

~~~
cbcoutinho
That's someones personal repo - you're looking for this one:

[https://github.com/vespene-io/vespene](https://github.com/vespene-io/vespene)

~~~
morenoh149
this is what's linked to from the docs so I think it's official

~~~
mpdehaan2
[https://github.com/vespene-io/vespene](https://github.com/vespene-io/vespene)
is the one!

------
p1necone
Good thing you didn't write this on top of Etherium instead. I suspect you
would quickly run into a situation where you... needed more Vespene gas.

------
cjbprime
Does every machine running a worker have to be a full node with Django?
Workers in a complex CI setup can often be e.g. raspberry pi machines, iOS
emulators, Windows machines without admin access deep inside a firewall that
make outgoing connections but not incoming, etc.

So, you might consider a protocol for builders to report their results (e.g.
over SSH) without being a full node, if that doesn't exist. Looks good!

~~~
mpdehaan2
Hey!

Any node _typically_ would run the webserver because it's not that heavy, but
doesn't technically have to. It could just run one worker process. Right now
the workers do need database access and I suspect that will stay that way for
the short term.

Windows needs to happen, but I'm not sure when it happens. (Also a good list
discussion).

I'm a bit curious about your raspberry pi use case - maybe a good topic for
the forum? [http://talk.vespene.io/](http://talk.vespene.io/)

------
caleblloyd
How do you manage normalizing webhook events from each source control
provider, i.e. GitHub, GitLab, BitBucket, etc?

I have been interested in building a general purpose CI/CD tool but this part
seems like the most "grunt work". Is there an open source library that
standardizes webhooks? Or would you be interested in publishing that portion
in a library?

~~~
dsumenkovic
Hey Caleb, if you are interested in using GitLab webhooks, this documentation
may help you
[https://docs.gitlab.com/ee/user/project/integrations/webhook...](https://docs.gitlab.com/ee/user/project/integrations/webhooks.html).

------
michaelmcmillan
Cool project! It strikes me that Django might not be the ideal for this
project, have you thought about that? This logic [1] looks shoehorned into the
framework.

[1] [https://github.com/vespene-
io/vespene/blob/master/vespene/mo...](https://github.com/vespene-
io/vespene/blob/master/vespene/models/pipeline.py#L82)

~~~
mpdehaan2
While you're welcome to that opinion, Django has been freaking awesome. This
whole thing got written in 3 months, not even working full time. I wrote
pipelines in like a _day_.

It seems like you're saying I should have used a many-to-many there rather
than 7 FKs. Probably, but shrug... it can be fixed later if it ever becomes a
problem.

~~~
michaelmcmillan
I would expect it has been easy, I’m talking about the way forward. I wish you
the best of luck anyway!

------
ensignavenger
This looks cool. Unfortunately it is Common Clause licensed and therefore not
Open Source. My organization would never consider a Commons Clause licensed
tool.

~~~
mpdehaan2
I'd argue the definition of what is open source is up for debate, but I had
some reasons for this and thought about it for weeks.

I hope people know my history and how I've run projects in the past.

But what this does for me is keep me from releasing anything open core. With
ansible, ansible was open source but the GUI was _not_ , and we had to hold
something back.

The traditional alternatives to OSS are to make a support business, which
somewhat encourages making buggy or hard to install software, or a consulting
business (IMHO, same) and all of those detract from building a product,
collectively, with some of your favorite people on the internet.

So basically Vespene is going to be like a pseudo-foundation.

The structure for this is described here -
[http://docs.vespene.io/partnership.html](http://docs.vespene.io/partnership.html)
\- which keeps consulting and support totally free for small shops, and
encouraging larger ones that could potentially make millions form Vespene to
give back a very small amount.

What normally happens is large companies close six figure contracts supporting
open source, and they don't give back a dime, and my goal here is to
essentially make a developer's salary off of this project by having those who
make _more_ give back a very small amount.

This seems pretty fair to me, but I also recognize that some people don't
agree with it.

That all being said, it's not a "no-commercial" clause in any way, and still
free for small consultancies, so I'm not anticipating any major problems.

Most of the time, software gets installed in a place of business _for_ that
business.

For some of my previous thoughts on this, I'd refer folks to
[https://medium.com/@michaeldehaan/why-open-source-needs-
new-...](https://medium.com/@michaeldehaan/why-open-source-needs-new-
licenses-d2d9d819a10) and later my current thinking
[https://medium.com/@michaeldehaan/going-with-the-commons-
cla...](https://medium.com/@michaeldehaan/going-with-the-commons-
clause-1bdab4c15e5d)

It's not something I've considered lightly, but this keeps MORE software open,
and for a small one-person shop, I think that's more noble than trying to hold
some code back and not release it at all.

Some people just want 100% free, gimme gimme, etc - and fine - you're entitled
to pick one of those things. That's ok.

I personally view this a little more pragmatically, under the view of
fairness, and also have taken steps (in the CLA, etc), to guarantee that trust
is never going to be abused.

For instance, the license reverts to pure Apache if Vespene were ever to
change hands.

On the downside, I can't get screwed over by IBM or Amazon while I'm trying to
crank out free software for everyone. That seems fair too, seeing the time
investment running an open source community and working on the code takes.

~~~
Sir_Cmpwn
You have a lot to say here, but I'd like to respond to just this:

>I'd argue the definition of what is open source is up for debate

It's not up for debate. It hasn't been up for debate for years. There are two
generally accepted definitions of FOSS (each addressing the F and OS
respectively):

[https://www.gnu.org/philosophy/free-
sw.en.html](https://www.gnu.org/philosophy/free-sw.en.html)

[https://opensource.org/osd](https://opensource.org/osd)

 _No one_ disagrees with these who isn't trying to sow discontent in the open
source community. Disagreeing with the OSD or FSD is akin to denying climate
change.

~~~
Nacraile
> No one disagrees with these who isn't trying to sow discontent in the open
> source community.

No. Everybody who disagrees with these has been driven from your ideological
bubble. "Open source" has become a general term that is part of the natural
language; nobody has the right to demand everybody use their preferred
definition. Many people who use the term will never have visited gnu.org or
opensource.org. Many others will have, but will have concluded that the
strictness of those definitions is silly. It seems quite reasonable to
disagree with a definition of "open source" that makes it practically
impossible to make a living off of your work. If you want to have a debate on
the merits, fine, but running around telling people they're using English
wrong because they disagree with you is just foolish.

~~~
Sir_Cmpwn
No, it hasn't. This is a lie which serves the interests of those who would
harm open source. This language is important. You can't take the argument of
"every word means what I want it to". The burden is on you to be understood,
and by misusing terminology like "open source" you are either ignorant or
deceitful. I wouldn't expand the definition of decongestant to include
adderall because the former sells better and I'm a pharmacy that needs to get
rid of my stock.

You might disagree with the principles of the definition, or think that the
definition isn't useful. But that doesn't make it any less of the definition.
If you want to do something different, you need to call it something else or
you are lying to people.

~~~
Nacraile
> You might disagree with the principles of the definition, or think that the
> definition isn't useful. But that doesn't make it any less of the
> definition. If you want to do something different, you need to call it
> something else or you are lying to people.

This argument can be applied to your definition just as well as it can to my
definition. Which underlines the pointlessness of arguing over the definition
of "open source", and therefore the pointlessness of telling somebody that
their thing is "not open source".

~~~
mkesper
If you follow your logic everything becomes void.

~~~
Nacraile
Not so. I'm just saying that it's valid to disagree on the definition of a
thing, and that when that happens you need to have debate about the real
underlying disagreement, rather than making a pointless and foolish
declaration of your correctness by your definition.

Although I'm not sure why I bothered typing that, given the current readings
I'm getting on my troll detector.

------
cagenut
I like how ansible was really func 2 and now vespene is _the real_ ansible 2

~~~
mpdehaan2
ha, totally not going to write another CM layer. That is one of the fatal
blunders, up there with fighting a land war in Asia and all that.

------
bribri
needs declarative syntax

~~~
mpdehaan2
There's a fair amount of that available in
[http://docs.vespene.io/importing.html](http://docs.vespene.io/importing.html)
... if you have more ideas can you stop by
[https://talk.vespene.io/c/ideas](https://talk.vespene.io/c/ideas) ? ... I
don't want to make the syntax more complex than the YAML that is there now,
but I think we can add other fields if there is something you want or a
capability that might be missing.

