Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Updatecli – What if Dependabot and Ansible had a child?
60 points by olblak on Feb 10, 2022 | hide | past | favorite | 33 comments
What if Dependabot and Ansible had a child? Well for me that could be Updatecli.

Updatecli is a project that I started to help maintain the infrastructure of the Jenkins project. I needed something flexible enough to update YAML with whatever information needed. Because let’s say it, everybody loves YAML. YAML is everywhere.

Run it from everywhere…

Updatecli is a command-line tool written in Golang and available for Windows, Linux, MacOSx, amd64, arm64, thank you Goreleaser All of that to say that it runs from wherever CI or laptop we need.

As of today, Updatecli opened over 3000 Pull requests on Github, and it evolved to update automatically Dockerfile, Markdown, Helm Chart, and of course a lot of YAML for tools like Puppet, Kubernetes, or Jenkins.

How does it work?

Updatecli loads pipeline configurations from YAML(s) or Golang templates then enforce the state defined by the pipeline configuration. A pipeline run as followed:

1. Clone in a temporary location any git repositories used by the pipeline.

2. Fetch information for every *source* defined, and then inject them as entry parameters into condition(s) and target(s).

3. Test that all *conditions* defined succeed otherwise abort the pipeline.

4. Enforce the state for every *target* defined. A state means different things depending on the resource type, more on this later.

5. Commit and open pull requests when needed.

6. Apply next pipeline

A Updatecli pipeline relies on resources aka “extension” aka “plugins” to adapt pipeline behavior. By combining them, we can easily automate scenarios for release workflow, GitOps, dependency management, documentation update, etc.

A simple scenario could be: * Retrieve the latest Golang version * Test that a docker image with the latest Golang version exist on Dockerhub * If it exists, then bump the version in a YAML file and open a pull request on GitHub with the change

As of today, there are 9 extensions for "sources", 8 for "conditions", 6 for "targets", 2 for git repositories, and 1 for pull requests.

A very simple pipeline is available on -> https://www.updatecli.io/docs/prologue/quick-start/

For more complex pipelines, you can look for directories named “updatecli/updatecli.d” at the root of repositories on https://github.com/updatecli or the Jenkins infrastructure repository such as https://github.com/jenkins-infra/kubernetes-management/tree/...

I maintain a documentation website to document the different configuration. It’s not perfect but it’s available on www.updatecli.io

What’s next?

Well, it depends on many things. Updatecli is since the beginning, a fun side project, I wanted to practice Golang programming while automating tedious recurring tasks. I built it in a way that I could reuse it across the different projects which I maintain. It’s rather simple to add new resources so I’ll keep adding them based on my needs, I welcome any contributions that would benefit the community.

More information on https://www.updatecli.io https://github.com/updatecli/updatecli




This looks really cool. How dos it compare to say [renovate?](https://github.com/renovatebot/renovate)

I look forward to trying this with gitea to automatically create pull requests to update ansible plays with the latest version of software.


Updatecli focus on dependencies which can't be easily identified based on the context. Hence the need to describe how to retrieve an information and then what to do after with it.

And then once you have the latest version, you also have the challenge to know how to apply that update.

More I dig in the dependency management topic, more I realize that we still have long way to go.


I wrote something similar but to keep a local system up-to-date: https://github.com/kdeldycke/meta-package-manager#readme

With Meta Package Manager you can aggregate the results of all package managers available on your machine:

    $ mpm outdated
    ╭──────────────┬─────────────┬─────────┬───────────────────┬────────────────╮
    │ Package name │ ID          │ Manager │ Installed version │ Latest version │
    ├──────────────┼─────────────┼─────────┼───────────────────┼────────────────┤
    │ curl         │ curl        │ brew    │ 7.79.1            │ 7.79.1_1       │
    │ git          │ git         │ brew    │ 2.33.0            │ 2.33.0_1       │
    │ openssl@1.1  │ openssl@1.1 │ brew    │ 1.1.1l            │ 1.1.1l_1       │
    │ rake         │ rake        │ gem     │ 13.0.3            │ 13.0.6         │
    │ Telegram     │ 747648890   │ mas     │ 8.1               │ 8.1.3          │
    │ npm          │ npm@8.0.0   │ npm     │ 7.24.0            │ 8.0.0          │
    │ pip          │ pip         │ pip     │ 21.2.4            │ 21.3           │
    │ regex        │ regex       │ pip     │ 2021.9.30         │ 2021.10.8      │
    ╰──────────────┴─────────────┴─────────┴───────────────────┴────────────────╯
    8 packages total (brew: 3, pip: 2, gem: 1, mas: 1, npm: 1, apm: 0, cask: 0, composer: 0).
Bonus point: it solves XKCD #1654 (Universal Install Script).

(edit: formatting)


Impressive project, nice work! I'd use the `outdated` command but even just the basic listing of installed packages across all managers looks useful to me.


Might be worth a "Show HN" of its own...


mac problems.... =) rolling distro's for the win.


I like the idea of automatic proposed changes triggered by external events, but I'm not sure a monolithic tool with plugins is the best way of going about it. The Unix way to solve this problem would be to decompose your program into a couple of different programs that just do one thing, with flexible inputs and outputs.

"Sources": what does this do? Collect information. This doesn't need to be a particular program, this could just be a data specification that any program could generate data according to the spec. Thus you could have a dozen different programs that could each generate the spec.

"Conditions": This also doesn't need to be a dedicated program. Any program that can read the data specification can have one or two lines of simple logic; a couple lines of shell script would work fine. For complicated conditions you could use a more complex language, or build applications with specific features that are needed often.

"Targets": This part seems a little complicated, could use deconstructing a bit more. If you want to modify a YAML file, that is one dedicated kind of action to take. If you want to commit that file to a Git repo, that's another action. Opening a PR is another action (PRs aren't a Git primitive). Those three things could be separated but run one after another, all as separate programs.

The plugin aspect is certainly useful for being able to share pre-made solutions. But those "plugins" don't need to be strongly coupled to a monolithic application. Drone.io is a fantastic example of a loosely-coupled cloud-native design; every plugin is actually a Docker container, which is just an abstraction around arbitrary applications that uses environment variables and command-line options. (Also, the data specs absolutely needs a version number, and should consider backwards compatibility)


> I like the idea of automatic proposed changes triggered by external events, but I'm not sure a monolithic tool with plugins is the best way of going about it. The Unix way to solve this problem would be to decompose your program into a couple of different programs that just do one thing, with flexible inputs and outputs.

Waow, thanks a lot for you feedback, especially to take the time to write this down. I'll do my best to answer as you cover different topics which have been scratching my head since the beginning of the project. But first keep in mind that I had a strong constrain which was my spare time :).

It's definitely something which I am planning to improve.

Currently everything is done at build time and rely on Golang interface. So a resource kind need to match a golang package. A package needs to implement the correct function for it to work. for example for a "source" you need to implement the "source" function as in the following example.

``` func (m *Maven) Source(workingDir string) (string, error) { // Implement whatever you need and return the source value return "the source value", "an error if needed" } ``` Like in this package https://github.com/updatecli/updatecli/tree/main/pkg/plugins...

And the matching between a package and a resource kind is done here https://github.com/updatecli/updatecli/blob/main/pkg/core/pi...

Ideally I would like it to be done at runtime similar to Terraform.

If needed you can still use the "shell" target but then you put a strong dependency on the tool used within your script. Let say that your script call a python tool, then you need to be sure that both your local and ci environment have the same python version. I personally consider it as a fallback when I don't have the time to implement something better.

Ideally I would like to be able to use resources which are very specific to environment and therefor not suitable for everybody.


this tool looks amazing. dependabot but open-source, with a plugin system, and you can run it locally. great idea, looking forward to trying it out!


agreed, FYI dependabot is open source for non-commercial use and you can run it locally

https://github.com/dependabot/dependabot-core/blob/main/LICE...


"Open Source, but not for X" is not open source, it's source available.


Thank you very much. Major difference with dependabot is that you can use it for more use cases as you define what, how, and when something should be updated


interesting. so if i understand this correctly, it's like a bridge between all those little version numbers and hashes in config files for modern infrastructure and full scale automation (from running of scripts to provisioning a short lived test environment) for checking if those little numbers and hashes need to be bumped. cool!


Kind of, the reason why it's a bit difficult to describe the project is because the different component mean different things to different people.

I built updatecli so I could use it to help me maintain infrastructure as code, then bit by bit, I discovered how other people were using it to solve challenges that I didn't imagine.

Scenario 1: https://github.com/jenkins-infra/kubernetes-management/blob/...

We monitor the latest Certmanager chart version available and we bump our git repository so Helmfile can pick the latest version and update our Kubernetes clusters. The purpose is to automate infrastructure as code.

Scenario 2: https://github.com/updatecli/website/blob/master/updatecli/u... We monitor the latest Updatecli version so we can update documentation on the website The purpose is to keep documentation up to date

Scenario 3: https://github.com/epinio/helm-charts/blob/main/updatecli/up... We monitor our application so we can automatically release a new version of a helm chart The purpose is to automate the release process.


I had the need for such a tool a couple of months ago and wanted to build a dedicated tool. And you did it ! Thanks !


Thanks, Feel free to open GitHub issue on https://github.com/updatecli/updatecli/issues was what you were looking for. That would be very useful


This does sound interesting. Regarding the idea of constantly updating dependencies automatically, I've heard of Dependabot, and haven't used it much myself, but isn't there a danger with an automatic update breaking things in production without supervision?

I'd guess I'm missing something in that question?


Depends on your risk tolerance. At a bare minimum you should have tests as part of your project and CI for detecting breaking changes from version bumps, with validating deploys to dev environments, and possibly human review of changes before deploys to prod.


Another thing to consider is malicious packages being published. In the Node ecosystem, the last few times this has happened, the community reacts quickly with a speedy (<12h) un-publish, and the whole incident might be over before your development team wakes up in the morning. However, if you immediately and automatically build every single dependency upgrade on your CI systems, you are dramatically increasing your exposure.

It's important to ensure your CI systems do not hold any important credentials, and are decoupled from your deployments. In a setup like this, at best, your source code may be stolen by one of these malicious packages. At worst, it will scrape for goodies such as `AWS_SECRET_ACCESS_KEY`.


Yes, an automatic update can break things. Personally, I am happy to have minor version updates be applied automatically if my test suite passes. For anything larger, I at least review the changelog to make sure there aren't any obvious breaking changes and then if the tests pass, I go ahead and deploy.


You need very good tests in place to automatically change something or a lot of trust in your dependencies :P.

It's not the case on most of my project, so I never commit directly to the main branch, instead I commit on a temporary branch then open a PR on GitHub so we just review and merge the changes.


May I suggest creating a JSON Schema for the yaml files, so reasonable editors can help complete the structure?

I didn't study the docs to know how much this is on the "ansible" side of things, where conditions can be almost anything, and thus describing possible legal values would be some monster work


Thank you for the feedback. So If I understand you correctly, you suggest to improve the feedback loop when writing updatecli configuration from an IDE.


I like how some are missing the point of the tool because you mentioned (assuming jokingly) that we all love YAML.

Anyway, looks really cool. Dependabot seems to some strange limitations, and I'll probably look into using this.

Out of curiousity, how are you linked to the Jenkins project? Are you just an OSS contributor or do you do this full time?


I worked for 5 years at CloudBees within the community team so maintaining the infrastructure project was one the things I worked on https://GitHub.com/jenkins-infra


> Because let’s say it, everybody loves YAML. YAML is everywhere.

Nope. Nope, nope, nope.

I know a lot of people, including myself, who hates YAML with a passion.

First, they hate the format itself. It's very easy to introduce hard-to-find typos. Plus it has multiple versions, no way to tell which one you use, and depending of the parser that ends up eating the file, you may get weird results. It's way worse then just the "Norway problem". Typically, a bad monday morning could look like this:

    >>> import yaml
    >>> yaml.safe_load(''' 
        port_mappings: 3333:33
        alternative_port_mapping: 3333:33,
        countries: [FR, US, UK, NO, IT] 
        1 : 1,0
        2: 01
        3: 1.0
        4: 1O
        5: 0b1
        6: 0x1
        7: 0i1
        8: hey,
        8.0: oh,
        version: [3.1, "3.1", 3.10, "3.10"]
    ''')
Which may turn into this:

    {
        'port_mappings': 200013,
        'alternative_port_mapping': '3333:33,',
        'countries': ['FR', 'US', 'UK', False, 'IT'],
        1: '1,0',
        2: 1,
        3: 1.0,
        4: '1O',
        5: 1,
        6: 1,
        7: '0i1',
        8: 'oh,',
        'version': [3.1, '3.1', 3.1, '3.10']
    }
It's a terrible, terrible format, that has too much variability, and way too many foot guns. Especially since typo-writing humans are supposed to edit those with their little clumsy hands.

But worse, YAML is mostly declarative, so no complex system can be described in it. Here we come to the second reason people may hate it: it's usually turned into an even worse DSL.

A very limited, badly designed, half documented and tested DSL with almost no tooling to write or debug the stuff. And each project has a different DSL, of course.

Take Ansible. If your playbook is more than 10 lines, it becomes a pain to write because you are basically using a programming language embedded into a markup language. You have no print() or breakpoint to dive into the mess it does. It has to reinvent ways to pass values around since there are no functions with references to parameters nor return values.

Mind you, this DSL calls Python under the hood, so they could have just exposed a declarative, idempotent Python API, but no. The entire world was required to recreate syntax highlighting, snippets, go-to-definition and so on for it. Which will never be good compared to the infra for real languages.

There are few reasons to decide to use YAML, apart from "this is the only rich format supported out of the box by this system that has comments". Which I totally get, but still.

If one can, use JSON, TOML or XML depending of the case, if we need only values.

And if we need more than values, let's either define a clean, restricted API with a full blown programming language, or, if the use case requires it, use CUELang (https://cuelang.org/).

CUELang is cleaner than YAML on all points, even for just data. It can provide a schema, can validate the data with that schema. It can embed logic. It can import and export from YAML and JSON so you stay compatible with the entire devops world.

And it's not Turing complete, but has useful constructs to repeat, branch and reuse code.


> Nope. Nope, nope, nope.

Maybe it's just me, but I read this as a joke. As much as we dislike YAML it's not really something we have the luxury of ignoring, especially in the infrastructure field.


Cluelang is a DSL which I already considered and that I like very much. It wasn't high enough in my priorities.

I struggled to use the HCL.

The DSL used by updatecli is not fixed in stone. It's just the interface exposed. The reality nowadays, is that YAML remains the most used DSL.


To add on top this. I started by supporting other DSL such as Json. Then I realized it was adding burden on me in terms of documentation since people would ask me for different examples.


Use a jsonschema, and validate your configs. You can have similar problems with most other formats, and the best way to make sure your configs are going to do what you expect, is to validate them.


Another yaml problem is that it can have both .yaml and .yml extensions and some tools choose to support only one


updatecli supports both


I love YAML as much as I like treading on a rake.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: