Show HN: Send a GitHub webhook to a private URL

[qrkourier]

https://github.com/openziti-test-kitchen/webhookz

I work on the OpenZiti project and I have a CI server that accepts GitHub webhooks, but I don't want to expose my server to the internet with open ports. I used the Python SDK for OpenZiti (overlay networking platform) to create a GitHub Action that sends the webhook to my private server via overlay instead of via the open internet.

This GitHub repo is a template that I made to show you how it all works. You can use it right away to run the sample server (httpbin-go) and see the GitHub Action in...action. Relevant threads include https://news.ycombinator.com/item?id=32596212 .

[caseysoftware]

Have you considered ngrok? You can open up a private tunnel from your local environment (local machine, container, device, whatever) to the public internet in seconds. Then you can layer on IP restrictions and even webhook verification to block any traffic not from your provider of choice. It's as simple as:

ngrok http 80 --verify-webhook=slack --verify-webhook-secret=[secret]

with a ton of providers out of the box: https://ngrok.com/docs/cloud-edge#webhook-verification

Also, we recently launched https://webhooks.fyi/ to serve as a community resource to capture patterns & practices around webhook implementations. That's a github pages site so pull requests welcome!

Disclosure: I work at ngrok and helped create webhooks.fyi :)

[gz5 → caseysoftware]

ngrok free is great for smaller scale, as you describe.

openziti simplifies scale:

+ mTLS

+ zero trust w/ inbound firewall rule of deny-all (rather than ACLs)

+ private DNS w/ wildcard domains

note: mTLS, wildcard domains etc are in ngrok $900/user annual plan but these are free for foss like the OpenZiti solution used by OP (and maybe free for other solutions too?)

[ChicagoBoy11 → caseysoftware]

Disclosure: I have never paid for ngrok but have used it sporadically for many years for some quick tunneling use-cases. It feels... magical, and a joy to use. OP, if it fits your use-case, would recommend giving them a try!

[qrkourier → caseysoftware]

That's useful that ngrok has centralized webhook verification! It's meaningful security for the first hop from GitHub to ngrok.

[scottydelta]

> but I don't want to expose my server to the internet with open ports.

One simple approach I take to solve this issue is by whitelisting (using ufw/VPS firewall) Github's webhook IPs listed at https://api.github.com/meta

This works flawlessly while keeping your CI server secure.

[qrkourier → scottydelta]

Did you find a way to auto-update your firewall from the dynamic allow list in the GitHub API?

[scottydelta → qrkourier]

Github rarely changes it's hooks IPs.

The current list has 4 IPv4 IP range and upon checking my server firewall(last updated 3 years ago), I can see I have the first 3 entries in there.

So in the last 3 years, Github has added 1 new IP range which is missing from my server but even then, no webhook call has ever failed to my CI server.

As a precaution I just updated my server firewall right now.

You could of course write a cron script to regularly check Github hooks IPs and update firewall if Github changes it's webhooks IPs.

[qrkourier → scottydelta]

Glad you got it updated before you missed an event! That's the worry that made me look for something flexible and software-defined that I could run in GitHub Actions.

[Yeroc → qrkourier]

The bigger worry would be if they removed some IP addresses from their list. Those IP addresses would be juicy targets for hackers to scoop up and attempt an attack knowing that people have whitelisted them and that they allow access to what is likely relatively poorly protected infrastructure.

[scottydelta → qrkourier]

from my previous comment:

> You could of course write a cron script to regularly check Github hooks IPs and update firewall if Github changes it's webhooks IPs.

This is way easier and simpler than any other solution. It will be a mere 6 line script.

[Spivak → qrkourier]

It's totally trivial in almost all setups. Here's Linux.

    ipset create ghwebhooks hash:net
    iptables -A INPUT -m set --match-set ghwebhooks src -m tcp --dport 443 -j ALLOW

    # in /etc/cron.daily
    ipset add ...
    ipset del ...
    ipset list ...
    ipset save >/etc/ipset.conf

[corytheboyd → qrkourier]

I was doing this with Cloudflare IPs, and iptables as the firewall. Pretty simple bash script scheduled with crontab worked just fine. Was doing this on a DD-WRT flashed netgear router. There's probably something similar that can be done here

[linsomniac → qrkourier]

I do something similar for AWS IPs in our iptables firewalls. I create an ipset for "aws", and then create rules that match that ipset. I then have a script that runs periodically and downloads the set of AWS IPs (AWS publishes that) and writes then runs the script that creates the ipset.

[0xbadcafebee]

Hi @qrkourier, you mention using the Python SDK, but it is not shown in the list of SDKs here, FYI: https://openziti.github.io/ And also there's only a couple listed here: https://openziti.github.io/api/index.html I therefore assumed there was no Python SDK...

[qrkourier → 0xbadcafebee]

I see what you mean. I'll make sure to suggest adding it in those places, then! There's another post about the Python SDK in https://news.ycombinator.com/item?id=32923851 and the repo is https://github.com/openziti/ziti-sdk-py/#readme

[ethomson]

Neat stuff - certainly this problem crops up quite a lot where an internal server needs to get GitHub webhook data.

In the past, I've had good luck using a webhook proxy. I've mostly just used https://smee.io/ which is simple and lightweight although seems to be mostly abandonware at this point. I dockerized it so that it could be used in a Kubernetes cluster, which was very useful for my GitHub Actions build cluster: https://github.com/ethomson/smee-client

There's also Hookdeck, which I haven't used in production, but have played around with, and it seems conceptually the same, but can be made more Enterprisey. Whether that's a bug or a feature is probably up to you.

[qrkourier → ethomson]

The proxy idea is interesting too. Does a webhook proxy entail a polling model for events? That is, does the private server have to poll the proxy to receive the webhook? I wanted the GitHub event to push to trigger actions on the private server.

[ethomson → qrkourier]

No - your local server will still listen for webhooks, but they'll come from the proxy's client software.

Basically, you set up your GitHub webhook URL as the proxy server (for example, smee.io). Then you run a client on your local machine that connects to the proxy server. When a webhook is fired, it will be sent to the proxy, then delivered to the connected client, which will then pass it along as a webhook to whatever machine you've configured.

There's disadvantages to having all this stuff running, of course, so I think that handling this at the networking layer instead of putting a proxy just for webhooks into place is an interesting strategy. Certainly, it sounds like the right solution if you're already using OpenZiti.

[hangonhn]

We ended up implementing an API Gateway in AWS that just proxies the request to our CI server after a simple authentication (and also checking the signature of the request as suggested by GH).

Another related option is that you can run GitHub Runners in your own environment and they will connect to GitHub to accept "jobs" from GitHub Actions. This is another thing we've started doing as we look to pare down our self managed CI. This is a very solid choice I think. In case anyone is interested: https://docs.github.com/en/actions/hosting-your-own-runners/...

[qrkourier → hangonhn]

That's a neat angle. I reckon you could position the self-hosted runner network-adjacent a private CI server and poke it directly from behind the firewall with a GitHub Actions job.

[qrkourier]

Also relevant, sending a GitHub webhook to private Jenkins server with custom params: https://news.ycombinator.com/item?id=31503429