

How to safely invoke Webhooks - jkarneges
http://blog.fanout.io/2014/01/27/how-to-safely-invoke-webhooks/

======
ebroder
For Stripe's webhooks, we currently use the mechanism described in this post
(resolve the IP address, munge the URL we're connecting to, and manually add a
Host header), but we've run into a lot of problems.

If your client library supports SNI, you're likely going to send the wrong SNI
hostname (we had to disable Ruby's SNI feature because sending an IP address
in the SNI field broke a handful of sites), and cert verification generally
becomes quite tricky.

We're in the process now of switching to a simple go-based HTTP proxy, which
tries to pass through connections as faithfully as possible, but rejects
connections to internal IP addresses. Here's the current implementation
(though I'm still playing with it):
[https://gist.github.com/ebroder/ae9299e0078094211bde](https://gist.github.com/ebroder/ae9299e0078094211bde)

This turns out to be way simpler - all HTTP client libraries have good support
for proxies, and it doesn't interfere with anything about the SSL handshake
(since HTTPS connections go through a CONNECT-based proxy).

We also looked at using Squid or Apache, but concluded that they were much
heavier weight than we needed and it was difficult to make them behave as
transparently as we wanted (e.g. they always want to add headers and such).

~~~
jkarneges
Great idea. A simple HTTP proxy with filter would be useful in isolating the
logic and keeping clients simple.

The effect would be similar to what we do with Zurl (also described in the
article), but more generalized.

------
Mister_Snuggles
This is a good start, but misses two potential problems:

\- The 172.16.x.x - 172.31.x.x address space, which is another internal-only
one. Chances are that you'll know what your internal addresses look like, but
things can change after something is built.

\- What about IPv6?

It's really hard to get something right when you only have a blacklist to work
with. It's much better to whitelist a few things that are allowed, but then
that would limit the utility of Webhooks.

~~~
jkarneges
Other internal ranges: Agreed. We should probably put a formal list somewhere
(maybe on webhooks.org) that all developers can refer to. Of course, there may
be times when you do want internal targets to work as well. I want to say the
problem should be something that people in charge of operations need to be
aware of, and webhook-sending apps need configuration options to specify
ranges. Then it's up to the expertise of the operator to make the right
decisions. This doesn't feel very satisfactory though.

IPv6: Good question. I don't know enough about how it handles address spaces.

~~~
Mister_Snuggles
Fortunately, formally listing all of the internal IP ranges is a solved
problem:

[http://www.ietf.org/rfc/rfc1918.txt](http://www.ietf.org/rfc/rfc1918.txt)
[http://www.ietf.org/rfc/rfc4193.txt](http://www.ietf.org/rfc/rfc4193.txt)

I have a feeling that IPv6 is going to be a lot more complex than simply
blocking FC00::/7\. I seem to recall that one of the benefits of IPv6 was that
NAT-ing would be a thing of the past and every system would have a public
address.

------
blaze33
What if I send you to
[http://lvh.me:10000/some/resource/?method=DELETE/](http://lvh.me:10000/some/resource/?method=DELETE/)
?

edit: ok, it doesn't pass the socket.gethostbyname check

~~~
jkarneges
Indeed. You, know, though... I was almost afraid to click this from my
personal machine.

Fortunately, links in a browser will always perform a GET, which hopefully is
unlikely to do anything terrible on your own computer.

~~~
michaelmior
Links in a browser will always perform a GET, unless there's some JS attached
to the click handler :) Fortunately I think HN is pretty safe.

------
scovetta
Another option: route traffic through an external (authenticated) proxy.

~~~
jkarneges
Yeah, this is point 3 under the mitigation section. The only gotcha with this
one is that you want to be sure there aren't any vulnerable local services on
the proxy server itself. But this should be manageable.

It also may be possible to bind the connect port on the proxy and then use an
iptables rule to prevent any connections from that source port from connecting
to localhost.

~~~
A1kmm
The local port for outgoing TCP connections is chosen randomly in the normal
case - and you can't have two TCP connections between the same pair of ports
and hosts, so it would be hard to write a good proxy server that let you pick
the local outgoing port.

You could use ipt_owner with iptables to block by source user matching the
proxy server, however. Or just configure your proxy to block by outgoing IP if
it supports this.

~~~
samplonius
I think the poster meant IP not port.

Most proxy servers can be configured bind to a specific IP. So if your proxy
server has an external IP and an internal IP, configure the proxy server to
bind to the external IP. And then use IPTables to block any connections from
the external IP to the internal network.

------
michaelmior
Interesting. Never implemented a service which uses Webhooks, but this is
something I had never thought about. Useful to keep in mind.

------
srowley
I'm not a Pythonista, but AIUI the example code will still follow redirects,
so internal pokes can still be made.

~~~
jkarneges
Thanks for the tip. I've switched the code to use Python's httplib rather than
urllib2, which is more low level and doesn't follow redirects.

