
Show HN: HTTP Mock – Intercept, debug and mock HTTP(S) with zero setup - pimterry
https://httptoolkit.tech/mock/
======
pimterry
Hey HN! I've been building this for a while, but up until now it's been a
read-only HTTP debugging tool, and today's my first foray into proper mocking,
which opens up a whole new world of fun (and practical use cases).

Let me know what you think :-)

~~~
pault
I hope this works out! Using fiddler us such a frustrating experience. The
documentation is practically non existent and the ux is atrocious, and as far
as I can tell the closest thing to a manual is a paid ebook by the developer
(I was so pissed when I discovered this). I would be very happy to have an
open source alternative!

~~~
altano
If you think Fiddler sucks get a Mac and try using the tools available there
like Charles Proxy. Ugh.

~~~
luckman212
Curious about this comment. I'm a Mac user and a Charles user. I find it to be
a great tool. Why the negativity?

~~~
altano
It's extremely hard to use compared to something like Fiddler, especially when
you are doing more than just tracing.

------
bdcravens
I understand why you're doing it, but the download link redirecting me to an
email capture page is quite jarring. If your download link is at the top of
your homepage, and there's more product details below that link, it's pretty
typical to hit download and assume it's a normal download link while you read
the rest of the page. An easily dismissible modal would work better (or put an
email capture in your app on first-run)

~~~
pimterry
Hmm, interesting. I really just did it without much thought, I didn't think it
was particularly unusual behaviour.

That said, this completely makes sense, I'm definitely going to look at that.
I could fairly easily swap the 'sign up for updates' in directly in place of
the download button when that's clicked I think, which would avoid the extra
page and the modal all together, make everything much nicer, and still give
keen users a clear opportunity to properly follow the project.

Thanks! Great point.

~~~
gatherhunterer
I don't see too much of a problem, just that if the download link does not
come up right away (this could be caused by a user's browser settings or
connection speed, which are beyond your control) it looks like they have been
taken to a "Here, have some spam instead" page. I would just switch the
position and styling of "Sign up for updates" and "You are now downloading..."
so that confirming the download request is the primary message on the screen.
You could also ask a user to wait three seconds and make them look at your
(now less prominent) appeal to sign-up for emails while they do so.

The project looks very nice and I do a lot of HTTP work so I will try it out
today. Good work!

------
swiley
_THIS_ would have been nice to have at the beginning of the summer.

The tests I wrote tested the boundary between my code and the HTTP library but
the API I was writing against wasn't particularly well behaved and so that
wasn't quite enough.

------
meritt
How is this achieving https interception with zero setup? Does that not
require a system-level root CA to be installed and trusted?

~~~
pimterry
> Does that not require a system-level root CA to be installed and trusted?

No, happily :-)

The trick is that it starts the application to be intercepted for you, so it
can control it a little. It then does some magic to get that specific instance
of the application to trust the certificate. There's a lot going on there, but
as an example: Chrome has a --ignore-certificate-errors-spki-list to inject
the hashes of extra CAs that can be trusted in this specific Chrome instance.
When HTTP Toolkit starts a Chrome process, it adds that command line option,
with the hash of your locally generated CA.

There's nothing here with a lasting effect, and other running apps on your
machine won't trust HTTP Toolkit unless you specifically configure them to.
Only the processes spawned by HTTP Toolkit trust your CA, which avoids a lot
of the downside of other similar tools like this.

~~~
debatem1
So, I'm confused. Does this work on a set of programs where you know eg the
right command line parameters to give them or on all programs?

For example, let's say I have a program which uses the openssl API to set a
custom trust store. Do you mess with openssl to make sure your cert is in
there, or will this break?

~~~
pimterry
It works on a set of programs, plus terminal interception that works on
anything (more or less) that you spawn from the given terminal.

For that terminal interception, HTTP Toolkit injects a lot of environment
variables, including adding some wrappers to your path, which allow it to
inject into a _lot_ of places. It sets `SSL_CERT_FILE` for example, which
reconfigures the default CA for any processes started from that terminal that
use OpenSSL. Again, all scoped to just this one terminal window.

In the case where you specifically manage the trust store as you describe I
suspect that'll break (the requests will fail, but they'll still appear in
HTTP Toolkit as just 'Certificate rejected for ABC.com', without the full
details). The vast majority of applications don't do that though; they rely on
the environment to tell them who to trust, and HTTP Toolkit creates the
environment. For specific cases that don't, I also include a bunch of extra
overrides and injections, so if there's anything that comes up that doesn't
fit it's fairly easy to work around it.

It's all open-source, so you can look through the specifics if you like. The
core terminal interception setup logic is
[https://github.com/httptoolkit/httptoolkit-
server/blob/maste...](https://github.com/httptoolkit/httptoolkit-
server/blob/master/src/interceptors/terminal/fresh-terminal-interceptor.ts).
That spawns a terminal and injects the environment variables from
[https://github.com/httptoolkit/httptoolkit-
server/blob/maste...](https://github.com/httptoolkit/httptoolkit-
server/blob/master/src/interceptors/terminal/terminal-env-overrides.ts), which
notably add a bunch of things from
[https://github.com/httptoolkit/httptoolkit-
server/tree/maste...](https://github.com/httptoolkit/httptoolkit-
server/tree/master/overrides) to your PATH/PYTHONPATH/RUBYLIB/etc.

~~~
debatem1
At that point why not just LD_PRELOAD and hook the creation of all cert
stores?

~~~
pimterry
I did look at it! It's still an option in my back pocket if I find things that
don't work nicely, but right now this is much easier. I'd need to use
LD_PRELOAD to hook lots of different libraries and tools (OpenSSL is just the
start, and I also need to hook proxy settings), plus handle different versions
of the hooked libs, and make it work reliably on Windows & Linux & Mac.

AFAICT, it's hard to do everywhere reliably for the general case, though as I
say I might use it for specific niche cases later. At the moment there's zero
native code required in this terminal hooking, which is very nice, and it's
very nearly identical logic for all platforms too. It's not so bad, right now
it's less than 1000 lines of code in total, and it works out of the box for
Python/Node/Ruby/PHP/curl/npm/apt-get, etc etc etc.

~~~
kingosticks
An LD_PRELOAD approach would be useful for working around that flavour of
certificate 'pinning' where the cert files are embedded in the application
rather read from the filesystem. I don't know how common that is compared to
the other pinning methods (public key pinning, hashing) but maybe you could
make it work for those also. I guess you'd have to implement it for each
individual library but does any other (free) mitm proxy offer this feature?

------
hartator
Love it. We definitely need a fiddler in macosx.

~~~
gmanley
Charles is pretty good, though it’s paid. It’s definitely nice to have an
opensource tool, though.

~~~
hartator
Charles is okay. But when you have been used to Fiddler is hard to go back.

------
ElijahLynn
Arch AUR package here everyone >
[https://aur.archlinux.org/packages/httptoolkit/](https://aur.archlinux.org/packages/httptoolkit/)

`yay -S httptoolkit`

AUR is 0.1.13 and latest is 0.1.14 fwiw.

~~~
ElijahLynn
WOW. Just WOW. Easiest intercepter I have EVER used. (Charles, Fiddler,
MITMProxy). After install I was able to edit requests and responses with the
MOCK rules in no time.

The language was confusing but I got through it and WOW.

Nice job on the UI too, Linux UI's are hit or miss and HTTP Toolkit's is
pleasant!

~~~
pimterry
Haha, thanks! Glad you like it.

When you say 'the language was confusing' though, what do you mean? I should
fix that.

~~~
ElijahLynn
It has to do around intercepting the response body and resume vs edit wording.
I'll try to submit an issue soon with more details.

------
mariocesar
There are full opensource alternative for this? I like the product, I'm
testing it right now. But I'm curious if there is a way to accomplish
something like this with existing opensource tools or a similar full
opensource project.

~~~
pimterry
HTTP Toolkit is fully open source, primarily AGPL:
[https://github.com/httptoolkit/](https://github.com/httptoolkit/). That
includes the Pro version too - it's not open core, it's just open source all
the way down.

There are other tools that exist anyway, but all the big ones (Fiddler &
Charles, for Windows & OSX respectively) are very much closed source. The
other similar open source competitor I'm familiar with is James Proxy
([https://github.com/james-proxy/james](https://github.com/james-proxy/james))
but that's now effectively unmaintained.

For comparison, using the internals of Fiddler by itself would cost $2,999 per
application per year
([https://www.telerik.com/purchase/fiddlercore](https://www.telerik.com/purchase/fiddlercore)).
The standalone internals of HTTP Toolkit OTOH are on GitHub:
[https://github.com/httptoolkit/mockttp](https://github.com/httptoolkit/mockttp).

~~~
xyzzy_plugh
The parent was likely referring to AGPL, a somewhat-encumbered copyleft
license, which most folks working in proprietary software are hesitant to
touch. They are probably looking for a non-copyleft license.

Though, as a development tool, I don't see any issues using AGPL software.

~~~
pimterry
Could be! It's all open-source but actually not all AGPL: the core desktop app
is, but the internal libraries like
[https://github.com/httptoolkit/mockttp](https://github.com/httptoolkit/mockttp)
are generally MIT or Apache 2, so if you're looking to dig into and use those
internals, it's pretty thoroughly unencumbered.

It's really just the runnable desktop app part that's AGPL. That encumbers you
if you're making a directly derivative work (no running off with the whole app
but closing the source, thank you), but otherwise it shouldn't limit you at
all. This isn't like using a (A)GPL library in your codebase; it's a full
application and you're an end user.

------
chrismatheson
Is this something that could be used in a CI pipeline as well?

~~~
pimterry
As a UI desktop app, not really, although it depends what you're trying to do.
However, the internals are all open-source too, so you can use them standalone
and automate HTTP & HTTPS with that. Take a look at
[https://github.com/httptoolkit/mockttp](https://github.com/httptoolkit/mockttp)

~~~
chrismatheson
I think (maybe postman) had a nice gui, and then the same mocks could be
loaded up as part of a test. I was thinking more along the lines of how the
application actually inspects network traffic, is that something that could be
setup & torn down within a unprivileged docker container for example?

~~~
pimterry
Yes, that's exactly what you'd want Mockttp for.

It's a Node.js library, so you'd need a Node container and a little script to
configure it. Mockttp can spin up an HTTP/HTTPS proxy configured to pass
through traffic in a couple of lines, and from there you can add any other
mocking rules you want (the Mock tab here is really just a UI over this) and
you can subscribe to requests/responses/whatever and report that data
elsewhere in any form you'd like (that's where all the data shown in the View
tab here comes from).

The details get more complicated of course, but it's definitely possible.

------
ElijahLynn
Totally nice side effect. Since this launches "fresh Chrome/Firefox". I can
use it to read all these paid news sites that track how many "free articles" I
have and don't have to worry about resetting all my cookies. Just launch HTTP
Toolkit then Fresh Firefox/Chrome, paste link and done.

~~~
meowface
You can also open the dev console and click "Clear site data". It won't evade
complex tracking/fingerprinting techniques, but I don't think paywall sites
use those to enforce the paywall (yet...).

------
dessant
Is intercepting Android traffic planned?

~~~
pimterry
Yes, definitely! You can follow progress by subscribing to this GH issue:
[https://github.com/httptoolkit/feedback/issues/10](https://github.com/httptoolkit/feedback/issues/10)
(or signing up to the general HTTP Toolkit mailing list ofc).

In the meantime, it's quite possible to intercept Android traffic, it's just
that you'll have to do the proxy & certificate setup yourself.

~~~
dessant
That's great, thanks for the link! In recent Android versions it has become
difficult to inspect the traffic precisely because certificates installed by
users are no longer trusted, so one has to either root the phone or modify the
apk.

Do you plan to automate app repackaging with the needed changes, or is there a
better method for apps to trust local certificates?

~~~
pimterry
> Do you plan to automate app repackaging with the needed changes, or is there
> a better method for apps to trust local certificates?

Eventually I'm aiming for the former, I'm fairly confident there is no better
method. To start with, it'll be an Android app that configures your proxy
settings (acting a VPN) and walks you through adding the certificate to the
user store, and for your own apps I believe that's useful already: you just
need to enable user stores in your app config
([https://developer.android.com/training/articles/security-
con...](https://developer.android.com/training/articles/security-
config#certificates)), and everything will Just Work.

For other apps though, on new-ish Android apps it's more difficult, as you
say. My plan is to try and rewrite them, _probably_ as a Pro feature, but TBC.
It should be a matter of:

\- Get hold of the app APK (I think you can pull it with some adb tricks,
slightly unclear)

\- Edit the XML config to enable the user cert store, change the app id too
(so it doesn't replace or have signature conflicts with the real app)

\- Repackage & reinstall the resulting APK over adb

I haven't tested this yet though, so who knows!

------
mcx
Would be cool to have something like this for debugging outgoing container
traffic in Kubernetes.

~~~
pimterry
Docker is on the list:
[https://github.com/httptoolkit/feedback/issues/1](https://github.com/httptoolkit/feedback/issues/1)
(you can subscribe to that for updates). Would that work for your setup?

------
kaushikt
Is netflix's PollyJS also trying to solve this?
[https://netflix.github.io/pollyjs/#/](https://netflix.github.io/pollyjs/#/)

------
guardiangod
Thanks for the tool.

Is it possible for HTTP Mock to dump both the entire SSL session traffic, and
the decrypted HTTP traffic (with fake tcp headers), to pcaps for future
analysis?

------
jame_bkk
How does HTTP Mock compare with Proxymam.io? Your tool is promising with zero-
setup, but it works with different macOS app, doesn't it?

I would try it tmr.

~~~
nghiatran_feels
The first noticeable feature is that HTTP Mock is cross-platform and Proxyman
is exclusively built on macOS.

I'm not sure how zero-setup works, but it's cool feature indeed. I was
frustrated to setup those certificate on Fiddler. I suppose that Fiddler tends
to expert users, not me.

------
sebazzz
How does this compare Fiddler?

~~~
pimterry
It has all the same core functionality, plus:

\- It's open-source

\- It's cross-platform

\- It can do all the interception setup for you, for a single specific target.
Fiddler et al set the system proxy, so there's a lot of other traffic going
on, and you then need to make applications trust the certificate manually too.
Notably some apps (like Node) ignore the system proxy settings entirely, which
means you need code changes. HTTP Toolkit does some clever injection, and
skips all of that.

\- Nice & modern UI

\- Built in docs for all standard HTTP status codes/headers/methods/etc

\- The Pro features: validation & inline docs from 1400+ API specifications,
testing & comparison of compression options, explanations for how your
response will be cached & why (and warnings).

I keep meaning to do a proper comparison page (any day now), but that's the
gist.

~~~
nachi
What about intercepting requests from other devices on the same network?
Fiddler and Charles both let you set a proxy on Android/iOS devices pointing
to your instance.

~~~
pimterry
HTTP Toolkit can definitely intercept Android & iOS or other network devices,
it just can't do it for you automatically (yet. Watch this space:
[https://github.com/httptoolkit/feedback/issues/10](https://github.com/httptoolkit/feedback/issues/10)).

If you want to do intercept something manually, there's general instructions
under the 'Anything' interception option. In short:

\- Set the proxy on the device to point at your machine's local IP, usually on
port 8000 (active port is shown under the 'Anything' option)

\- Trust the generated MITM certificate on your device, so you can intercept
HTTPS too (path to the certificate also shown under 'Anything').

Fundamentally, HTTP Toolkit, Charles & Fiddler are all HTTP & HTTPS proxies,
so they all allow intercepting the same things, it's just a matter of how easy
the setup process is.

~~~
debatem1
I know it isn't exactly what you're doing, but you might want to look at
nogotofail on the Android side. It has all the per-app goodness baked in, and
you could probably borrow its approach.

~~~
pimterry
Interesting! Yeah that definitely looks relevant, I'll take a look.

------
barberousse
Am getting too many redirects opening that link

~~~
pimterry
Huh, that shouldn't happen! What browser & OS are you using?

~~~
kevinday
I'm getting the same on Chrome/Mac. Requesting /mock/ just 301's me back to
/mock/

[https://pastebin.com/vmgSU6nj](https://pastebin.com/vmgSU6nj)

~~~
pimterry
Hmm, no idea. I'm using Chrome on a Mac, and it works fine, and my analytics
suggest at least 400 other Chrome+Mac users have been getting through, so it's
not a general problem...

Any chance you can export a HAR from your browser dev tools and send it to
help@httptoolkit.tech? There's something very odd happening there, could be a
Netlify bug.

~~~
kevinday
It's working now - nothing changed as far as I know on my end, but if it
happens again I'll do that.

~~~
pimterry
How bizarre, might be a Netlify hiccup somewhere? No idea, but I'm glad it's
working now!

------
jordache
why do I need to create an account for this?

gawd... I just need a Fiddler in macOS... one can't really improve what
Fiddler provides.

~~~
pimterry
You don't need to create an account for this! Click download, use the app,
it's fine.

You can provide your email after download if you want updates from the mailing
list (or you can not), and you can click 'Get Pro' somewhere and provide your
email if you want a paid Pro account. The core app will give you pretty much
everything Fiddler does though, on Mac, and doesn't require your email or
anything else.

------
viana007
For this type of HTTP intercept I like to use Fiddler and Charles.

