Let me know what you think :-)
Hopefully this is your open source alternative - give it a go, let me know if there's something Fiddler does for you that HTTP Toolkit doesn't!
Agree that it's good to see new alternatives.
That said, this completely makes sense, I'm definitely going to look at that. I could fairly easily swap the 'sign up for updates' in directly in place of the download button when that's clicked I think, which would avoid the extra page and the modal all together, make everything much nicer, and still give keen users a clear opportunity to properly follow the project.
Thanks! Great point.
The project looks very nice and I do a lot of HTTP work so I will try it out today. Good work!
update: K, I had gotten side tracked and forgot that it already downloaded and left me on the email submission page. User error. Downloads fine without email.
The tests I wrote tested the boundary between my code and the HTTP library but the API I was writing against wasn't particularly well behaved and so that wasn't quite enough.
No, happily :-)
The trick is that it starts the application to be intercepted for you, so it can control it a little. It then does some magic to get that specific instance of the application to trust the certificate. There's a lot going on there, but as an example: Chrome has a --ignore-certificate-errors-spki-list to inject the hashes of extra CAs that can be trusted in this specific Chrome instance. When HTTP Toolkit starts a Chrome process, it adds that command line option, with the hash of your locally generated CA.
There's nothing here with a lasting effect, and other running apps on your machine won't trust HTTP Toolkit unless you specifically configure them to. Only the processes spawned by HTTP Toolkit trust your CA, which avoids a lot of the downside of other similar tools like this.
For example, let's say I have a program which uses the openssl API to set a custom trust store. Do you mess with openssl to make sure your cert is in there, or will this break?
For that terminal interception, HTTP Toolkit injects a lot of environment variables, including adding some wrappers to your path, which allow it to inject into a _lot_ of places. It sets `SSL_CERT_FILE` for example, which reconfigures the default CA for any processes started from that terminal that use OpenSSL. Again, all scoped to just this one terminal window.
In the case where you specifically manage the trust store as you describe I suspect that'll break (the requests will fail, but they'll still appear in HTTP Toolkit as just 'Certificate rejected for ABC.com', without the full details). The vast majority of applications don't do that though; they rely on the environment to tell them who to trust, and HTTP Toolkit creates the environment. For specific cases that don't, I also include a bunch of extra overrides and injections, so if there's anything that comes up that doesn't fit it's fairly easy to work around it.
It's all open-source, so you can look through the specifics if you like. The core terminal interception setup logic is https://github.com/httptoolkit/httptoolkit-server/blob/maste.... That spawns a terminal and injects the environment variables from https://github.com/httptoolkit/httptoolkit-server/blob/maste..., which notably add a bunch of things from https://github.com/httptoolkit/httptoolkit-server/tree/maste... to your PATH/PYTHONPATH/RUBYLIB/etc.
AFAICT, it's hard to do everywhere reliably for the general case, though as I say I might use it for specific niche cases later. At the moment there's zero native code required in this terminal hooking, which is very nice, and it's very nearly identical logic for all platforms too. It's not so bad, right now it's less than 1000 lines of code in total, and it works out of the box for Python/Node/Ruby/PHP/curl/npm/apt-get, etc etc etc.
`yay -S httptoolkit`
AUR is 0.1.13 and latest is 0.1.14 fwiw.
The language was confusing but I got through it and WOW.
Nice job on the UI too, Linux UI's are hit or miss and HTTP Toolkit's is pleasant!
This just earned a subscription, will get it processed soon! Thanks again, a breeze to use!
When you say 'the language was confusing' though, what do you mean? I should fix that.
There are other tools that exist anyway, but all the big ones (Fiddler & Charles, for Windows & OSX respectively) are very much closed source. The other similar open source competitor I'm familiar with is James Proxy (https://github.com/james-proxy/james) but that's now effectively unmaintained.
For comparison, using the internals of Fiddler by itself would cost $2,999 per application per year (https://www.telerik.com/purchase/fiddlercore). The standalone internals of HTTP Toolkit OTOH are on GitHub: https://github.com/httptoolkit/mockttp.
Though, as a development tool, I don't see any issues using AGPL software.
It's really just the runnable desktop app part that's AGPL. That encumbers you if you're making a directly derivative work (no running off with the whole app but closing the source, thank you), but otherwise it shouldn't limit you at all. This isn't like using a (A)GPL library in your codebase; it's a full application and you're an end user.
Morally bad and ultimately self-defeating, but I'm guessing legally fine?
In practice, I'm not too worried. It's a fair bit of a hassle, as you'd have to rebuild everything yourself. It'd also require a fork that would make keeping up to date with the main versions new features painful, and it's been moving pretty quickly!
It's only a few dollars a month, if that's expensive enough for you that the time & hassle to fork the whole thing would be worthwhile, email me at firstname.lastname@example.org and I'll just give you a free subscription.
I think I more saw it as a challenge to be honest! But if I do get to that point in the future I may take up your generous offer.
Great README docs by the way - spot on in content and delivery!
You can, however, write small python scripts that have read-write access to the http flow (request and response objects).
I guess it's a more low level tool, because it doesn't provide mocking and other helpful tools out of the box.
It's a Node.js library, so you'd need a Node container and a little script to configure it. Mockttp can spin up an HTTP/HTTPS proxy configured to pass through traffic in a couple of lines, and from there you can add any other mocking rules you want (the Mock tab here is really just a UI over this) and you can subscribe to requests/responses/whatever and report that data elsewhere in any form you'd like (that's where all the data shown in the View tab here comes from).
The details get more complicated of course, but it's definitely possible.
In the meantime, it's quite possible to intercept Android traffic, it's just that you'll have to do the proxy & certificate setup yourself.
Do you plan to automate app repackaging with the needed changes, or is there a better method for apps to trust local certificates?
Eventually I'm aiming for the former, I'm fairly confident there is no better method. To start with, it'll be an Android app that configures your proxy settings (acting a VPN) and walks you through adding the certificate to the user store, and for your own apps I believe that's useful already: you just need to enable user stores in your app config (https://developer.android.com/training/articles/security-con...), and everything will Just Work.
For other apps though, on new-ish Android apps it's more difficult, as you say. My plan is to try and rewrite them, _probably_ as a Pro feature, but TBC. It should be a matter of:
- Get hold of the app APK (I think you can pull it with some adb tricks, slightly unclear)
- Edit the XML config to enable the user cert store, change the app id too (so it doesn't replace or have signature conflicts with the real app)
- Repackage & reinstall the resulting APK over adb
I haven't tested this yet though, so who knows!
Is it possible for HTTP Mock to dump both the entire SSL session traffic, and the decrypted HTTP traffic (with fake tcp headers), to pcaps for future analysis?
I would try it tmr.
I'm not sure how zero-setup works, but it's cool feature indeed. I was frustrated to setup those certificate on Fiddler. I suppose that Fiddler tends to expert users, not me.
- It's open-source
- It's cross-platform
- It can do all the interception setup for you, for a single specific target. Fiddler et al set the system proxy, so there's a lot of other traffic going on, and you then need to make applications trust the certificate manually too. Notably some apps (like Node) ignore the system proxy settings entirely, which means you need code changes. HTTP Toolkit does some clever injection, and skips all of that.
- Nice & modern UI
- Built in docs for all standard HTTP status codes/headers/methods/etc
- The Pro features: validation & inline docs from 1400+ API specifications, testing & comparison of compression options, explanations for how your response will be cached & why (and warnings).
I keep meaning to do a proper comparison page (any day now), but that's the gist.
If you want to do intercept something manually, there's general instructions under the 'Anything' interception option. In short:
- Set the proxy on the device to point at your machine's local IP, usually on port 8000 (active port is shown under the 'Anything' option)
- Trust the generated MITM certificate on your device, so you can intercept HTTPS too (path to the certificate also shown under 'Anything').
Fundamentally, HTTP Toolkit, Charles & Fiddler are all HTTP & HTTPS proxies, so they all allow intercepting the same things, it's just a matter of how easy the setup process is.
I actually use extension functionality to extend Fiddler a bit to emulate our internal authenticating proxies we have at work (like PingFederation).
There's the language debate to start with. E.g. Fiddler uses JScript.NET, which isn't exactly the language of choice nowadays. Realistically whatever language you choose, you have some big problems. Different groups want different tools, some want external packages, some want data stores, it's a whole thing.
It's a good point though, and I'd love to solve it. I've added an issue to my feedback repo to look at it: https://github.com/httptoolkit/feedback/issues/37. If you've got a minute, could you comment there and record the kind of scripts you'd like to write?
I'm not really sure what emulating internal authenticating proxies means in practice, for example. Are you rewriting requests to inject auth params, or validating outgoing requests, or something else?
Any chance you can export a HAR from your browser dev tools and send it to email@example.com? There's something very odd happening there, could be a Netlify bug.
gawd... I just need a Fiddler in macOS... one can't really improve what Fiddler provides.
You can provide your email after download if you want updates from the mailing list (or you can not), and you can click 'Get Pro' somewhere and provide your email if you want a paid Pro account. The core app will give you pretty much everything Fiddler does though, on Mac, and doesn't require your email or anything else.