This is awesome! The first thing I discovered was how much network noise crashlytics.com was causing. Used AdBlock's[0] DNS proxy feature to black-hole the offending domain (they even mention blocking crashlytics.com in their FAQ[1]).
Note that both AdBlock and Charlex rely on iOS's VPN feature, and only one can be enabled at a time.
Looks like this app is the one mentioned in that article. They appear to do both Safari content blocking and use the VPN profiles to block DNS requests (as in no traffic hits their servers).
Genuine question here: How is it not absolutely terrifying that an iOS App Store app can man in the middle HTTPS communications made by other apps? Is there some way in which this isn’t poking a hole in exactly the sort of security sandbox that iOS tends to be good at? (And yes there probably is some part of what’s going on that I don’t understand, that’s why I’m asking the question)
I know that without it we wouldn't have amazing apps like https://itunes.apple.com/us/app/adblock/id691121579 , but i'm really not sure if its worth that risk for a common user that apple mostly targets with iOS.
Can't say much about the security, but I suspect it's working by pretending to be a VPN provider and then proxying the traffic. It's then able to install a CA root to generate any certs it needs to MITM traffic. Cert pinning will prevent this from working, but that's the only thing that will.
Supposed sandboxing against malicious apps is precisely why I run iOS rather than Android. I get that Charles isn’t malicious, but what’s keeping any random free game app from doing the same thing? (Again, intended as a real question not a rhetorical one)
Setting up Charles requires two explicit authorization steps (each requiring passcode/fingerprint/face verification): First, network interception requires adding a VPN config (the dialog warns that "All network activity on this iPhone may be filtered or monitored when using VPN"). Second, SSL MITM requires installing and trusting a root CA certificate (the relevant prompts in iOS are less clear -- they say that the cert will not be trusted until you enable it, but don't explain the implications if you do enable it).
"what’s keeping any random free game app from doing the same"
It's not that simple, as author stated, it was some of the most challenging code he ever wrote.
Charles desktop app is well respected in the developer community. There is no reason that the iOS app will be treated any differently.
1a. This requires your passcode, and any time a system is asking for your permission to do something is worth questioning. Even my least technosavvy friends and family have learned that if something is asking for your password and you don't know why, abort.
How do you intercept traffic from apps that use cert pinning? Is the only way to patch the app binary and reinstall the patched binary using a dev certificate?
How exactly does one go about patching the binary – is there a tutorial somewhere?
Won't an app worth its salt use certificate pinning to prevent this mitm ? In other words - Can I use Charles to sniff FB or watsapp traffic ? I do not use both services, but interested in analyzing their traffic.
Yep. App Transport Security mandates that you have to explicitly whitelist the domains [0] which you want to access via plain http. This however, has nothing to do with certificate pinning, which the OP was mentioning.
[0] Of course you can use the blanket NSAllowsArbitraryLoads to allow plain HTTP everywhere.
Apps can do this by presenting a configuration profile to the user. This requires entering the passcode and a few steps - it’s not something they can do silently.
It's your phone. Of course software you installed should be allowed to do anything you want.
The fact that Android has recently made it impossible to MITM apps is really making me consider switching. I don't think I will, because in many other ways Android is still more open, but the analysis is no longer as lopsidedly in Android's favour.
I use the desktop product daily so I picked this up. I frequently proxy my phone through my desktop but I figured this would be fun to play with if nothing else.
I turned it on for literally one second and the first thing it captured was traffic from an app I used briefly several years ago and not since. Cool!
I generally would only be needing to inspect requests when developing at my workstation, so how is this native app providing additional value beyond what the Charles Mac software already provides?
Big fan of Charles over here, I just don't understand the use case for the native app.
"Running Charles on your iOS device means you no longer need to fiddle with WiFi network proxy settings. It also means that you can capture and measure network traffic that goes over the Mobile / Cellular data network.
Measuring networking performance over Mobile data is especially important for your mobile apps (as that is how a lot of users experience your app), and it can reveal large or slow requests, as well as opportunities to increase perceived performance by parallelising network calls."
AFAIK before this you could only inspect traffic over WiFi connections since you had to set the proxy address via WiFi network settings.
You can't use the Mac app to inspect any requests that go over the cellular connection.
Using the Mac app also requires being able to connect to your Mac from your iPhone (in order to use it as a proxy), which is not doable on many setups. For example, at work we use a different wifi network for mobile devices than we do for laptops.
Came here to say the same thing: If you're interested in seeing the traffic caused by your own app (and also making that info accessible to other stakeholders during dev time), netfox is the way to go. Super easy to integrate and provides usually enough info. Also no tinkering with the system settings or third party apps required.
I'm working at a European bank in their iOS team. We use cert pinning for all of our apps, but I have never heard or seen teams using it outside of this project.
I guess it's mostly used if the application is doing something critical like money transactions etc.
Sucks that more and more 3rd party apps are adding pinning to their code so you can't sniff their traffic. This is a great tool for first party debugging though :) Nice work Charles!
I've never done this personally, but I'm pretty sure there is no way to protect against hooking and/or patching functions in Secure Transport (iOS's low-level TLS stack), since all network traffic goes through these APIs. I'm sure there's something similar in Android.
You're not forced to use system facilities for TLS on Android. Back when you needed up to date TLS support for your app on older Android versions you would use e.g. BouncyCastle instead of the system's TLS facilities. Probably the same for iOS.
You can still patch statically linked libraries. Either statically in an editor and resign or dynamically by changing page protection to rwx, and writing a jump to the alternative implementation. Latter requires entitlements, or jailbreak on iOS.
Sure, but then you'd have to JB the phone. Most of this stuff is pretty straightforward, but not exactly 'trivial' - especially given the context is an iOS app specifically aimed at making MITM easier.
Certificate pinning is inherently security by obscurity; it's intended as an annoyance for anyone trying to reverse-engineer the service, rather than an insurmountable barrier.
Basically anything you do client-side falls into that category. If your code runs on my device, there's not much you can do to stop me from fiddling with it.
That gives you the same access as full control over the network. You still can’t make the app believe it has a connection encrypted using a specific certificate.
Charles works by sitting between a server and a phone. It decrypts traffic from the server, displays this to the user to 'inspect', and then re-encrypts the traffic to be sent to the phone.
It re-encrypts the traffic using its own SSL certificate (technically a CA, but no need to get bogged down in details).
In many modern apps, code has been added such that an app will only accept traffic which has been encrypted by the original server certificate (ie: the Uber iOS app will only accept traffic which has been encrypted by a certificate from www.uber.com). When Charles attempts to sit between this traffic, the app will not allow network connections, and thus no traffic can be inspected by Charles.
This procedure of an app only permitting traffic encrypted by a predefined certificate is known as "certificate pinning" or "pinning" for short. It is becoming more and more common for native apps.
Then Geotrust, in a petulant hissyfit, email ~20,000 of their customers private keys to DigiTrust - and the carefully laid plans of updating the app with newer ssl certs pinned 12 and 6 months in advance of expiry are - ummmm - revealed to be flawed...
If you do pinning, you could just as well use a self signed cert for your API and pin that. If your API is not just used for the app, add a private proxy/load balancer that uses your pinned cert.
Awesome news. Charles has been such a helpful debugging tool over the years. Less so for web stuff in these days of browser dev tools being so advanced, but the ability to inspect traffic system wide is still really useful outside webdev, and sometimes it can be useful to verify something dev tools tell you.
All developers should get this for iOS, it’s bound to be useful and if not it will at least be interesting to see what you’d phone is getting up to online!
Yes, but the problem with Charles (well, iOS related at least) is that iOS websockets don't go through the HTTP Proxy configured. They're just considered a raw socket. Thus, even on desktop Charles, it's a nogo.
I like this a lot, but most of the time I use Charles for more than recording traffic. For example, checking how my apps behaves if I throttle certain endpoints, or rewrite responses. Hoping those features makes it into a future version!
Maybe it is worth mentioning that iOS has the ability to throttle the network itself—it's under "Settings"->"Developer"->"Network Link Conditioner".
There is also a pref pane in Addition tools download for Xcode which allows to do the same on the Mac.
Ok, I need to clarify, I had macOS in mind and this note in the mitmproxy documentation:
> Note that the rdr rules in the pf.conf given above
> only apply to inbound traffic. This means that
> they will NOT redirect traffic coming from the box
> running pf itself. We can’t distinguish between an
> outbound connection from a non-mitmproxy app, and
> an outbound connection from mitmproxy itself - if
> you want to intercept your OSX traffic, you should
> use an external host to run mitmproxy. Nonetheless,
> pf is flexible to cater for a range of creative
> possibilities, like intercepting traffic emanating
> from VMs. See the pf.conf man page for more.
That's for transparent mode only though, maybe that had me confused.
iirc, you have to do additional setup for Android 7+ apps -- installing a custom apk with modified manifest.xml for each app that you want to intercept TLS traffic.
Imagine if the user could compile their own kernels for iOS^W^W [edit] that can control an iPhone. She enables IP forwarding in the kernel configuration. Maybe she can also disable some crucial bits for interacting with the baseband. She only wants wifi to work.
Then she uses this phone with the custom kernel (phone #1) as a gateway for another phone (phone #2). She can easily block ads and other undesired traffic destined for phone #2, using a variety of methods of her choosing (firewalls, dns, proxies, etc.).
She does not use phone #1 for anything other than being a gateway for phone #2. There does not have to be any data of any value to an advertiser generated, sent from, or stored on phone #1 (e.g, logs). It is just a gateway.
Sending all traffic over a VPN to a server of your choosing essentially accomplishes the same thing. It's true that there could be some kind of code in there that overrides this choice, but the implications would be pretty severe.
"If you want to do this, you can set up a [third party server] in a cloud somewhere..."
s/you/she/g
s/want/&s/
If she were able to use Android as gateway for iPhone on same subnet, "sett[ing] up a [third party server] in a cloud" is precisely what she would be trying to avoid.
Note that both AdBlock and Charlex rely on iOS's VPN feature, and only one can be enabled at a time.
[0] https://itunes.apple.com/us/app/adblock/id691121579?mt=8http...
[1] https://www.adblockios.com/privacy/