Reverse Engineering Native Apps by Intercepting Network Traffic
The ability to debug web applications is baked into every major browser -- just click Inspect Element and you'll see lots of information. It's not quite as easy to do this with native apps, especially if you don't have their source code. I'd like to show you how to understand the behavior of an application by inspecting its network requests (with or without SSL). I'll also discuss some security implications relevant to developers who are building their own API (private or public).
Note: I am not a lawyer and this is not legal advice, however using these techniques may violate a product’s TOS and may be illegal in some cases. Consult a lawyer if you have any doubts.
Setup
To get going, we need to install a web debugging proxy on a computer (we'll use Charles Proxy), configure our native device to use that proxy, and install a root CA certificate on our device.
You can skip right to the Traffic Analysis section to see this in action.
1. Install Charles Proxy
Get it online at http://www.charlesproxy.com/. You can start out by using the trial version, but I highly recommend buying it. It's a great use of $50 for all of its features. Besides, you'll probably find it annoying that it closes after 30 minutes of use :-)
2. Configure your device to use the computer as a proxy
Make sure your device and computer are on the same wireless network. If you're using an iPhone or iPad: go to Settings -> Wi-Fi, then click the blue arrow on the far right of your network. Scroll down to HTTP Proxy and choose Manual. Enter your computer's IP address and 8888 for the port.
If you're using an Android: go to Settings -> Wi-Fi, long press your network, then press Modify network. Press Show advanced options, set Proxy settings to Manual, and similarly put in your computer's IP address and port 8888. If you don't see that setting, make sure you have a recent version of Android. Most older versions of Android before Ice Cream Sandwich don't let you configure the HTTP proxy, so you won't be able to use this technique.
3. Install the Charles root certificate on your device
The above setup will let you intercept regular traffic, but you won't be able to make sense of encrypted traffic. Many apps use HTTPS for their network communication, so this step will let you analyze them as well.
You'll need to install the Charles root SSL CA certificate on your device. As of May 2013 this can be found at http://charlesproxy.com/charles.crt, but check the latest documentation (here and here) if that link doesn't work. Put that URL into Safari or Chrome on your device, approve it, and you're almost good to go.
Behind the scenes, Charles will effectively perform a man-in-the-middle attack on your encrypted traffic. It will supply its own certificate to your device and the destination host, transparently encrypting and re-encrypting all data sent and received.
SECURITY NOTE: Make sure to remove the Charles certificate when you're done debugging, or your legitimate HTTPS traffic could be compromised later. See Final Notes below.
3a. Turn on SSL Proxying
Inside Charles, go to Proxy -> Proxy Settings and click the SSL tab. Ensure that Enable SSL Proxying is checked. Click Add and type in * for the host field. This will allow Charles to intercept traffic sent to any host. For fine-grained debugging, you should replace the wildcard with only the hosts that you specifically care about.
If Charles asks you to automatically configure your computer's Network Settings, go ahead and reject it. This post isn't about debugging local applications on your computer (although this has many possibilities).
NOTE: Some apps may stop working or report connection errors if you try to inspect their traffic this way. This is likely due to additional security on their part (see SSL Pinning, below).
4. Start analyzing
Open up a browser on your device. Charles should display a prompt asking you to approve your device. Click Allow and you should start seeing data appear in the main window.
Traffic Analysis
For this walkthrough, I wrote a simple iOS app that does simple authentication against a backend, sends the user's current latitude and longitude, and displays a list of nearby places returned by the backend. However, for the sake of science let's assume we're analyzing an app that we downloaded from the App Store, and we know nothing about how it works. Let's see how that looks inside Charles.
Clicking on the individual rows will show the request and response bodies. Charles comes with some very nice functionality by default, such as JSON parsing, timing charts, and more.
We can now deduce quite a bit of information:
The app is communicating with a backend at sandbox.nickfishman.com
It's using Flurry Analytics and Apple Push Notifications
It's using a RESTful web API and supplying an access_token parameter to each request
Playing with the app further would let us enumerate most of the backend's API endpoints and learn how they operate. We also have a valid access_token, so we can probably masquerade as a legitimate client and hit the API programmatically (even if the API is private and undocumented).
Useful Charles features
Setting breakpoints and modifying requests in-place: One of the most powerful features of Charles is the ability to set breakpoints on certain URL patterns. When a request triggers such a breakpoint, Charles lets you modify the request headers and body in-place before sending them on to the server. This lets you instantly test how the backend reacts to changing parameters, without needing to build even a simple custom HTTP client. In our example, we could forge the latitude/longitude parameters sent by the client and see how the backend responds to different locations around the globe.
Replaying requests: We can replay any of the requests in the list to see how the backend reacts. Right click on a request and click Repeat. We can also stress test a particular API endpoint by issuing many requests in parallel (Repeat Advanced).
Simulating latency: Charles also lets us place artificial limits on the bandwidth and latency of requests. This helps simulate slow network conditions, which sometimes cause apps to severely misbehave. For Mac users, you can do even more testing along these lines using Apple's Network Link Conditioner.
Saving sessions: You can save an entire recording session for later reference or processing.
See the documentation for more details on these features and more.
Security implications and preventive measures
This example shows how easy it is to intercept and even actively modify a native app's network traffic. Furthermore, it shows that even HTTPS APIs aren't safe from this kind of meddling. How can we build secure web APIs in the face of such challenges?
Never trust the client
Perhaps the most important conclusion: never trust the client. It's easy to masquerade as the client without disrupting any of the cryptography behind the authentication process. A client with a valid auth token can still misbehave -- either intentionally or due to buggy code. We shouldn't assume the client will always use the API as intended.
Well-known public APIs go to great lengths to handle abusive clients. Some preventive measures include rate limits, internal alerts for clients that supply invalid parameters or make strange API calls, and offline data mining to classify abusive or fraudulent behavior. You might consider implementing some of these techniques even if your API is private or undocumented. Check out nginx's HttpLimitReqModule and Netfilter limits as starting points.
Understand the limitations of HTTPS
Using HTTPS allows your app to establish a secure channel of communication with the server that prevents attackers from reading, modifying, or replaying data sent along the channel. The cryptography behind HTTPS is incredibly sound, but the protocol itself has some important limitations.
We've seen one of those limitations in this walkthrough. It's easy to intercept a secure channel with a tool like Charles if you can get the device to trust your (rogue) root CA certificate. This is straightforward if you have physical access to the device. This doesn't mean HTTPS is broken, but it does mean you should carefully consider the assumptions you're making by using HTTPS.
In general, you can assume that only your app and your backend can read data transferred over HTTPS. However, you cannot assume that the user of your app is unable to read that data. The user can discover any private API endpoints you use and learn more about how your app works under the hood. These discoveries are sometimes embarrassing (this is how Arun Thampi found that Path was uploading a user's entire address book to its servers without permission). Don't send anything over the network that you don't want the user to discover.
NOTE: The rogue root CA certificate issue can also be exploited by governments for surveillance purposes. This is outside the scope of this post, but you can learn more here.
SSL Pinning
One of the most promising solutions to the rogue root CA certificate problem is SSL pinning, also known as certificate pinning or certificate validation. In this technique, a known copy of the server's certificate is bundled with the app itself. When the app makes an HTTPS request, it validates the server's provided certificate against its known copy. If the OS validates the certificate chain against a (potentially rogue) root CA but the certificate doesn't match the app's expectations, it will reject the connection and display a network error.
The Chrome and Twitter iOS apps have already implemented this technique. For example, here's what happens if I try to use Chrome through Charles:
You can learn more about how to implement SSL pinning in this blog post and in Twitter's Security Best Practices guide. Keep in mind that SSL pinning can also be broken by jailbreaking or rooting the device and modifying the app binary (see the iSEC Partners presentation at blackhat USA 2012). However, this attack is more complicated and requires a motivated attacker. SSL pinning is still a good approach for keeping your app's API traffic hidden from casual observers.
Final notes
When you’re done debugging, it’s a good idea to remove the Charles root CA certificate from your device. Otherwise, there’s a chance that legitimate encrypted traffic from your device (such as usernames, passwords, Facebook session tokens, payment information, etc) could be intercepted by someone else on your network with Charles. It’s unlikely, but don’t take that chance. On iOS, go to Settings -> General -> Profiles, and remove the Charles Proxy one. On Android, go to Settings -> Security -> Trusted credentials -> User, and remove the Charles one.
In this guide we used Charles Proxy because it's super flexible and yet easy to use. A powerful, free, and open-source alternative is mitmproxy -- it's definitely worth checking out.
As you've hopefully seen, viewing native app network traffic is almost as easy as using Inspect Element in the browser. In general, you're in trouble if the security of your app depends on nobody discovering the inner workings of your private API. On the other hand, you're in much better shape if your app keeps all its confidential and valuable business logic in the backend, the backend uses a sensible authentication protocol, and the only real attack against your API is a DDoS.
Happy reverse engineering!
