I have enjoyed playing Pokémon Go. So have tens of millions of other people, so many that the developer Niantic was completely unprepared to handle this level of interest at first. While they had their hands full, I thought it would be fun to try to reverse engineer their protocol and try to mess with the app. Practical applications of this include finding the exact location of nearby Pokémon, reporting security bugs, and of course cheating. It would definitely be fun to find any glaring security holes that were now suddenly affecting a huge number of people.
Eventually I gave up on this after Niantic caught up to the reverse engineers and made it more difficult to decrypt the traffic. This is probably a losing battle for Niantic, but it sufficiently discouraged casual tinkerers like me. The more dedicated ones are still working over at /r/PokemonGoDev. Since I spent some time on it, I figured I’d write up what I learned from the experience. Learn from failure, and all that.
Aside: don’t use Windows
This is neither here nor there, since people already have plenty of preferences about OSes, but it ended up getting in my way in this case. You certainly can do anything on Windows that you can on another other computer. But when you’re trying to do anything UNIXy, like talk to an Android device over adb, Windows isn’t going to help. I went so far as to even install a Windows version of bash to try to improve the shell experience on the work laptop from my summer job that I used for this project, but that didn’t really help. Maybe Microsoft’s own upcoming port of the core Linux utilities will be better, or maybe I’m just not good enough at computers to immediately get the hang of this version. Whatever.
Don’t touch any variant of Android on an x86 host
Initially I wanted to do all my experiments on a PC, since it’s easier to mess around on a “real” computer than on a physical Android device. Even with root, you have to jump through more hoops, and there are more tools available on a PC. I figured cleartext packet captures might be easier to come by if the source was a VM on my laptop. This is wrong, but never mind, here’s what I tried:
- Standard x86 build of Android installed by me in a Virtualbox VM
- Standard x86 build of Android in someone else’s pre-made VM
- Genymotion Android VM on a Windows host
- Bluestacks Android ARM emulator on a Windows host
- Bluestacks Android ARM emulator on a Windows guest VM
Only the last one of these allowed me to successfully run Pokémon Go and isolate the traffic from it, and then only briefly. This entire class of activity is too niche to be supported well. Just use a real device.
I would leave it there and move on, but in the interest of getting some kind of value from the time I wasted on these attempts, I’ll write down some of the problems I encountered.
x86-native Android in a VM
This would seem to be the most promising way to run Android on a PC, because it’s the fastest. Unfortunately basically no one actually uses Android on PC hardware for anything, so not a lot of people have an interest in making this work. There’s no worldwide network of Cyanogenmod devs who have the same needs as you with your exact device. There’s a project, and you can find people talking about it on xda-developers, but it’s not exactly mainstream.
The basic experience of running an x86 Android build in a VM is actually not bad — it supports the mouse and keyboard and all that no problem. But neither of the versions I tried actually worked when it came to the most important thing: installing Pokémon Go from the Google Play store. The Play app did not say that my device was not supported, which was nice, since indeed the newer builds (since 29.2) are supposed to support x86 devices like the Asus Zenfone. But installation failed, and checking the provided link in the error message was no help. That generic error page actually had a note added to it recently that specifically mentioned that the Play Store could not help with Pokémon Go problems. Ha.
Perhaps we can sideload the apk? I got a copy off my ARM device, which failed to run on the x86 VM. Checking the logcat, the app was looking for some ARM libraries that clearly would not work here. I tried a copy of the app from apkmirror (of course I wouldn’t do this on my real device), which said that it should now support x86, but the same problem appeared. Some people online suggest installing an ARM compatibility layer for your x86 Android device; no luck.
So, Android on x86 exists, but that’s about all I can say about it.
The most promising prepackaged commercial solution seemed to be Genymotion, which is free for personal use and can automatically download appropriate ROMs for your preferred Android version. It is also an x86 VM, but this is not entirely obvious at first, because the virtual devices it creates make themselves out to be “phones”. To make a long story short, some of the same problems occurred here. In fact, it was worse, since the Play Store said off the bat that the device is not compatible with Pokémon Go. Since Genymotion is targeted at gamers, other people have tried sideloading it as well, and again people online suggested a Genymotion ARM Translator layer, which can be downloaded from some sketchy site. I’m not particularly attached to any of the data in this brand new VM, so I tried it, but again no luck.
I downloaded another commercial Android VM/emulator(?) AndyRoid, but 16 scanners on VT think this is malware. Hmm. It’s a downloader, clearly, so maybe a false alarm, but let’s go ahead and pass on that one.
Next up: an actual ARM emulator, Bluestacks. This one is also targeted at games, and Pokémon Go is even on their homepage, so this seemed promising. Indeed, it installs and launches OK. Progress!
Unfortunately, this simplified game platform thing has a few drawbacks. For one, being an emulator, it’s slow. Oh well. But more problematically, since it uses a highly modified custom ROM, some settings aren’t available in the normal place. Like, say, developer options, network settings, and the ability to even view your own IP address. There’s no file manager. The interface is clunky. I think I had to do something special to get adb working (I’ve tried to forget some of this). All around this is a pain.
The first unworkable problem to crop up with this emulator is that it is hard to isolate the traffic from it in Wireshark. It is a regular desktop app like any other, so it uses the same network interface as everything else. What would be better would be to have a virtual interface set up for it that we can isolate. So, I installed it in a Windows VM (on a Windows host, this is so silly). This nested virtualization started to get annoying, but worked.
With Bluestacks, you can access the full Play Store to get access to things like GPS spoofing apps, and an opaque rooting binary. Who knows what that thing does. But again, I didn’t have any real data exposed, so who cares (I took the calculated risk that this platform is obscure enough that these apps would not contain specific VM escape exploits for it, and it was within another VM anyway).
I followed a series of extremely sketchy instructions from Bluestacks themselves on how to spoof GPS in a way the app wouldn’t detect, and everything was just barely working. I left for the day, and when I came back the next…it didn’t work anymore. Honestly I don’t even remember what the exact problem was. Something had borked the entire Bluestacks environment. Screw this.
Use a real device
Seriously, just do this. You can’t capture the traffic in Wireshark as easily, but actually you don’t want to anyway. My initial Wireshark experiment with the layered VMs indicated that the interesting traffic appeared to be encrypted with TLS (as you would hope). To strip this off, you could try to recover the keys from the device’s memory, or something, but the best way is to MITM the connection with your own proxy.
You could try setting this up yourself with something like sslsniff, if you know what you’re doing when it comes to certificates and ARP spoofing and so forth (you know the difference between layer 2 and layer 3, right?). Moxie’s instructions are pretty good. An even friendlier option, and ultimately a more useful one, is mitmproxy, which is extremely easy to set up and is especially handy for parsing the intercepted HTTP traffic in real time. This tool automates the process of installing its own root cert on the device. Then you can watch the requests and responses flow by and have it parse the json/xml/protobuf for you. There’s a Python library to work with the packet capture format it uses. I had no problem getting decrypted traffic from Pokémon Go v29.2 when routing my device’s traffic through this proxy.
Now then, let’s talk about what’s in there. I expected it to be something reasonably straightforward like json, which mitmproxy and any number of other tools/libraries have no trouble parsing. Nope, it uses protocol buffers, a library to make compact binary protocols easier to use on the developer end. Decoding the structure of protobuf data is easy enough, but unfortunately if you didn’t write the protocol, you have no idea what any of it means. As my networking professor put it, binary protocols are more efficient, but text protocols are more useful for humans who have to program them. Protobuf makes binary easier for developers, but not really for reverse engineers who don’t have the protocol definition.
The smart people at /r/PokémonGoDev have made pretty good progress on reversing some of what’s in there and have some .proto files available to help third-party programmers parse it. They aren’t complete, though, and you’ll need some experience with protobuf to really get a handle on this. Rather than figuring the rest out yourself, here’s someone who has already made a lot of progress on this task. Looks handy! Even better, here is a whole MITM toolkit ready to go, apparently. OK sweet.
Just as I found all of these tools, a wrinkle appeared. Version 31.0 of the app uses certificate pinning, and believe it or not, the certificate/keys generated by mitmproxy are not the ones the app expects. This is general good security practice, but in this case it was probably added specifically to stop the kind of thing we are doing. Niantic is not happy about all the mapping apps and bots that have sprung up.
This guide has a nice walkthrough of ways to patch the binary to remove the cert check. That’s a fun example of how to patch binaries in general to disable certain behaviors (like checking serial numbers…not that anyone should ever do such a thing). One issue that arises here is that Google login will no longer work, since the apk will not be signed properly, and you will have to use a Pokémon Trainer Club account.
I tried one of the suggested patch methods, but it did not immediately work, and I lost patience. Niantic wants to shut us out from reading and injecting traffic, and I don’t blame them! Part of my goal here was to figure out how hard such a thing would be. It’s clearly not impossible, but they are trying to make it harder. I don’t have the time to try to keep up in this escalating battle. When it comes to bots, honestly I hope Niantic wins this arms race. The bot writers don’t think they will. Whatever happens, those of us casually poking around will have our work cut out for us. After this many hurdles, I’m calling it quits and am going back to just enjoying playing the game like a normal person.