Mobile trivia apps, such as HQ Trivia and The Q, modernize game show-style entertainment, allowing anyone to try their hand at winning cash prizes from their smartphone from anywhere. However, these applications are susceptible to cheaters and bots. For example, a developer by the name of Toby Mellor recently demonstrated how he created a near-realtime system to cheat on HQ Trivia. His post (linked above) details his use of Google cloud APIs to create this system.
While the demo was certainly impressive, I felt that I could improve on this system as a proof-of-concept. Armed with his post, I set out to make a better version of his system. However, I set some constraints on myself to make this more of a challenge:
- The system must run offline, with the exception of the QAS (question-and-answer system).
- The system allow a human enough time to tap in the answer.
- The system should require minimal human intervention, since a human will be using the system in real time.
- The system must be functional within one hour.
I was now ready to get started.
Instead of doing fancy (and slow) network-request hijacking using
mitmproxy, I decided to run OCR at intervals on a portion of the image coming from the phone and pipe that to Google to provide a quick answer for the human operator to tap in. The system looks something like the following:
TL;DR: a region of the iPhone’s screen will get captured, put through OCR, and then searched up on Google to get an answer.
Capturing the Screen
First, I needed a way to capture only a region of the QuickTime window on my Mac into a temp file. That turned out to be trivial using the excellent CLI
I initially used interactive selection, where the user manually makes a rectangular selection to capture:
screencapture -ci screens.png
However, I quickly realized that was extremely inefficient and that hardcoding in rect coordinates (a feature
screencapture supports) was a much better option:
screencapture -Rx,y,w,h screens.png
Note: The rect coordinates are in the form
(x, y) being the coordinate of the top left corner.
Optical Character Recognition
It was time to find a passable OCR system that was fast and accurate. The only one I could think of was
tesseract, an OCR engine maintained by Google. While not the best system in the world, its functionality out of the box was pretty great in my testing.
In order to extract text from our image using Tesseract, I used the following,
tesseract screens.png ocr
which takes the image and puts the extracted text into a file,
ocr.txt. Now, it’s finally time to answer this question.
Constructing a Google Search URL and Launching Chrome
One little nasty step we have to do is to put our URL-encoded search query into a Google Search URL and then push that URL to a browser of choice (I used Google Chrome). I used Python for this step because of its familiarity.
URL Encoding the Search Query
A typical Google search URL looks something like this:
Notice how the search query is a query string, the spaces are replaced with
+ signs, and special characters are encoded. Python 3 can do this easily with
So our code might look something like this:
query_encoded = urllib.parse.quote_plus(query)
Programmatically Launching Chrome
Now, we need to pass the URL to Chrome programmatically. With a little bit of trial and error, I found out that the Unix binary, located at
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome, can open URLs without launching a new browser instance. Using that to our advantage, we simply pass the URL as a shell argument, and violà! It works.
Stitching These Components Together
Now, it’s time to mesh these components together. I decided to keep it simple and created a shell script to run each component and pass around data. My shell script ended up looking like this:
#!/usr/local/bin/zsh echo "Realtime Screen OCR" while true do echo "Waiting for trigger" read screencapture -Rx,y,w,h screens.png tesseract screens.png ocr OCR=`cat ocr.txt` python3 launch.py $OCR echo "Opened Chrome...waiting for next question" done
The script waits for the enter key, grabs the predefined rect, runs OCR on it, and then launches Chrome. Then, it goes back to waiting for the enter key.
Testing and Debrief
I ran tests on this system using video replays of previous HQ Trivia shows. It worked well but struggled on the more difficult questions. Here’s a video of the system providing an instant answer to a question:
In my opinion, this checks all the boxes–it works offiline, it’s fast, and it’s pretty much self-sufficient.
However, it’s not always 100% accurate. This is because I didn’t develop an actual QAS system that breaks down sentence structure using NLP and then searches for detected entities within a corpus. But this shows that it’s possible to build something like this. This system took me one hour to build from planning to completion; somebody with more time and dedication could build something even more accurate and scam the companies behind these apps for thousands of dollars.
Preventing people from cheating, especially on devices they own, is extremely difficult. It’s also quite difficult to do right (see https://en.wikipedia.org/wiki/Sony_BMG_copy_protection_rootkit_scandal). The dev mentioned in the Introduction, Toby Mellor, suggests a software check for screen capture devices. While a good first step, this doesn’t really prevent alternative methods of screen capture (such as just straight-up filming the screen with another camera). I suggest fooling the underlying algorithms underneath.
“How do you do this”, you may ask? The answer is adversarial perturbations, a term coined by Evtimov et al. in the paper Robust Physical-World Attacks on Deep Learning Models. By making the text harder to be read by a bot, you are esssentially rate-limiting cheating. So how would this look like in practice? Maybe something like this, which completely fooled Tesseract:
(Well, not exactly that. That picture is quite horrible aesthetically)
HQ needs to take advantage of its animated aesthetic and add a pattern, different fonts, particle effects, or something similar to confuse OCR systems.
There’s already been research into how to fool OCR systems, such as this OCR-resistant font, but I think that with the advent of deep learning, nothing short of overlaying/blending unpredictable patterns will prevent determined attackers from developing systems to cheat.
﹡ ﹡ ﹡
Thanks for making it to the end!
For resume, please email
emailapplecrazy *** a t *** gmail dot com .