Blog
When Athletic Abilities Just Aren't Enough - Scoreboard Hacking Part 2
9/26/2022
This is part two of a three part series. Please go through the first post in order to understand the background on the device. The sections are listed below:
- Introduction & Signal Analysis - Part 1
- Stealing the AES Key via Hardware Hacking - Part 2 (This Post)
- Full Protocol replication in Python and GNU Radio - Part 3
In the previous post, we discovered that the wireless traffic is encrypted using AES-128 ECB mode. In this post, we will go through how to uncover the AES key. Since this is done via hardware hacking, we must map out the hardware before we hack anything. It is hard(ware) to hack something without understanding how it works first!
PCB Mappings
Mapping out the entire PCB from a complete electrical level is not required. When I reverse engineer boards I think of three things: main components, communication lines and test points. All of this will be explained below.
Components on the Board
The chips on the boards have part information on them; nothing is scraped off or blank. For instance, one chip is labeled as Atmel ATMega328PB. Additionally, the silk screen is quite nice for documentation as well. This makes it substantially easier for us to reverse engineer what each component is doing. We can simply google the component identifier to find the documentation for the chip. Once we have the documentation for every component, we can understand what's the CPU, what's storage and everything else you need to know about the board to perform attacks.
The components on the scoreboard controller and receiver are listed below, with a picture of the receiver in Figure 1:
- ATMega328PB: An ARM MCU from Microchip. I commonly refer to this as the Atmel Processor and main processor below.
- RFM69HCW: RF module from HopeRF.
- Raspberry Pi 3 Model A+: A Raspberry Pi with HDMI, SD Card and other easy ways to interface.
- Splatch Antenna: A surface mount antenna.
Reading up on the part information provides a clean story for what is going on. The Atmel processor is the only large compute and storage on the board. Hence, it must decide what data to send wirelessly by sending it to the RF module. The RF module is magically converting the data and sending it wirelessly through the antenna. At this point, the RF module on the receiver interprets the data from the air and sends this to the Atmel processor on the receiver. Finally, the Raspberry Pi is receiving the data from the Atmel processor and displaying it over HDMI. With this, we now generally understand what every part does from reading about the documentation for the chips on the board. One of this understanding comes from recognizing the capabilities of the component and drawing logic conclusions from there. A generalization of this is shown below in Figure 2.
Communication Lines
For the RF module to send out data, something has to tell it what to do, right? So, figuring out the flow of data will get us closer to our goal of finding the AES key.
According to the RF module documentation, the only way to communicate with the chip is over SPI (more on how this works later). If this is the case, then the Atmel processor must be communicating with it over SPI.
The Atmel processor needs to talk to the Raspberry Pi as well. The Pi has a plethora of GPIO (General Purpose In/Out) pins that are connected between the two boards. Since this could be any protocol, this could be tricky to decipher. Sniffing the traffic & reverse engineering the firmware would allow us to figure this out; however, Jesse and I ended up seeing a class on the Raspberry Pi application (more on this later) with serial in the name. This made it fairly obvious that this communication line was serial but this could have been discovered in other ways.
Marked Test Points
Both of the boards have plenty of test points with various labels. To understand expected developer functionality, it is important to trace these markings to think of ways to attack it. From reading documentation and looking at the text on the PCB, two main ideas come to mind:
- In-Circuit Serial Programming (ICSP): The main way to program and debug the Atmel processor. In Figure 5, this is labeled as ISP.
- Serial Interface: A common way to debug devices. If you have ever done any hardware hacking on a router, this is the communication protocol used for interacting with it. An image of the headers can be seen in Figure 4.
Threat Model
After reverse engineering the PCB, we can now brainstorm attacks for this. Writing down all of your ideas in your notes tends to be extremely useful. From this process, we can come up with 5 ideas.
The Raspberry Pi is built to be usable by everyone. Extracting the firmware from the MicroSD should be easy to do. Additionally, the Raspberry Pi has a USB drive plugged in. Maybe one of these has the key!
The next two are the debugging interfaces for the Atmel processor. The ICSP interface for programming could be used to dump the firmware. The label ISP in Figure 5 shows this interface. On top of this, the serial port may provide an interactive shell to interface with the host on the device. Both of these could be used to extract the key, if successful.
Finally, we have the communication lines themselves. Data being sent from the RF module to the Atmel Processor could be sensitive; the RF module needs to be configured by the Atmel Chip in order to properly function. Additionally, communication between the Atmel (main) processor and the Raspberry Pi may have sensitive information as well. Whether something is sensitive or not comes down to how the device was built though.
To sum it up, we have the following attack vectors:
- Raspberry Pi: Reverse engineer the firmware and read the assets from the MicroSD and USB Drive.
- Atmel Processor: Two debugging/programming interfaces for reverse engineering (Serial and ICSP).
- MitM: Sniffing communication on the buses. In particular, the line from the RF module to main processor and the main processor to the Raspberry Pi.
Raspberry Pi - Easy Mode
MicroSD
On the Raspberry Pi platform, the MicroSD card holds the firmware for the device, which has two partitions. First, a FAT16 file system. This contains the Raspbian OS kernel, hardware device trees and a myriad of configuration files. The second partition is a FUSE-ext2 file system used for the host file system (/ and etc.) We copy both of these partitions onto our computers but we want to enable dynamic debugging for testing. This can be done by putting the device into Single User Mode by altering the boot parameters inside cmdline.txt to contain init=/bin/sh. This allows us to have an interactive shell to backdoor the device, such as adding our own SSH user.
The most exciting thing on the device is an application that handles the scoreboard HDMI display and the serial communication from the Atmel processor. Luckily for us, this is a C# application, which is notoriously easy to decompile with the tool DNSpy. This tool even recovers the variable names, function names and strings from the application. Amazing.
From reverse engineering this application, we find the code for parsing a wireless packet from the transmitter by looking for serial communication. A snippet from the decompilation is shown in Figure 6. The code takes the serial buffer and references specific bytes for different fields. Because of the amazing variable names, it is trivial to see what each byte represents in the array.
For instance, in Figure 6, we are parsing the game clock. At lines 13-17, it is taking array[2-5] and using these characters as the content for the game clock. Additionally, array[19] is used in order to determine whether to show a colon (: - 'C') for seconds or a period (. - 'D') for milliseconds on the clock. By doing this for every byte in the packet, we can completely reverse engineer the format. A complete dissection of the message shown is shown in Figure 7.
Hurray! At least we know the packet format being used now. This will be a huge help when attempting to construct a payload in the future. To make our lives easier, all of the data is sent as ASCII characters. So, a 1 on the game clock is represented by the ASCII character for 1 (0x31) in the message.
USB Drive
The USB drive has several configuration files for the administrative web server, scoreboard display and the firmware. For instance, one of the files chooses which scoreboard format and team names are being used. One of the funnier files was signature.txt. This held a signature, which was really a SHA-256 hash with some prefixes. Cryptography is hard to use properly!
Sadly, the key is nowhere to be found on the Raspberry Pi. Additionally, it appears that the scoreboard data is already decrypted by the time the serial parsing was done. This means that the decryption is either done by the Atmel (main) processor or by the RF module. Down the rabbit hole we go!
Atmel Chip Attacks (failed)
From earlier in the post, we mentioned two attacks on the Atmel Processor to try to get the key: dump the firmware over ICSP and interact with a serial shell. From reading the markings on the board, both of these seemed possible.
Setup & Solder
As seen in Figure 5, the debugging interfaces are well labeled. However, there are only headers and no pins to connect to. So, we got to fire up our soldering iron!
Since these are through-hole components, putting them into place is fairly easy. To get them to stick in the hole while we solder from the other side, I usually use electrical tape on the pins. The only tricky part is soldering to the Chip Select (CS) line because there was not a handy-dandy header marked for this. However, there is a smaller hole (via) that could be soldered to, which required a smaller pin.
With a $25 soldering kit from Amazon, we can get the job done. I am quite poor at soldering, but can usually get the connections good enough. I just verify all of the connections with a multimeter once done.
ICSP (ISP) Interface
Both the transmitter and receiver have ICSP headers on them. All we have to do is attempt to connect to it now via the defined protocol. Since the ICSP protocol communicates over SPI, we can use our handy bus pirate to connect. Additionally, we tried a different adapters as well.
Sadly, many hours of attempts on the device would yield no results. Is this our own incompetence or a built in security feature? The Atmel Processor has a security mode that disables the reading out of memory. After all the attempts, the only logical conclusion is that the security mode is turned on. Good job to the team on that! Now, we are down 2-0 to the home team after failing at this.
Serial Interface
The serial interface ends with the same outcome as ICSP. With the ICSP/ISP interface, the headers on the PCB went to the expected locations on the chip but did not function. In the case of the serial headers, the TX trace runs to ground instead of TX on the chip. It is likely that the test boards contained a proper interactive shell but the production boards had a slight change to remove the TX functionality entirely. Nothing comes out of RX when trying to receive data either.
The reason we include the failures in this post is to demonstrate our entire line of thinking. A year of research tends to be condensed into only the successes and none of the failures. I believe that leaving out the failures misrepresents the project and promotes imposter syndrome when ideas do not pan out as expected. Even though we are down 3-0 after this, there are several ideas we can still try!
SPI and Sniffing Setup
At this point in the baseball game, we are down 3 runs. The traffic is encrypted and the key is nowhere to be found. However, we still have one more trick up our sleeve: Man in the Middle (MitM) attacks. Before we dive into this though, we need to learn to speak the language that the RF module and Atmel Processor use to communicate: SPI.
SPI - Layer 1
Serial Peripheral Interface (SPI) is a one-to-many protocol. There is a single controller to many peripherals. In our case, the controller is the Atmel processor and there is only one peripheral: the RF module. The controller decides which components to speak to and when the peripheral speaks to the controller, which is the Atmel Processor in the case of the scoreboard.
In SPI, there are four main wires:
- Controller Out - Peripheral In (COPI): The wire where the controller is sending data to the peripheral. On the board in Figure 5, this is labeled as MOSI.
- Controller In - Peripheral Out (CIPO): The wire where the peripheral is sending data to the controller. This is the opposite of the previous item. On the board in Figure 5, this is labeled as MISO.
- Clock (CLK): A reference for how long a bit of data is.
- Chip Select (CS): The controller selecting which peripheral to communicate with. This is active low; this means that a low (0) tells the peripheral to activate.
The flow of SPI is explained in the steps below, as well as the diagram in Figure 9.
- The controller will drop the
CS(Chip Select) line low for the peripheral that it wants to interact with (RF Module), as shown in Figure 9 on point 1. This means that the peripheral is now active. -
The controller will send a clock (
CLK) and aCIPO(data) signal. The clock signal is shown in Figure 9 on point 2. -
On the falling clock edge, the peripheral interprets the data on the
CIPOorCOPIline as either a 0 (low) or a 1 (high). This is shown in Figure 9 on point 3 and represented throughout the diagram with the vertical red lines. - Step 3 happens continuously until the CS line is turned off for the peripheral.
Now, we understand how the protocol works from the physical layer. But, we still do not know what these raw bits represent. Back to the documentation for the RF module!
SPI - Layer 2 Protocol
The layer 2 understanding of the data depends on the chips being used. In the case of our RF module, SPI was writing directly to configuration registers. These registers are used for sending data wirelessly through the antenna and a myriad of settings, such as the bit rate. The protocol is shown in Figure 10 and Figure 11 with an in depth analysis of the format below.
The first bit of the data determines the mode of the call. This is otherwise known as the R/W bit. If the bit is a 0, this is a read. If the bit is a 1, then the operation is a write. In Figure 10, this is a 0, making it a read operation.
The next 7 bits are used for the address where the mode (read/write) is being performed on. This gives us a range from 0x0-0x7F for configuration registers. For example, the register 0x34 is used for storing a portion of the sync word. If we put the full byte together for Figure 10 we are doing a READ from address 0x34.
The interpretation of the final byte depends on the mode (R/W bit) being used. If this is a read, then the peripheral will send data over the CIPO line for the controller to read. If the mode is a write, then the peripheral will use the next 8 bits on the COPI line and write this data to the configuration register. In the example, the operation is a read. So, in Figure 11, the CIPO line has the first byte of the sync word 0x2D coming from the RF module.
Salae
A logic analyzer is a digital oscilloscope that will record wired signals and help the user decode the information on the communication lines on the board (buses). My logic analyzer of choice is the Salae Logic. However, there are many others on the market that do just as well for a cheaper price.
To setup the Salae to sniff the SPI traffic, we have to wire up all of the lines (CS, CLK, CIPO & COPI) and tell Salae what each line means. The setup is exactly the same as the ICSP/ISP section above; this image can be seen in Figure 12. The red wire is COPI, the orange wire is CIPO, the green wire is CS and the black wire is CLK.
Man in the Middle (MitM)
A Man in the Middle (MitM) attack stems from two entities having a conversation. Then, someone between these two entities can either modify or listen to this conversation. In the case of the RF module and Atmel Processor, we can listen (or theoretically modify) the traffic going between the two components. We are the Evil Eve in a world of Alices and Bobs. An interactive demonstration of the MitM attack performed on the scoreboard is shown in Figure 13.
In part 1, we read through the documentation of the RF module several times. While reading through the documentation, something should catch your eye:
The fixed key is stored in a 16-byte write only user configuration register, which retains its value in Sleep mode.If the authors mention that the key is still maintained in sleep mode, does this mean that it would NOT survive in other modes? It turns out that the RF module configuration are erased once it loses power. This means that the AES key and other settings had to be written on every power up! By design, the key has to be sent over the SPI interface; this was the second big light bulb of the research. Can we sniff the key off of the SPI Bus? If the RF module is performing the encryption, then we absolutely can!
Egg Hunting for Keys
With our trusty Salae setup to listen as the Evil Eve, we can boot up the device and record the configuration process. The 0th bit of register 0x3D, is the setting AesOn. The first byte of the AES key is within the configuration register 0x3E. The key ranges from 0x3E-0x4D, since it is a 16 byte key. Both of these can be seen in Figure 14, which is taken directly from the documentation.
Initially, one would search for 0x3D and 0x3E in the recordings from the logic analzyer. However, we must add the R/W bit in order to find the write operation for turning on AES and setting the key bytes. This gives us 0xBD and 0xBF to search for.
After correcting the bytes we are searching for, we find a 0xBD for setting the RegPacketConfig2 register in the recording! This is writing a 1 to the AesOn setting. With this write, it is turning on AES encryption. Wow, this is actually real; encryption is really being used by this scoreboard.
And, finally, the moment we are all waiting for: the AES key. Directly after turning on AES, the full 16 bytes of the AES key are sent on the bus. Finally, we have solved the mystery of the scrambled bits in the air! Initially, when I discovered this, I sat in my chair shocked. I just stared at the AES key in front of me. This felt like a three run homerun, tying the baseball game at 3-3. The full key and operation in Salae are shown in Figure 15 below:
What was the key? 0x01020304050607080102030405060708 It was so ridiculously simple it could have been brute forced by guessing easy to type characters. This is almost like the RF module had a template with AES enabled by default, with this key being used.
Other Configuration Settings
Before, we were making educated guesses and assumptions about the wireless settings. Now, we can read all of the settings for the RF module. Using this, we can verify our previous understanding and also learn about configurations that we did not know about. For instance, we can read the operation for setting the sync word to be 0x2DD4 directly from the bus, as we expected. Is there anything new? Absolutely!
First, data whitening is turned on. This randomly permutes the bits being sent over the air in order to prevent clock de-synchronization from occurring. Remember how the length bytes 0xE6 and 0xCB were confusing? Well, this is because the whitening includes the length byte, effectively randomizing these bits.
Secondly, there is a Cyclic Redundancy Check (CRC) turned on as well. If the CRC does not pass, then the packet will believe it is corrupted and not be processed. Hence, we have to get the CRC check right as well.
A list of the elements making up the packet are listed below, along with a diagram in Figure 16. It should be noted that the data whitening occurs on the length, message and CRC only.
- Preamble: 1010 (0xA) continuously. This is used to turn on the receiver.
- Sync Word: The clock sync for when to start listening for actual data.
0x2DD4in this case. - Length: The amount of bytes in the payload. This does NOT include the filler padding bytes for the block alignment on the AES encryption.
- Payload/Message: The data being transmitted. This is AES encrypted in our case.
- CRC: A checksum ensuring the data was properly received. This is turned on in our case.
Plaintext Scoreboard Traffic Sniffing
Besides the configuration data, we can read the scoreboard traffic being put on the wire, prior to being encrypted. This is amazing! Now that we have the AES key, settings and plaintext data for the scoreboard, we can attempt to decrypt the data with a known solution. This makes our lives substantially easier.
The plaintext on the wire was almost identical to the serial protocol; the only difference was the first 5 bytes. The first byte sent over the wire is a length byte used by the variable length packet mode. The two length bytes are 0x19 for the shorter packet and 0x34 for the longer packet. The next four bytes were
a static 0xFF 0xFF 0x00 0x00 at the beginning of the data; Jesse and I have no idea why this was sent.
Where to Next?
We found the AES key to tie the baseball game at 3-3! We have passed the major blocker of this project. So, it's smooth swimming from here, right? Recreating the packets is not as simple as encrypting the data. To send this properly, the data must be encrypted, whitened and contain a proper CRC. Although there is some documentation about how these work, they are not complete, forcing us to do more reverse engineering. Additionally, getting the bits into the air as wireless signals is not trivial either. Regardless, we are in a great position to take a 4-3 lead in this baseball game.
In the final post (Part 3), we will go over all of the tricks required to get the packet 100% correct. From data whitening shenanigans to finding the CRC algorithm, the final installation will have it all. From there, we will use GNU radio to send real packets to the scoreboard and uncover ways to affect live sporting events, such as the Super Bowl and March Madness. See you in the next post to pwn this once and for all!
