Make Something That Isn't A Computer A Computer - Episode 3: Broken IP Camera

This one has been sitting on my shelf for a while now. I've talked about these kinds of cameras before, and now we have the opportunity to see what they're made of - both inside and out - to see what makes them tick, and to see how safe, performant, and reliable they are. We'll look at the firmware to see what's going on inside of it, strip it of all of its camera bits, and make it a generic computer that's somewhat reliable.

This camera is an X10 AirSight XX51A... or is it? It's almost certainly a rebranded generic camera. I mean, come on... just look at it. You've seen these before. And I'll bet the inside will look familiar to anybody who's taken one of these apart before.

This camera has three main parts: the base board, the computer board, and the camera itself. The problem was that the camera kept rebooting over and over again. Why? I don't know. But let's find out.

First off, I'd like to provide a link to what was perhaps the biggest (and only) source of help I could find on the internet. This blog post was absolutely instrumental in starting out with this project, including communicating with the device while offline. This was useful to get into the device without it having a stable connection to the network (which, at the beginning, it didn't).

The website describes two ways to get a root terminal on a camera similar to (or exactly like) this one. The first is over telnet. It runs a telnet server on the default port so it was easy enough to detect. The password for 'root' is '123456'. So secure. Very safe. Such wow.

Unfortunately I couldn't get in this way because the camera kept crashing before it made it to the network. So I tried the other way to get in: using the serial terminal. According to the image on another website (this board is very similar to the NixCore X1, the pinout of which is duplicated below), there are RX and TX pins on the computer board.

They weren't populated, so I soldered wires directly to the pads on the board. I don't have a dedicated Serial to USB converter, so I used an Arduino. Since the serial level was 3.3v, I opted to use a ESP8266 with software serial. The baudrate of the camera was 57600. And, with that, I was able to see the boot sequence, And the error. Not that it helped. The full boot sequence and the error dump can be viewed here, but here's the important part:

Fixing recursive fault but reboot is needed!
CPU 0 Unable to handle kernel paging request at virtual address 00071d00, epc == 800262e0, ra == 800266e0
Oops[#29]:
Cpu 0
$ 0   : 00000000 00000008 00200200 24020001
$ 4   : 8009f674 8ea50060 00071d00 da1908c6
$ 8   : 8009f69c 8033802c 80370000 80380000
$12   : 80370000 fffffffe ffffffff 00000010
$16   : 8009f674 8009f674 8009f674 80323ce0
$20   : 11d37400 003d08c5 80376b48 00000000
$24   : 00000007 803238e0
$28   : 80322000 803239e8 8009f784 800266e0
Hi    : 00000000
Lo    : 00000000
epc   : 800262e0     Tainted: P
ra    : 800266e0 Status: 1100f802    KERNEL EXL
Cause : 0080000c
BadVA : 00071d00
PrId  : 0001964c
Modules linked in: usbvideo watchdog gpio i2s i2c rt2860v2_sta(P)
Process C! (pid: 272629845, threadinfo=80322000, task=8009f674)
Stack : 00000000 00000008 00000004 00000001 8002f134 802e0a84 8009f674 80323ce0
        80340000 80323f00 802f7c58 0000000b 8009f674 80323ce0 80340000 80323f00
        00030000 00000000 80334110 8002f134 80340000 80338ec0 00000001 00020a60
        00000007 803238e0 80340000 80323f00 802f7c58 80323ac8 8009f674 80323ce0
        80340000 80323f00 00030000 00000000 80334110 8000f86c 00800010 80340000
        ...
Call Trace:[<8002f134>][<802e0a84>][<8009f674>][<8009f674>][<8002f134>][<8009f674>][<8000f86c>][<80010ef8>][<80009e00>][<8002cc4c>][<8009f674>][<80149648>][<8002f124>][<8009f674>][<8009f674>][<8009f674>][<8000f86c>][<8009febc>][<8001117c                                                                                >][<80011114>][<801fcea8>][<80009e00>][<801fd270>][<801fd20c>][<801fd1b4>][<8009f7b4>][<8009f674>][<8009f674>][<8000ebac>][<800470cc>][<80172c40>][<8009f674>][<8000ebac>][<8000ec84>][<8009b7d0>][<80009e18>][<8002cc4c>][<80153fc0>][<8000f                                                                                84c>][<8000f850>][<8009f674>][<8001440c>][<80014388>][<8000f7a4>][<8000f798>][<8002c208>][<8002c6f4>][<8002c7ec>][<80009e00>][<8002cc4c>][<8005101c>][<8000f798>][<8000f7a4>][<8000f84c>][<8020cdd4>][<8001440c>][<80014388>][<8020ca38>]

Code: 8c830028  3c020020  34420200 <acc30000> ac660004  ad020004  8c87001c  24a80004  28e30000

Well uh... okay? I'm not totally sure what this meant or what was crashing, but the first step is to figure out which part in software is broken.

There's a script that kicks everything off on the camera. Guess what it's called! test.sh. Yeah, I know. It's very short, here's what was in it after I commented some stuff out. Basically I overwrote it with a blank file and ran the commands in it one by one to see where it crashed. The answer was that it crashed when it loaded the big program: ipcamn.

Before I continue here, I'd like to set parameters for what constitutes making something into a computer. Because defining the goal here is super duper important. I think I'll make rules for what would make posts like this 'successful.' Here they are:

1. The device shall now longer be specifically what it was intended for. In this case, it will no longer be an IP Camera. Just like the drone board is no longer useful for flying a drone.
2. We must be able to get some kind of terminal access to it. Root preferred.
3. We must be able to compile programs for it.

Easy enough. We already have the first two. We're still loading all of the kernel modules it originally shipped with, but we're no longer running the IP Camera software (because something about its interaction with the hardware is broken). We also have a root terminal. So now let's try to compile something for it. I'm also interested in seeing what this ipcamn program actually consists of. But let's do the compile thing first.

This thing uses the MIPS little-endian architecture. It's based off of the RT5350F. It also uses uClibc. So we just need to compile for that. Sounds easy right?

This is the same "toolchain" problem we ran into on the last post of this nature. I couldn't find or build a toolchain appropriate for this architecture. I've seen one person build OpenWRT for this board, but it should be technically feasible to compile for the firmware image that's on here now.

Well, after some finegaling, I made Buildroot produce a toolchain (all of which you can get here) that I could use to build some simple programs. Of course, I made a simple hello world program, and then I compiled a tiny http server for it. Both worked well after I moved them over. At this point, I was moving files over by running a python tftp server and fetching them using the serial console (the client command was tftp -g -l localfile -r remotefile ip port, just in case the busybox help isn't useful). This was super annoying, so I tried moving the compiled Buildroot version of busybox over to the board. It didn't work. Then I found a precompiled version on the busybox website that did work. Now I could set up a temporary FTP server and pull stuff off of it efficiently instead of the tedious TFTP protocol (despite the fact that it's, you know... trivial).

So now I can pull software off of it en masse and delete it to get our very limited flash space back (the new busybox binary is pretty large). I wrote a new test.sh script to load some symbolic links to make the environment more... complete. The original firmware can be found here. Okay, good. So we've met the criteria for this to be a successful post of this series.

I've disassembled the two main files to look for anything fishy. I couldn't find much, but it appears they're mostly statically compiled with a couple of references to some shared libraries. You can see ipcamn.asm and videocatch.asm. Both are long and hard to read, but are interesting to peruse.

I hope this was interesting. It was very interesting to me since I've been struggling with these cameras for a few years now and I've always wanted to see what they're made of. Now I know, and that's pretty cool.