Building a Communications Standard from the Ground Up

Before you read this, just know that this is by no means a good way to do any of this. Hell, it's probably wrong. But it's how I'm doing it for a school project. There are probably many ways and standards in existence that do what I already want to do and do it better, but that's no fun, not a waste of time, and doesn't make for a very good blog post.

So, in this project, I need to have (theoretically) many sensors connect to several aggregators that then forward data to a Redis Database. These sensors are going to be outdoor sensing circuits. What they're exactly doing isn't important, but the connection is. The issue is that because there are potentially many of them within close proximity of each other, WiFi isn't a great idea. On a campus environment with many WiFi networks already present, the last thing we need is more congestion in the air. So what can we use?

I had a RF transmit/receive pair sitting in my drawer, so I opted to go for that. It has a decent range, and the band should stay quiet while nothing is transmitting. The use case here calls for infrequent updates to the cluster, so relative radio silence is totally achievable.

My first experiment in sending data yielded mixed results. When I sent a bit over the communication, it was received just fine, but then it the signal line would go crazy with noise. My guess is that there's signal amplification on the receiver board when there's no discernible signal and only level changes are transmitted, but that's pretty useless for me. So to cut down on noise, I put a 100uF capacitor across the signal line and ground. Did it help? A little, but we're just getting started, as this is just the first layer to the data transmission onion.

The software needs to be able to know the difference between signal and noise. So I have several layers to this. The first software layer is the bit layer. We need to be able to tell when a bit is coming through. It's not good enough to differentiate between a high and low signal, because of the noise, so how do we go about doing this?

It's all about timing. We assume that the time between full messages will be longer than, say 120ms. If we get a pulse longer than that, then it's ignorable and not a bit. We then differentiate bits by the length of the pulse. Each bit takes up ~150ms. The first third is always on, the the last third is always off. The middle third tells us if it's a 1 or a zero. Each message is preceded by a zero as a blanking period so the timing code on the receiver resets correctly. It'll hopefully make sense when you look at the code linked below.

That means we're on to the protocol layer. So now that we have a method by which to transmit bytes, we can send numbers. I was going to send 64-bit integers over the line but, wouldn't you know it, the Arduino's 8-bit architecture doesn't support bit-shifting 64-bit numbers. So I settled for 32-bit numbers. This is the entire frame - so that much data can't actually go into the packet. 16-bit messages can be passed this way.

I decided on a 8-bit preamble (0xBD) as a marker regarding when to start listening and where the message is starting. Then the 16-bit message. This can be any unsigned integer. Then we have a 4-bit checksum (calculated as an XOR of all 4 nibbles of the message). Then we have a final end of message marker (0xB) that signifies completes a frame. This frame is used on the receive side to make capture a stream of bits correctly as a snapshot of a 32-bit sliding window.

So, how reliable is this method?

I set the transmitter to send incrementing numbers with 1 second delay between them (to test the noise ignoring features) and it works pretty well. Sometimes there's a bitflip error or something like that, but the checksum catches it and invalidates the message which, in my opinion, is pretty cool. So no we're sending integers over RF with a little bit of error checking. That's pretty cool. It's slow, but we could theoretically build a whole system on top of this. We could even increase the pulse frequency a little to get faster transfer times. But speed isn't the objective here.

So now let's talk about talking to Redis with an Arduino. How do you do it? It's actually super simple because Redis is plain text over TCP.. So, we just need to connect to WiFi, connect to the Redis host, send text, and expect text back, right?

Actually, this time, yes. That's all it takes.

WiFiClient client;

if (!client.connect(host, port)) {

  Serial.println("connection failed");

  return;

}

// This will send the request to the server

char currentCommand[100];

snprintf(currentCommand, 100, "set %s_%d %d\n", aggrigatorId, sensorId, value);

client.print(currentCommand);

String line = client.readStringUntil('\r');

Serial.println(line);

Serial.println("closing connection");

client.stop();

We'd want to check for the "OK" string here in line, but that's really all there is to it. I appreciate that Redis is so simple and yet it does what it does very well. I imagine that's why it's so popular. Redis is nice. It allows for crazy scale since everything is lazily uploaded. And since Redis is in-memory, it's fast. Now every time you poll data, data is real time. Your database is now your source of truth. I realize this is shifting to more of an architecture discussion, but I'm running out of things to write about.

So there you have it. It's a simple, straight forward communication protocol layered up like any other protocol. From the wire to the data layer complete with error checking. Pretty cool, yeah? Here's the code, now go rule the world with it.