SerialToTCPBridgeProtocol

An error tolerant serial UART to TCP connection, raw data bridge.

Host side gateway service written in Go that listens on COM ports for serial clients. This is meant to bridge the gap between tcp connections and serial devices using UART/RS-232/Virtual COM over USB, etc. Clients implementing the protocol client have a tcp like api that they can use to make connections to real servers. The goal of the project is to have the means to connect the simplest and cheapest devices to the internet, albeit indirectly. For now, I use this to connect microcontrollers on development boards to servers running on localhost through the Virtual COM port over USB, without requiring any Ethernet/Wi-Fi hardware.

See STM32SerialToTCPBridgeClient for an example of a client, written in c, that connects to a MQTT broker from a STM32 Nucleo F334R8 development board.

Details

• The protocol provides the app an in order, duplicates free and error checked byte stream by adding a CRC32 and simple retry mechanism. See this for background.
• The Protocol Gateway opens a real TCP connection to a set destination on behalf of the Protocol Client.
• The Protocol Client connects to the Protocol Gateway over a serial-like connection, which can possibly corrupt data.
• The gateway forwards traffic bi-directionally, as long as tcp connection is open and serial line is good.

Future plans

• Add ping option to periodically test serial line and drop upstream connection if timeout
• Multiple connections per client to servers
• Capability to scan system and listen on all found COM ports for clients
• Turn into OS service

Protocol Packet Format

Structure of packet

Every protocol packet of n bytes uses the scheme above, with the Payload part only present in certain packet types.

Length

The first byte of every packet denotes the length in bytes for the rest of the packet that follows. It is an 8-bit unsigned value that that has a maximum of 255. Thus the maximum possible packet size is n = 256, with a Length value of 255. The minimum packet size is n = 6 for packets that have no payload.

Command

The second byte of every packet denotes the packet type and the instruction for the protocol. The receiver of a packet will process it and execute an action based on the value of Command. The byte is split into two nibbles with the lower 4 bits set to one of the protocol command headers, and the higher 4 bits reserved as flags for each header type to add meta-information to the command.

The protocol command headers are enumerated unsigned integer values:

• 0: Connect
• 1: Connack
• 2: Disconnect
• 3: Publish
• 4: Acknowledge
• 5-15: Reserved

Payload

The payload is a number of bytes that add data to the packet for use by the protocol. It is only present in some packets and has a maximum size of 250 bytes due to the total packet max of 256.

CRC32

The last 4 bytes of every packet is the calculated CRC32 hash value of the entire packet, excluding the CRC itself of course. This is used by the receiver for verifying the integrity of the packet and any user data it might carry.

Packet Command Types

Connect

The Connect packet is sent from a Protocol Client to a Protocol Gateway to open up a new connection. Upon receiving the Protocol Gateway attempts to open a TCP connection using the destination specified in the payload of the packet.

The Destination Type flag at bit 7 specifies the nature of the destination address in the payload:

• 0: The payload is a 4-octet IPv4 address followed by a 2-byte port number
• 1: The payload is a utf-8 hostname string followed by a 2-byte port number

Connack

The Connack packet is sent from a Protocol Gateway to a Protocol Client when it has successfully opened up a TCP connection on behalf of the Client.

The Protocol Client will wait for this packet after sending a Connect packet itself. It will assume its connection attempt to the upstream TCP server was successful when received before a configured timeout.

Disconnect

The Disconnect packet is used to end a connected session between two nodes and is sent when one of the following occurs to one of the nodes:

• It has timed out a configured number of times while receiving bad packets/junk.
• An error has occured acquiring user data to send in a Publish packet.
• It has resend a Publish packet too many times after waiting for an Acknowledge packet.

The Protocol Gateway will also send this packet to the Protocol Client when the upstream TCP server connection closes or fails, and vica versa when the Client app wishes to close the connection itself.

If the Protocol Gateway fails to open a TCP connection on behalf of the Protocol Client, it will send this packet instead of a Connack, with the payload set to a string error message.

Publish

The Publish packet can be sent by both the Protocol Gateway and Protocol Client. It is used to forward user data (i.e. the TCP payload) between two protocol nodes on both ends of the serial connection.

The Sequence Flag at bit 7 is used to uniquely identify a publish message. A protocol node sending a publish packet will set this bit and wait for an Acknowledge packet in return from the remote node with its Sequence flag set to the same state.

A protocol node will send consecutive Publish packets with alternating states for the Sequence Flag. It will also keep track of the flag state of received Publish packets to know what state to expect for the next packet.

If an Acknowledge has not been received in a configured time, the protocol node will resend the last publish packet and wait again. After a configured number of retries, the node will send a Disconnect packet and drop the session.

Acknowledge

A protocol node receiving a Publish packet must always send an Acknowledge packet back with the same Sequence Flag state. It will use the payload data only if the Publish packet's Sequence Flag is the expected one. Otherwise it will discard the data. This enables the receiver node to avoid duplicate data.