|
Serial Communication Protocols for Embedded Systems |
|
First of all I should make this statement. This article is NOT about writing SLIP, PPP, or any other High Level Protocol connections to embedded controllers. There is already a world of information available on the web concerning the implementation of the well defined communication protocols on any number of hardware platforms. This article is aimed at a fairly common issue when dealing with an embedded system with a serial port. The basic issue is "Okay, I have a serial port on my controller, now what kind of messages do I want to send to it and receive from it ? ". This may seem foolish to some of you but I've had the question asked of me a number of times in my role as a consultant. Often the serial port is not needed for the microcontroller's primary operating task and as such has no definition. You as the developer can implement something to make your life easier and as such you should take advantage of this opportunity. "If the serial port isn't needed for the main job of the microcontroller, why should I spend any time working on "non-deliverable" code?" If this question comes to mind, I would suggest the following. First of all, "non-deliverable" is often not the case, often the true statement would be "not-required" which is a different case entirely. If you are working in an environment in which all code delivered must have a requirement associated with it, you have two choices.
(1) Update the requirement document to reflect the need for diagnostics software. Just remember that this requirement will need to be validated by testing as will all other requirements. (2) Write all of the serial communication debugging software so that it may be removed from the final deliverable code easily ( #ifdef DEBUG_CODE, #endif sequences around all of the code which is developed for this interface should work in most cases.). "But what's in it for me?" I have found that having a method available that will allow me to control and or monitor an embedded controller is ALWAYS useful. You may be able to extract timing information, get retry counts, monitor code execution delays, etc ... . You may also be able to perform in-system testing of your board which may be impossible from the "outside". I will often implement the ability to run specific pieces of a process controller on command from the serial port so that I can validate these sections of code under any number of operating conditions. This type of testing can often be impossible to perform in a fully operational system as I often don't have control of the other pieces of controlling software/hardware. (If your controller is working in a system which takes an hour to perform a full task cycle, running a dozen different variations of signal timing will take a couple of days, if they can be generated at all. By forcing the timing signals in a test environment the time to find and fix software bugs in your code becomes much more reasonable.) It is also nice to get the current version information from your code when needed as it can save you some hassle when an older version of code is running that does not have all of the features required for the current implementation. "Okay, so I will be adding serial port code to my microcontroller, what resources will I need?" The answer to this question can be complex. We need to determine what you want to send and receive as well as what resources are available before we can answer it. If you have several hundred bytes of Ram and several thousand bytes of code space available for this purpose implementing a robust serial interface can be fairly simple. If the microcontroller you are working with is very limited your capabilities may be more limited but you can still implement a useful interface module. If you have the resources available, an ASCII protocol can be a very nice thing to work with. If you have enough RAM and Code space to implement an ASCII protocol you may be well served by going that route. Under these circumstances I will choose this type of interface most of the time. The reason is convenience as almost any terminal emulator program may be used to send messages to the unit and to receive messages from the unit. As many of us are working in one of the "Windows" environments, being able to work with the rather anemic Hyperterminal application can be a plus. (My comments about Hyperterminal are with respect to use as a debugging tool for embedded systems. It doesn't allow for binary data to be displayed, there is no way I have found to clear the screen, etc. ... ). I generally lay out a messaging protocol which has a predefined start character for all messages (sometimes I will use one start character for messages going to the target and a different character for messages coming from the target. This allows monitoring messages and determining easily which device sent the message without having to decode the message body.) As an example I will set up a simple protocol to illustrate this method. For a start character I will utilize the Exclamation Point character '!' and I will choose to end each message with a Carriage Return (CR) character [0x0d]. This has the advantage of being compatible with standard text files and is easily viewed in Hyperterminal. I will also specify that the Line Feed (LF) character [0x0A] may be sent at any time during the message transmission, and will be ignored when received. (Simply thrown away when received by the target system.) By sending a LF after each CR I now make the messaging very easily viewed on the terminal screen as each message will be placed on a separate line. I will further designate that each message will contain a two-character command sequence following the start character that will tell the receiver what type of message is being received. The command sequence characters will be used in decoding the message to determine the format of any associated data that is sent along with the message. Below are examples messages built to this specification:
As you can see, this type of messaging can be quite easily defined, with message specific data placed into the message as needed. The code needed to decode this type of message scheme may be implemented utilizing a table of message parameters based upon the message type (my preference) or it could be decoded in linear code if that is your preferred method. When I implement this type f a protocol I generally set up my serial routines to place all received characters into a ring buffer, and since the protocol contains a fixed end character I will set a flag in the receive ISR when a terminating character is received. In this way the main application only needs to check the state of this flag to determine when a message needs to be decoded. Decoding of these messages I generally leave to a module called from the main program as decoding some of the more complex messages can take a bit of time and I do not like doing any time intensive work in an ISR. (Interrupts should be short and fast in my not-so-humble opinion). The decoder module will generally read the complete message from the ring buffer into a working buffer to be decoded. This makes room for more messages in the ring buffer and eliminates the possibility that a new message will corrupt the one I am attempting to decode. Reading the complete message to a local buffer also simplified error handling, as it is often a pain to remember to flush the remainder of a partially decoded message when an error in decoding the message is encountered. (Yes Virginia, you do need to deal with errors in messaging. Not all messages will be received properly and you need to deal with all possibilities as having one bad message hang up your message decoder permanently is a "Bad Thing" ? ) By having already read the complete message, there is no need to go back to clean up the Ring Buffer.Filtering, message decoding/validation, and other useful stuff. I should mention that reading the message from the Ring Buffer is another place where a filtering mechanism may be implemented. When you begin reading a message from the Ring Buffer, keep in mind that these messages are packetized and we know what the first character should be. I normally will only start storing bytes to my local buffer when I first encounter my start character. This eliminates the need to align my code with the position of the start character in the local buffer, which may be needed if the Ring Buffer happens to contain extra characters, which are not part of a message. One case where this can happen is when a remote controller performs a reset and it's serial port babbles some indistinguishable gibberish for a character or two. Uarts can sometimes detect noise in the system as valid data as well. To deal with some of these conditions you may also want to implement a checksum or CRC within the message packets so that you validate each packet when decoding it. I have implemented a simple single byte checksum (additive) on some systems so as to detect when a remote controller has sent a bad message. If you are working in a particularly noisy environment or have other special circumstances (such as client requirements) a full 32 bit or higher CRC may be needed for your application. I tend to place the Checksum or CRC in the packet just before the terminating character and do NOT include the terminating character in the calculation. (This is simply due to the fact that if the message terminating character is garbled, I won't attempt to decode the packet anyway so I already know the terminating character is valid as it was needed to trigger the message decoding process!) Before decoding the message I pass the message through a message validation function which verifies that the message matches the checksum. I then normally eliminate the checksum portion of the received message so that it doesn't accidentally get decoded as part of the message body. Note also that when implementing an ASCII message protocol, each byte of the checksum / CRC requires two characters in the message as the validation information will need to be converted to ASCII characters and placed within the message itself. Below is the previous example with the addition of a simple additive checksum :
"But, how much Ram and Code space does this all take ?". I have to admit at this point that I can't answer that question. It depends on a number of factors I can't know. You can control some of these factors, such as how long of a message are you willing to support, with the understanding that your Ring Buffer may need to store the message and you will need a buffer for decoding the message if you implement it the way that I laid it out above. Now you can implement a modified version of the above by utilizing a minimally sized ring buffer and pulling characters from the ring buffer periodically so that the ring buffer doesn't have to hold the full message. This requires more interaction from the main application but it has the benefit of requiring less Ram than the previously described method. You could also utilize a timer interrupt to perform a check of the ring buffer to determine whether there are characters in the ring buffer to be added to the decode buffer. As you can see there are many variations on the theme that could be used. The bottom line is that being an ASCII protocol, there will be more data transferred than if you were to utilize a binary protocol. There may also be considerable conversion code required should there be a significant amount of binary data which needs to be transferred to the target device. (On the send side, conversion of the binary data into an ASCII Representation of the binary data (generally into ASCII/HEX form). On the receive side the translation from the ASCII/HEX representation of the data back into the binary form for local use.)"I have a lot of data to transfer to my microcontroller and most or all of it is binary. It doesn't seem that an ASCII protocol makes much sense for this case, does it ?". I would agree. When I have been in that position in the past I have gone with a binary protocol. I tend to utilize a modified binary protocol that I call the FE protocol. The name comes from the fact that the value 0xFE was chosen as the escape character in this particular scheme and I needed some way to designate the method. (Okay, I don't have much of an imagination. I've learned to live with it.) I have implemented this protocol both in a WindowsÔ application and in an embedded application. At some point I will add the code for both of these programs to this web page as examples (not necessarily as something to follow but as proof it works.) "Why a modified binary protocol?" I have implemented the protocol in this way due to some limitations I have found with pure binary data transfers on various systems I have been involved with. My biggest concern with a true binary transfer mechanism is the need to maintain synchronization between the sender and the receiver. As an example, let us assume that I want to be able to transfer a file between my PC and my remote target and the target will write the data received to a storage device for safe keeping. We will further choose to use the value 0x01 as a start of packet marker. The packets will need to contain a count of the number of bytes to be transferred in a packet (the remote device is somewhat limited and we can't just stream data to it, a valid real-world constraint). So when the receiver gets the 0x01 byte it begins it's receive packet routine. The next byte received is taken as the byte count and then a counter is used to determine when a full packet is received. When the store procedure is finished, the remote device sends some message back to the PC to trigger the next block and all is well, the next block is sent down. This continues until the file is transferred. This is the best case scenario and will work most of the time. Here comes the issue. What happens if a single byte of noise is received at any time during the transfer? Since we didn't design in a checksum/CRC we are not going to know it happened and we are going to write junk. We also are now out of sync with the sender, as we will have the last data byte waiting in our input buffer as soon as we start our next read. We will probably have to throw this byte away as it doesn't fit our protocol. Okay, we can fix this by adding a checksum/CRC into our packet start mechanism and validate the packet before we write it to our storage device. The next scenario is worse. In this case the receiver doesn't receive the start byte, or it is received garbled. Since the receiver doesn't know that the byte it received was the start byte, it throws it away as well as every other byte received until a valid start character is received. This now becomes a serious headache if the next 0x01 byte received is actually part of the data, as we will now take the next byte received as a byte count and will ignore any valid start bytes within the next byte count worth of received characters. Of course the newly added checksum/CRC probably won't match so we will not write the packet to the storage device but we are now out of sync with the sender and it is quite possible that we will remain that way. One way around this is to now implement a timeout requirement into the protocol so that the receiver can resync by going silent and performing a resynchronization with the sender. (Essentially the receiver just waits for a valid start of sequence and flushes all received data until a valid block is received.) This adds complexity and realtime constraints to the protocol and makes the implementation less than optimal in my opinion. Don't take this wrong, I have worked on systems that utilize this scheme and many are working in the field at this time. This doesn't mean that in some instances this scheme isn't causing problems. Often the reason it works is more due to the lack of eroneous data (noise in the system) than due to the robustness of the design. "What can be done to overcome these "Limitations" ?" I'm glad you asked that question. (So I had to ask it myself, you might have asked it, eventually, maybe?) What I have implemented in the Modified Binary protocol outlined below eliminates all of the headaches described above, at the price of some communication overhead (extra bytes). (I do need to state that I am not the first to utilize this type of scheme by any stretch of the imagination. I am merely one of the many who have found it to be a useful alternative to a pure binary transmission protocol, or transport layer if you prefer.) The FE Protocol description. The basic design of this modified binary protocol consists of a key byte value, in our case this value is 0xFE. This byte was chosen do the original application for which I needed this protocol. I implemented this transport layer mechanism originally to communicate with a device programmer I was building, namely a flash device programmer. The value 0xFE was chosen because the erased value of the flash devices I was working on contained the value 0xFF and therefore when reading blocks of data from the devices I was very likely to encounter this byte value. Therefore I didn't want this to be my key value as the key value when encountered in the actual data is encoded as a two byte sequence and if the value is encountered often the effect would be to double the number of bytes on the transport mechanism (no better than an ASCII protocol in this case). I chose 0xFE as the value closest to the most common value to be encountered essentially on a whim. You could chose any value you would like and the system will work properly. How an FE packet is constructed. The start squence for each message from the "master" to the "target" begins with the two byte values 0xFE/0x01. This is followed by, in my implementation a single byte which defines the mesage type. The mesage type may be any value, though if the value chosen is 0xFE then you will need to "escape" this byte in the message. To "escape" a byte in this protocol, the original value of 0xFE is replaced with the two byte sequence 0xFE/0x00 (as explained before, this one value generates a two byte sequence so choosing it well is important for system throughput. Each packet is ended with the sequence 0xFE/0x02. Messages originating from the "target" device, destined for the "master" device begin with the sequence 0xFE/0x03 and end with the sequence 0xFE/0x04. This scheme allows for the synchronization between the sender and the receiver to be performed on each packet. It also makes available over 250 other special sequences which can be implemented as shortcuts such as 0xFE/0x05 utilized as an acknowlegement sent from the "target" to the "master" to let the master know that a packet of data has been received properly. This comes in handy when the "target" device may take a good deal of time working with the data before sending the "master" a packet telling the master unit that the "target" has succesfully dealt with the data. By sending this quick acknowlegement the master knows that the receiver is processing the data and will presumably contact the master when the processing is done. Below are two tables which hopefully will show the basic encoding which is performed when utilizing the FE Protocol I have been describing. Note that the Start and End sequences are different for messages going different directions. This helps to make certain that the unit which sent a message does not end up attempting to decode it's own message should a loopback connection be enabled. "Master" to "Target" messages.
"Target" to "Master" messages.
To further illustrate the use of this mechanism, below are the equivelant messages to those shown previously, with the difference being that the messages are shown encoded in the FE protocol:
FE 01 4D 54 00 12 0B FE 02 =
Reset command 0xFE 0x01 Message Start (2 bytes) 0x4D 0x54 Command ID (2 byte) = 'MT' 0x00 0x12 0x0B Binary Value 0x120Bh 0xFE 0x02 Message End (2
bytes)
To further enhance this messaging protocol a CRC or checksum could be incorporated with little trouble. I would place the checksum value directly before the Message End sequence. Remember that since this is using the FE protocol you will need to encode any CRC or checksum value which contains the binary value of 0xFE to 0xFE/0x00 before sending the data to the target system. An example of 8051 Kiel compatible code implementing the FE Protocol is available via this link. This application is a flash programmer I built which is able to program 29F010 and 29F040 devices. It is controlled via this Windows application. The WIndows app obviously also handles the FE Protocol. These zip files contain source code examples only. Now for the more legal types out there, "I don't guarantee that this code will do anything." and "Use at you own risk". Okay, you have been warned.
Well, I guess I'm done spouting off about this topic. I hope that it has been of some use to somebody. Any feedback, constructive comments, or questions may be sent to this address . -Arkwolf
|