Using XML to describe embedded devices (and speak to them)

There was not really a search needed to find the right language for device description. It's pretty obvious: from all kinds of web standards, XML is the most simple one - and its transform methods into other sources are well defined using the XSLT standards guarded by the W3C.

What we get with XML:

Ok, so now you might say: Nothing new, that's just XMLRPC, right? No, it isn't. Remember that XMLRPC really passes plain XML data (i.e. structures) around. So it needs an XML parser on client and server (or master and slave) side. This turns out in a very bloated and rather academic solution which won't run on a simple uC or FPGA.

Well, from the above item concerning translation, you could guess: We translate the XML into compact C code. So, we can create a library that wraps register procotol accesses into properties. That means, a somewhat CPU featured control instance can access an abstract Property list and does not have to bother about a low level protocol. Whether the library is running on a PC or inside the target is a priori irrelevant, or based on the abstraction level you wish to expose to the outside world. But we'll get to that later in the section called “netpp - The network property protocol”.

One important aspect of development is always a certain learning curve and potential booby traps that can cost you time or cause frustration from the beginning. Remember, we want to save us some work, not create more. This is where a validating editor steps in. We need a easy to use interface which presents us a structure - and a choice of possible elements that can be inserted to the structure. To elaborate for the example of a device: Our 'root node', using geek terms, is typically the device entity. Then, the device node has children entities, like for example a few registermaps, one register bank accessed over i2c, another over SPI, etc. Inside a register, we might want to define a single bit for access...well, you get the point: Due to the nature of XML, we're editing a tree structure.

For the convenience of style sheet support, and because there is a free evaluation version that works perfect, we use the XMLMind XML Editor, short XXE. Below (Figure 1, “Screenshot of XXE”), you can see an example of XXE in action. All entity nodes are presented as simple, collapseable color bars, the editing of attributes is straightforward. On the right upper pane, you can see what happens if we try to insert nodes: the XXE will only allow us to insert nodes that conform to your XML dialect (which is specified in an XSD file). So, we can normally not mutilate a device description, which masks out quite a few potential early failures. In the bottom status line to the left, you see a green "ok" sign. This is the validation status. If there's something dodgy, it will turn into a yellow warning sign or a red X sign, when we have errors. A descriptive error report is shown in the right lower pane, when we click on the status sign.

Figure 1. Screenshot of XXE

By the way: The XXE web resource where you can also download the entire package is: http://www.xmlmind.com/xmleditor/

So, we are turning XML into source code. This is kind of easy to understand: XML will somehow be translated into static C structures. So far so good. But how does the entire development flow look like?

Figure 2. Authoring flow scheme

From the scheme in Figure 2, “Authoring flow scheme”, we can see three main software tools involved:

I guess we don't have to mention that those tools are all free. And yes, we're not constrained to Linux, these tools are found in the Windows world as well, for example in the Cygwin or Mingw environments.

So again, the development process for you as a developer is typically:

As we hinted above, we don't have to necessarily turn the XML description into C. We also normally turn it into documentation templates or VHDL register maps, ready to be synthesized with our RTL implementations. That means, ideally, that we touch this single XML file only for the hardware definition.

In this example, we would like to define a GPIO for a LED output and map it into a named Property that we can access the abstract way. In Figure 3, “Register setup” you can see the GREEN bitfield node inside a LED_Slot1 register which might live in a CPLD on your custom hardware. Not that we normally access LEDs from the outside world, but well, it's just an example.

Figure 3. Register setup

To define the bit location in the register, we edit the 'lsb' and 'msb' attributes and give the bit a name - it makes sense, to stick to a C notation standard, but you're free to use your house rules. Done! We have a register and bit definition. Now, we are going to map this bit into a named Property in Figure 4, “Property mapping”. Since we like it the object oriented way, we just started packing the various LEDs into a struct -- but the 'Green' Property would not necessarily have to be encapsulated in the parent struct. Anyhow, after we've named the properties as shown below, we have to somehow map it to the hardware entity. This is done by inserting a 'regref' node and specifying the name of the register we defined above. To refer to one specific bit, we have to fill in the 'bits' attribute as well.

Now we can access the LED as an abstract property by the name 'LED.Green' in the device's namespace.

Figure 4. Property mapping

Say we have made a hardware definition for a device. So we write some embedded software framework, compile the beast and go. Now we are designing a better version of the device, which has some extra features but is else downward compatible to the 'old' model. But we do not want to keep several software versions around for various devices, it would be nice, if one single software could serve them all. This can be achieved using derivation: A base device class is defined, and another class can inherit its properties - like in C++ or Python.

As you might know from implementing a few low level protocols: it is always simplest to stay with a master and slave relationship on an interface. So the slave never initiates a conversation or connection, it only responds to master requests. This might appear as a limiting factor, but we'll find a very easy trick to get around that - when running in a networked environment.

Let's recall the brave new world of embedded devices principle: Every device should be able to speak to another device. That means, in the protocol basics that netpp will have to provide:

So: The master library side is a priori unaware of the device's properties -- why should we be, the device is able to tell us what its properties are.

The typical interfaces of a modern device are Ethernet and USB, maybe Firewire. netpp does not care much about the physical layers, it sits on top of already existing protocols. In netpp, an interface is abstracted as a Hub, which is in fact a logical hub with ports where you can plug devices in. In a mostly Internet-Protocol ruled world of communication, we know so far that you can speak to almost everything in the neighbourhood (and in the entire world) based on a UDP or TCP protocol and an IP address. The logical Hub just has to somehow find the devices attached -- in this case, find in the local network, who is able to speak netpp, and whether it is UDP or TCP or both. So we can consider UDP and TCP being a Hub. USB is a physically different story, but to netpp it's just another Hub.

When probing a device, a Hub must provide a probe method to list the attached and known devices. That's what satisfies the 'Hello, anybody there?' principle. Initiating a connection is another Hub method, reading and writing to the device...ok, you get the point.

So that means, we can somewhat easily implement other Hubs, if we have to.

The netpp protocol is actually very simple. It is based on a 14 byte header and an arbitrary length payload. If the payload is too big for the logical or physical packet size (defined by the Hub), it is simply split internally. We will not go into the details of the protocol, but we should mention it is completely symmetric: the master and the slave is using the same protocol state machine. This makes it easy to actually reverse the roles of two devices, provided that they're in a network: a master can be slave as well, and run two fully asynchronous connections in different threads (netpp is thread friendly, unlike many other remote procedure protocols).

So what's the deal with the protocol? It just provides a few atomic remote procedure calls (RPC):

That's it. This makes the protocol lean and mean, and leaves much room for extensions.

Nice if we have a protocol, but how do we finally speak to the device? We need an application, or a front end that we can conveniently work with.

Here's what netpp currently offers:

An example of the command line tool in action is shown in this little movie, found here.

In Table 1, “Memory footprints of libfiletransfer.a for various architectures”, you can see a few examples for a rudimentary file transfer protocol implementation based on netpp. All binaries were compiled with GCC, but there is support for various popular compilers. This is the stripped library, not including any static C library code (of which very little is used anyhow).

Devices that normally speak netpp with all network options enabled, are normally Linux featured. It is possible to run netpp 'standalone' on top a simple UDP stack, as done for example for the Blackfin architecture. Memory usage can further be squashed by omitting the networking portions, but that is not really adviseable.

The most prominent example might be the VisionKit framework which is working in a few industrial imaging devices.

For a Linux based IP-camera prototype, it was required to use several complex system-on-chip (SoC) sensors, possibly in a multiplexed mode, to acquire images in different operating modes, depending on the environment situation. The sensors are normally controlled via I^2C (or Two Wire Interface) and have a quite extensive number of registers and properties, thus manual addressing of these properties within the program code was not an option. Nearly all I^2C register properties of all existing sensor head boards are wrapped into several sensor specific device files, which are included in a modular way into a main camera description file. The basic common functionality is defined by the camera device base class, for each pluggable sensor module, another device class is derived from the camera base class. Depending on the plugged in camera module, the 'videoserver' determines, which derived device class is to be used, and communicates the according device root node to the front end user.

Further, some SoC sensor devices use different contexts, for example, exposure time in video mode and exposure time in snapshot mode. For both operation modes, a context area is reserved, containing shadow values for a raw sensor register set. Depending on the used mode, the entire context area is copied into the raw registers.

To address properties within these contexts, the user defines a struct array which is accessed by name like: Context[0].ExposureTime. Offset, size and base address of these context tables can be specified in the XML description.

So you want to get your hands on this?

We set up a netpp resource site here: http://section5.ch/netppres

If you have a linux system, the source should compile from the box. It is recommended to just play around with the netpp master and the devices/example source. All other things are documented in Doxygen, you find a link to the API documentation and the detailed concepts via the above page.

Have fun and thanks for reading!