Using XML to describe embedded devices (and speak to them)
This article discusses one of the typical development cycles in embedded device and communication design and presents a possible, light weight solution using the free DClib/netpp framework.
Assume we're faced with the design of an embedded device, be it a simple SoC unit or a more complex, uC controlled engine with various attached peripherals. From first prototype to the market, the following development cycle is typically walked through:
Design hardware, choose peripherals, I/O, define internal registers
Write embedded software: Peripheral control/access, state machine, self test routines, ...
Write front end (user interface) software to remote control the device
Write documentation about operation modes and device properties
Implement production test procedures for final device release
Many intelligent devices nowadays are equipped with a more complex operating system (OS) and USB interface or even Ethernet. Quite a number of black boxes have a (uC)linux OS running under the hood, there are plenty of open reference designs around, enabling a short design cycle. We can conclude from that that many of these OSes are considered as standard, even if the possibly bureaucratic approval instance might be missing.
When it comes to interfacing with a user or another device, things can and do have to become more bureaucratic: An interface must be properly standardized, otherwise it will not work right.
What again is not standardized, is, how you communicate on the abstract level, rising the rhethorical questions for example: Do you operate/configure the device via touch screen? Or a web interface? Or a TCP protocol, a login shell, or an USB library? How do you see all operation modes and properties the device has? And how does the device internally react, when you change a property, do you show the front end user settings, commands, or even function calls?
Well, there are quite a number of existing solutions that try to tackle the abstraction problem of remote control and property description. Let us list a few:
EDDL: Electronic device description language
GenICam: An attempt to standardize Camera communication
Google Protocol buffers: A smart way for data exchange and remote procedure calls
XMLRPC: Remote procedure calls via XML
Hardware solutions like modbus, CAN, etc.
Last but not least: A simple web server.
Well, as you would guess, none of these solutions really 'did it' for us. Why is that? We could list all the pros and cons for each of the above in detail, but generally, the problem was the abstraction and the code complexity, or just the fact that they're not opensource or license free. The Google Protocol buffers appear most simple, but they lack a specific hardware definition. Specific hardware protocols such as modbus are too unflexible and feature too little abstraction. A web server is kinda out of scope as well, for example, you wouldn't want to implement a web client in a device just to communicate with another device. If you did, you would have to parse a lot of irrelevant data -- well, a web interface is just not made for that.
Let's nail down the boundary specs that a perfect solution should have:
Most abstract and generic description language for hardware entities like registers, commands, and also abstract 'Properties'
Easy way of mapping a hardware entity into a 'Property'
Light weight and compact code generation - we want the least manual coding possible!
A protocol independent of the physical layer
A simple way of making the device report all it can do - the property list
Open, free, platform independent
The summary for this little perfect embedded world is:
Make all devices just so intelligent that they can speak to each other, and tell you what they are capable of.
Well, we're not saying, we're there yet. But we're getting closer...
The basic idea behind the DClib is:
Write a device description file, preferrably using an entry form or style sheet based editor
Automatically turn the device description into a slave (embedded device) library that runs on the embedded target
Speak to all your devices with one single, unified client module.
But we're not done yet with the shopping list: We need to elaborate methods to actually access hardware entities. As a hardware developer, you might be used to deal with:
a low level register based protocol such as i2c, SPI, ...
single bit fields that need to be accessed within a register
As a firmware developer, you may have to deal with:
local configuration variables in uC memory
code that sets and gets register values based on the peripheral's protocol
other handler code controlling the operation modes of the device
Ideally, you would want to have this automatized and somewhat abstracted, but not too far. So, assuming that the DClib framework takes off some burden, what coding requirements would you be left width?
This should give you a quick idea, but isn't the whole story:
You write a device_read() and device_write() method to access one or several peripherals on your target (using software address decoding)
You implement some global variables for the configuration (preferrably in a current context structure)
You wrap some of the devices functionality that can't be covered with registers or variables into Getter and Setter functions.
But let us now look at the technique to describe the hardware.
There was not really a search needed to find the right language for device description. It's pretty obvious: from all kinds of web standards, XML is the most simple one - and its transform methods into other sources are well defined using the XSLT standards guarded by the W3C.
What we get with XML:
Validation: Prevent the user from writing the 'wrong' XML dialect
Translation: Translate XML into a large number of desired formats using XSLT style sheets and standardized processors
A large number of editors supporting adaptable style sheets
Ok, so now you might say: Nothing new, that's just XMLRPC, right? No, it isn't. Remember that XMLRPC really passes plain XML data (i.e. structures) around. So it needs an XML parser on client and server (or master and slave) side. This turns out in a very bloated and rather academic solution which won't run on a simple uC or FPGA.
Well, from the above item concerning translation, you could guess: We translate the XML into compact C code. So, we can create a library that wraps register procotol accesses into properties. That means, a somewhat CPU featured control instance can access an abstract Property list and does not have to bother about a low level protocol. Whether the library is running on a PC or inside the target is a priori irrelevant, or based on the abstraction level you wish to expose to the outside world. But we'll get to that later in the section called “netpp - The network property protocol”.
One important aspect of development is always a certain learning curve and potential booby traps that can cost you time or cause frustration from the beginning. Remember, we want to save us some work, not create more. This is where a validating editor steps in. We need a easy to use interface which presents us a structure - and a choice of possible elements that can be inserted to the structure. To elaborate for the example of a device: Our 'root node', using geek terms, is typically the device entity. Then, the device node has children entities, like for example a few registermaps, one register bank accessed over i2c, another over SPI, etc. Inside a register, we might want to define a single bit for access...well, you get the point: Due to the nature of XML, we're editing a tree structure.
For the convenience of style sheet support, and because there is a free evaluation version that works perfect, we use the XMLMind XML Editor, short XXE. Below (Figure 1, “Screenshot of XXE”), you can see an example of XXE in action. All entity nodes are presented as simple, collapseable color bars, the editing of attributes is straightforward. On the right upper pane, you can see what happens if we try to insert nodes: the XXE will only allow us to insert nodes that conform to your XML dialect (which is specified in an XSD file). So, we can normally not mutilate a device description, which masks out quite a few potential early failures. In the bottom status line to the left, you see a green "ok" sign. This is the validation status. If there's something dodgy, it will turn into a yellow warning sign or a red X sign, when we have errors. A descriptive error report is shown in the right lower pane, when we click on the status sign.
By the way: The XXE web resource where you can also download the entire package is: http://www.xmlmind.com/xmleditor/
So, we are turning XML into source code. This is kind of easy to understand: XML will somehow be translated into static C structures. So far so good. But how does the entire development flow look like?
From the scheme in Figure 2, “Authoring flow scheme”, we can see three main software tools involved:
GNU make: Take care of building what needs to be built
xsltproc: Turns XML into C code (or other) using XSLT descriptions
The GNU C Compiler (gcc): Compiles all the source into a target executable
I guess we don't have to mention that those tools are all free. And yes, we're not constrained to Linux, these tools are found in the Windows world as well, for example in the Cygwin or Mingw environments.
So again, the development process for you as a developer is typically:
Edit the XML file and define device properties (registers, bit fields, etc.). Also insert documentation nodes in those entities
Write some code to support your target framework. This normally consists of a hardware layer, defining how registers are accessed, and some handler code dealing with more complex properties.
Run "make" to build it all, and end up with a target executable.
As we hinted above, we don't have to necessarily turn the XML description into C. We also normally turn it into documentation templates or VHDL register maps, ready to be synthesized with our RTL implementations. That means, ideally, that we touch this single XML file only for the hardware definition.
In this example, we would like to define a GPIO for a LED output and map it into a named Property that we can access the abstract way. In Figure 3, “Register setup” you can see the GREEN bitfield node inside a LED_Slot1 register which might live in a CPLD on your custom hardware. Not that we normally access LEDs from the outside world, but well, it's just an example.
To define the bit location in the register, we edit the 'lsb' and 'msb' attributes and give the bit a name - it makes sense, to stick to a C notation standard, but you're free to use your house rules. Done! We have a register and bit definition. Now, we are going to map this bit into a named Property in Figure 4, “Property mapping”. Since we like it the object oriented way, we just started packing the various LEDs into a struct -- but the 'Green' Property would not necessarily have to be encapsulated in the parent struct. Anyhow, after we've named the properties as shown below, we have to somehow map it to the hardware entity. This is done by inserting a 'regref' node and specifying the name of the register we defined above. To refer to one specific bit, we have to fill in the 'bits' attribute as well.
Now we can access the LED as an abstract property by the name 'LED.Green' in the device's namespace.
Say we have made a hardware definition for a device. So we write some embedded software framework, compile the beast and go. Now we are designing a better version of the device, which has some extra features but is else downward compatible to the 'old' model. But we do not want to keep several software versions around for various devices, it would be nice, if one single software could serve them all. This can be achieved using derivation: A base device class is defined, and another class can inherit its properties - like in C++ or Python.
As we have learnt from the previous paragraph, a hardware description can be abstract-mapped onto a Property level. We have just assigned a register bit to a property, or with other words, we have exported it to the user -- how would we now set this bit from "outside"?
The DClib just covers the function set needed to remote control the device (which may largely be generated from the device description), but it does not yet define a protocol to the outside world. So, we need a protocol which should be property based, not be aware of internal hardware definitions.
As you might know from implementing a few low level protocols: it is always simplest to stay with a master and slave relationship on an interface. So the slave never initiates a conversation or connection, it only responds to master requests. This might appear as a limiting factor, but we'll find a very easy trick to get around that - when running in a networked environment.
Let's recall the brave new world of embedded devices principle: Every device should be able to speak to another device. That means, in the protocol basics that netpp will have to provide:
Probe devices on a certain interface ('Hello? Anybody there?')
Initiate a connection to a device (by id)
Query a property list
Control a property (Get/Set) or query its attributes (children)
So: The master library side is a priori unaware of the device's properties -- why should we be, the device is able to tell us what its properties are.
The typical interfaces of a modern device are Ethernet and USB, maybe Firewire. netpp does not care much about the physical layers, it sits on top of already existing protocols. In netpp, an interface is abstracted as a Hub, which is in fact a logical hub with ports where you can plug devices in. In a mostly Internet-Protocol ruled world of communication, we know so far that you can speak to almost everything in the neighbourhood (and in the entire world) based on a UDP or TCP protocol and an IP address. The logical Hub just has to somehow find the devices attached -- in this case, find in the local network, who is able to speak netpp, and whether it is UDP or TCP or both. So we can consider UDP and TCP being a Hub. USB is a physically different story, but to netpp it's just another Hub.
When probing a device, a Hub must provide a probe method to list the attached and known devices. That's what satisfies the 'Hello, anybody there?' principle. Initiating a connection is another Hub method, reading and writing to the device...ok, you get the point.
So that means, we can somewhat easily implement other Hubs, if we have to.
The netpp protocol is actually very simple. It is based on a 14 byte header and an arbitrary length payload. If the payload is too big for the logical or physical packet size (defined by the Hub), it is simply split internally. We will not go into the details of the protocol, but we should mention it is completely symmetric: the master and the slave is using the same protocol state machine. This makes it easy to actually reverse the roles of two devices, provided that they're in a network: a master can be slave as well, and run two fully asynchronous connections in different threads (netpp is thread friendly, unlike many other remote procedure protocols).
So what's the deal with the protocol? It just provides a few atomic remote procedure calls (RPC):
Walk through property hierarchy ('Select')
Obtain property TOKEN by name or vice versa ('Name')
Set property to a value, get a value ('Set'/'Get')
Request/Send status ('Request'/'Status')
That's it. This makes the protocol lean and mean, and leaves much room for extensions.
Nice if we have a protocol, but how do we finally speak to the device? We need an application, or a front end that we can conveniently work with.
Here's what netpp currently offers:
a C/C++ API
a Python interface, useful for test bench development
a command line interface ('netpp')
optional wrappers for GUI toolkits
An example of the command line tool in action is shown in this little movie, found here.
In Table 1, “Memory footprints of libfiletransfer.a for various architectures”, you can see a few examples for a rudimentary file transfer protocol implementation based on netpp. All binaries were compiled with GCC, but there is support for various popular compilers. This is the stripped library, not including any static C library code (of which very little is used anyhow).
Table 1. Memory footprints of libfiletransfer.a for various architectures
|Architecture||No optimization||Optimization '-O'|
|i86 (32 bit)||54 kB||47 kB|
|x86_64||56 kB||47 kB|
|i86 Windows (mingw32)||39 kB||30 kB|
|Blackfin||37 kB||31 kB|
|armv5tel||58 kB||41 kB|
|armv5tel (thumb)||42 kB||33 kB|
Devices that normally speak netpp with all network options enabled, are normally Linux featured. It is possible to run netpp 'standalone' on top a simple UDP stack, as done for example for the Blackfin architecture. Memory usage can further be squashed by omitting the networking portions, but that is not really adviseable.
The most prominent example might be the VisionKit framework which is working in a few industrial imaging devices.
For a Linux based IP-camera prototype, it was required to use several complex system-on-chip (SoC) sensors, possibly in a multiplexed mode, to acquire images in different operating modes, depending on the environment situation. The sensors are normally controlled via I^2C (or Two Wire Interface) and have a quite extensive number of registers and properties, thus manual addressing of these properties within the program code was not an option. Nearly all I^2C register properties of all existing sensor head boards are wrapped into several sensor specific device files, which are included in a modular way into a main camera description file. The basic common functionality is defined by the camera device base class, for each pluggable sensor module, another device class is derived from the camera base class. Depending on the plugged in camera module, the 'videoserver' determines, which derived device class is to be used, and communicates the according device root node to the front end user.
Further, some SoC sensor devices use different contexts, for example, exposure time in video mode and exposure time in snapshot mode. For both operation modes, a context area is reserved, containing shadow values for a raw sensor register set. Depending on the used mode, the entire context area is copied into the raw registers.
To address properties within these contexts, the user defines a struct array which is accessed by name like: Context.ExposureTime. Offset, size and base address of these context tables can be specified in the XML description.
Let's summarize what we have learnt so far (yes, this is only a tiny window of the entire story):
We can describe hardware and generate source code and documentation from it
We can speak to a device over network or other interface and control/query its properties
We have a unified library environment that works for all netpp capable devices
We can master this development plus the test bench with freely available tools
The legitimate question is: Where does it pay off? Because: We have to learn another language, framework and technique, maybe introduce risks of getting stuck...
Let me present you a few clear Pro's that might turn out in a very positive development balance:
When you maintain/sell a family of devices with different capabilities
When you have to create an automated test bench for your device family
When you have to create an API plus a control tool to configure your device
When you have a complex device with many many registers
When you are doing a top to bottom design approach: Think first what the device should do from the user perspective, prototype the high level software on an eval kit or simulator, and when the marketing guys like the projected features, implement the real first device prototype quickly.
Now for the cons: You wouldn't probably want to give these tools a try when you:
don't like OpenSource
have less than 64kB of memory in your uC, although we managed to run a stripped down DClib on a tiny 8 bit CPU
- Have a powerful solution already (feel free to let me know!)
So you want to get your hands on this?
We set up a netpp resource site here: http://section5.ch/netppres
If you have a linux system, the source should compile from the box. It is recommended to just play around with the netpp master and the devices/example source. All other things are documented in Doxygen, you find a link to the API documentation and the detailed concepts via the above page.
Have fun and thanks for reading!
Next post by Martin Strubel:
Using GHDL for interactive simulation under Linux
Thank you so much for detailed expaltion of XML in embedded, I got lots of information about xml though ur post but still I didn't understand how XML works for Embedded Device Drivers (Not Linux Based). What is XML, how I can write configuartion file for SPI, I2C or CAN or any other embedded uController. What is the significance of XML?
If possible reply me, I have searched on net but only your page explains me some Embedded XML thing. I will wait for your answer.
To post reply to a comment, click on the 'reply' button attached to each comment. To post a new comment (not a reply to a comment) check out the 'Write a Comment' tab at the top of the comments.
Registering will allow you to participate to the forums on ALL the related sites and give you access to all pdf downloads.