high bandwitch ethernet communication

Hello,

In our application we have to receive and merge several proprietary
serial channels (200 MHz) over fibers, and send all the data over
Gigabit Ethernet. The bandwidth is ~60 MByte/s, sustained.

While generally sending this amount of data is possible over Gbit
Ethernet, doing so in an embedded system isn't easy. That's because we
need to send it by UDP or TCP, for which a TCP/UDP/IP stack is
required (software).

Since the translation of the proprietary format is certainly done in
an FPGA, I tried to calculate how to implement the whole process in an
FPGA. For example, I can take an Altera Stratix II GX (with a built in
Gbit Ethernet PHY), add Altera's MAC and use a TCP/IP stack running on
the Nios II soft-core processor. Unfortunately, as Altera's appnote
440 shows, the maximal bandwidth attainable this way is only 15-17
MByte/s. For the sake of comparison, benchmarks of Gbit Ethernet
adapters on PCs show a maximal bandwidth of 80-90 MByte/s.

However, I wouldn't like to build in a Pentium into the embedded
system. Any suggestions / recommendations on how to solve the
problem ?

Thanks in advance

On Wed, 05 Sep 2007 14:59:30 +0000, eliben wrote:

> Hello,
> 
> In our application we have to receive and merge several proprietary
> serial channels (200 MHz) over fibers, and send all the data over
> Gigabit Ethernet. The bandwidth is ~60 MByte/s, sustained.
> 
> While generally sending this amount of data is possible over Gbit
> Ethernet, doing so in an embedded system isn't easy. That's because we
> need to send it by UDP or TCP, for which a TCP/UDP/IP stack is
> required (software).
> 
> Since the translation of the proprietary format is certainly done in
> an FPGA, I tried to calculate how to implement the whole process in an
> FPGA. For example, I can take an Altera Stratix II GX (with a built in
> Gbit Ethernet PHY), add Altera's MAC and use a TCP/IP stack running on
> the Nios II soft-core processor. Unfortunately, as Altera's appnote
> 440 shows, the maximal bandwidth attainable this way is only 15-17
> MByte/s. For the sake of comparison, benchmarks of Gbit Ethernet
> adapters on PCs show a maximal bandwidth of 80-90 MByte/s.
> 
> However, I wouldn't like to build in a Pentium into the embedded
> system. Any suggestions / recommendations on how to solve the
> problem ?
> 
> Thanks in advance

My first choice would be some other, more embeddable, processor running
off to the side.  A PowerPC from Freescale, or an ARM processor from just
about anybody, comes to mind.  I suspect that even a modest such processor
would get some pretty high speeds if that's all it was doing.

You may have to bite the bullet and write your own stack that's shared
between a processor and the FPGA.  I know practically nothing about
TCP/IP, but I'm willing to bet that once you've signed up to writing
or modifying your own stack there are some obvious things to put into the
FPGA to speed things up.

Slapping a big, limited temperature range, power hungry Pentium into an
embedded product would not be my first choice either.

-- 
Tim Wescott
Control systems and communications consulting
http://www.wescottdesign.com

Need to learn how to apply control theory in your embedded system?
"Applied Control Theory for Embedded Systems" by Tim Wescott
Elsevier/Newnes, http://www.wescottdesign.com/actfes/actfes.html

On Sep 5, 10:59 am, eliben <eli...@gmail.com> wrote:
> Hello,
>
> In our application we have to receive and merge several proprietary
> serial channels (200 MHz) over fibers, and send all the data over
> Gigabit Ethernet. The bandwidth is ~60 MByte/s, sustained.
>
> While generally sending this amount of data is possible over Gbit
> Ethernet, doing so in an embedded system isn't easy. That's because we
> need to send it by UDP or TCP, for which a TCP/UDP/IP stack is
> required (software).
>
> Since the translation of the proprietary format is certainly done in
> an FPGA, I tried to calculate how to implement the whole process in an
> FPGA. For example, I can take an Altera Stratix II GX (with a built in
> Gbit Ethernet PHY), add Altera's MAC and use a TCP/IP stack running on
> the Nios II soft-core processor. Unfortunately, as Altera's appnote
> 440 shows, the maximal bandwidth attainable this way is only 15-17
> MByte/s. For the sake of comparison, benchmarks of Gbit Ethernet
> adapters on PCs show a maximal bandwidth of 80-90 MByte/s.
>
> However, I wouldn't like to build in a Pentium into the embedded
> system. Any suggestions / recommendations on how to solve the
> problem ?
>
> Thanks in advance


Check out Stretch http://www.stretchinc.com

They have processors with 4 gigabit ethernet ports and a
hardware-assisted stack that can keep up at full gigE
bandwidth on at least 3 at the same time.  Getting data
into the Stretch processor memory can be via the
"coprocessor bus" interface or by using one of the
MAC interfaces as a simple FIFO.

On Sep 5, 9:59 am, eliben <eli...@gmail.com> wrote:
> Hello,
>
> In our application we have to receive and merge several proprietary
> serial channels (200 MHz) over fibers, and send all the data over
> Gigabit Ethernet. The bandwidth is ~60 MByte/s, sustained.
>
> While generally sending this amount of data is possible over Gbit
> Ethernet, doing so in an embedded system isn't easy. That's because we
> need to send it by UDP or TCP, for which a TCP/UDP/IP stack is
> required (software).
>
> Since the translation of the proprietary format is certainly done in
> an FPGA, I tried to calculate how to implement the whole process in an
> FPGA. For example, I can take an Altera Stratix II GX (with a built in
> Gbit Ethernet PHY), add Altera's MAC and use a TCP/IP stack running on
> the Nios II soft-core processor. Unfortunately, as Altera's appnote
> 440 shows, the maximal bandwidth attainable this way is only 15-17
> MByte/s. For the sake of comparison, benchmarks of Gbit Ethernet
> adapters on PCs show a maximal bandwidth of 80-90 MByte/s.
>
> However, I wouldn't like to build in a Pentium into the embedded
> system. Any suggestions / recommendations on how to solve the
> problem ?
>
> Thanks in advance



If you have the choice between UDP and TCP, UDP is much simpler and
fits an FPGA well. The big issue in choosing between the two is if you
require the guaranteed delivery of TCP, or can tollerate the potential
packet loss of UDP.

As an example, we make a card that acquires real time data in a custom
protocol that is wrapped in UDP. We use a Xilinx Virtex-4 FX60, and a
protocol offload engine that uses the Xilinx PicoBlaze soft processor
to deal with the protocol stack.  The PicoBlaze is an 8-bit soft
processor.  It looks at each incomming packet and reads the header to
see if it is one of the real time streams we are trying to offload.
If it is, it sends the header to one circular buffer in memory and the
data to another circular buffer.  If it is not, it sends it to a
kernel buffer and we let the Linux network stack deal with it.

With this setup, we can consume data at over 90 MB/sec per Gigabit
Ethernet port.  The data part of the packet is 1024 bytes, and each
GigE port has its own PicoBlaze dedicated to it.

I did notice that you want to send GigE instead of receive it like we
are doing, but this method should work for sending a custom protocol
wrapped in UDP with some minor changes.

How is the GigE that you are sending the data over connected? Is it
point to point, a dedicated network, or something else?

The network that the data we deal with is set up as multicast data on
VLANs.  The VLANs are allocated to guantee the required bandwidth
through the switches, and data sources use traffic shapping to put out
their packets at a steady rate. In that environment, UDP works just
fine.

This is the card that I am talking about:

http://www.fastertechnology.com/products_p6.html

Regards,

John McCaskill,
www.fastertechnology.com

eliben wrote:
> Hello,
> 
> In our application we have to receive and merge several proprietary
> serial channels (200 MHz) over fibers, and send all the data over
> Gigabit Ethernet. The bandwidth is ~60 MByte/s, sustained.
> 
> While generally sending this amount of data is possible over Gbit
> Ethernet, doing so in an embedded system isn't easy. That's because we
> need to send it by UDP or TCP, for which a TCP/UDP/IP stack is
> required (software).
> 
> Since the translation of the proprietary format is certainly done in
> an FPGA, I tried to calculate how to implement the whole process in an
> FPGA. For example, I can take an Altera Stratix II GX (with a built in
> Gbit Ethernet PHY), add Altera's MAC and use a TCP/IP stack running on
> the Nios II soft-core processor. Unfortunately, as Altera's appnote
> 440 shows, the maximal bandwidth attainable this way is only 15-17
> MByte/s. For the sake of comparison, benchmarks of Gbit Ethernet
> adapters on PCs show a maximal bandwidth of 80-90 MByte/s.
> 
> However, I wouldn't like to build in a Pentium into the embedded
> system. Any suggestions / recommendations on how to solve the
> problem ?
> 

Try the Xilinx GSRD appnote. Performance of over 700 Mbps is possible 
with the embedded PowerPC.

/Siva

eliben wrote:
(snip)

> While generally sending this amount of data is possible over Gbit
> Ethernet, doing so in an embedded system isn't easy. That's because we
> need to send it by UDP or TCP, for which a TCP/UDP/IP stack is
> required (software).

TCP is somewhat complicated, but UDP is pretty simple.  You might
want external software to handle ARP and routing tables (if needed),
but it should not be too hard to create and send a UDP packet
in an FPGA.  You don't say anything about receiving.
Among the complications of IP are reassembling fragmented
packets.  You should be able to avoid that (if you control both
ends, and the path in between).

-- glen

We've are using the MPC8349E at 400Mhz core and got only 480mbit/s
sustained UDP data rate.  These processors are marketed as
communications processors but only have low level HW support (on
Ethernet layer).  All the upper level IP and UDP protocols are handled
in software (when running linux).  So it takes up CPU time.  Same
setup on two desktop PCs running linux yeilds 840mbit/s sustained UDP
rate.

> My first choice would be some other, more embeddable, processor running
> off to the side.  A PowerPC from Freescale, or an ARM processor from just
> about anybody, comes to mind.  I suspect that even a modest such processor
> would get some pretty high speeds if that's all it was doing.
>
> You may have to bite the bullet and write your own stack that's shared
> between a processor and the FPGA.  I know practically nothing about
> TCP/IP, but I'm willing to bet that once you've signed up to writing
> or modifying your own stack there are some obvious things to put into the
> FPGA to speed things up.

Thanks, this is the design I'm now leaning towards. However, I have a
concern regarding the high-speed communication between the FPGA and
the outside Processor. How is it done - using some high speed outside
DDR memory ?

Eli

> If you have the choice between UDP and TCP, UDP is much simpler and
> fits an FPGA well. The big issue in choosing between the two is if you
> require the guaranteed delivery of TCP, or can tollerate the potential
> packet loss of UDP.
>
> As an example, we make a card that acquires real time data in a custom
> protocol that is wrapped in UDP. We use a Xilinx Virtex-4 FX60, and a
> protocol offload engine that uses the Xilinx PicoBlaze soft processor
> to deal with the protocol stack.  The PicoBlaze is an 8-bit soft
> processor.  It looks at each incomming packet and reads the header to
> see if it is one of the real time streams we are trying to offload.
> If it is, it sends the header to one circular buffer in memory and the
> data to another circular buffer.  If it is not, it sends it to a
> kernel buffer and we let the Linux network stack deal with it.
>
> With this setup, we can consume data at over 90 MB/sec per Gigabit
> Ethernet port.  The data part of the packet is 1024 bytes, and each
> GigE port has its own PicoBlaze dedicated to it.

So where is the actual UDP communication implemented ? In Linux ? What
processor is it running on ? Is it an external CPU or the built-in PPC
of Virtex 4 ?

>
> I did notice that you want to send GigE instead of receive it like we
> are doing, but this method should work for sending a custom protocol
> wrapped in UDP with some minor changes.
>
> How is the GigE that you are sending the data over connected? Is it
> point to point, a dedicated network, or something else?
>

We can assume for the sake of discussion that it is point to point,
since the network is small and we're likely to use a fast switch to
ensure exclusive links between systems.

Eli

On Thu, 06 Sep 2007 05:20:14 +0000, eliben wrote:

>> My first choice would be some other, more embeddable, processor running
>> off to the side.  A PowerPC from Freescale, or an ARM processor from just
>> about anybody, comes to mind.  I suspect that even a modest such processor
>> would get some pretty high speeds if that's all it was doing.
>>
>> You may have to bite the bullet and write your own stack that's shared
>> between a processor and the FPGA.  I know practically nothing about
>> TCP/IP, but I'm willing to bet that once you've signed up to writing
>> or modifying your own stack there are some obvious things to put into the
>> FPGA to speed things up.
> 
> Thanks, this is the design I'm now leaning towards. However, I have a
> concern regarding the high-speed communication between the FPGA and
> the outside Processor. How is it done - using some high speed outside
> DDR memory ?
> 
> Eli

I've always seen it implemented as a plain ol' asynchronous memory
interface, as seen in the '70's and '80's.  Most processors support it,
and it's not too shabby.

If you have to have the processor fondling the data bits then mapping the
FPGA as synchronous static RAM or SDRAM may be quicker, but it'll
certainly be more problematic.

-- 
Tim Wescott
Control systems and communications consulting
http://www.wescottdesign.com

Need to learn how to apply control theory in your embedded system?
"Applied Control Theory for Embedded Systems" by Tim Wescott
Elsevier/Newnes, http://www.wescottdesign.com/actfes/actfes.html