RocketIO over cable

Anyone have experience with directly driving a cable with RocketIO? I
am interested in any information/experiences/advice regarding linking
two FPGAs via RocketIO over a cable. I have seen some signal
characterization information for high-speed links over copper, but
usually less than 800Mhz. I believe my implementation would use a a
less than 1 meter, but would like to know it works at 3, 5,
10...meters. Ideally I would like to run the link at 10gbits, but
6gbits could work. How feasible is this, or is it back to the drawing
board?

Thanks in advance!
Dennis

Haven't had any experience with this yet, however in anticipation of an 
upcoming project I have been looking into solutions.  So far it seems like 
the highest electrical performace solution is to use a Zd type connector and 
associated cable.  This is a relatively new connector series that has been 
incorporated in the PICMG telecom standards.  These connectors and cables 
are made by a number of companies including:

Tyco Amp:
http://catalog.tycoelectronics.com/TE/bin/TE.Connect?C=22438&F=0&M=CINF&BML=10576,16358,17560,17759,17654&GIID=0&LG=1&I=13&RQS=C~22438^M~FEAT^BML~10576,16358,17560,17759,17654^G~G

ERNI:
http://www.erni.com/ermetzdfront.htd

Gore:
http://www.gore.com/en_xx/products/cables/copper/backplane/gore_eye-opener_airmaxvs_cable_assemblies.html

Hope this helps
Brendan

"vt2001cpe" <vt2001cpe@gmail.com> wrote in message 
news:1156447377.879040.291470@b28g2000cwb.googlegroups.com...
> Anyone have experience with directly driving a cable with RocketIO? I
> am interested in any information/experiences/advice regarding linking
> two FPGAs via RocketIO over a cable. I have seen some signal
> characterization information for high-speed links over copper, but
> usually less than 800Mhz. I believe my implementation would use a a
> less than 1 meter, but would like to know it works at 3, 5,
> 10...meters. Ideally I would like to run the link at 10gbits, but
> 6gbits could work. How feasible is this, or is it back to the drawing
> board?
>
> Thanks in advance!
> Dennis
> 

Brendan Illingworth wrote:
> Haven't had any experience with this yet, however in anticipation of an
> upcoming project I have been looking into solutions.  So far it seems like
> the highest electrical performace solution is to use a Zd type connector and
> associated cable.  This is a relatively new connector series that has been
> incorporated in the PICMG telecom standards.  These connectors and cables
> are made by a number of companies including:
>
> Tyco Amp:
> http://catalog.tycoelectronics.com/TE/bin/TE.Connect?C=22438&F=0&M=CINF&BML=10576,16358,17560,17759,17654&GIID=0&LG=1&I=13&RQS=C~22438^M~FEAT^BML~10576,16358,17560,17759,17654^G~G
>
> ERNI:
> http://www.erni.com/ermetzdfront.htd
>
> Gore:
> http://www.gore.com/en_xx/products/cables/copper/backplane/gore_eye-opener_airmaxvs_cable_assemblies.html
>
> Hope this helps
> Brendan
>

Thanks for the reply! These connectors look useful. I have noticed that
people seem to have had success with FR4 connections of 40 inches or
less. In som some cases that includes transmission through a backplane
connector. Hope that helps with your application!
--Dennis

I have completed three designs now which linked V2Pro's together over a 
cable at 2.5 or ~3.1 Gb/S.

One used HSSDC cables which are used for fibre channel. Worked fine at 5M, 
although I did spec low loss and very thick TurboQuad cable.

I have also had sucess with samtec high speed flex cables (4 x 3.1 Gb)
http://www.samtec.com/high_speed_connectors/2006/SI_C2B.asp?m=hs#HSF

Finally, I ran 4 x 2.5 G over 4 x SFP connectors and copper cable 
http://www.methode.com/datamate/sfp/index.html

Basically, if you are careful with the layout and use a connector/cable 
designed for high speed signals you should be ok. I was lucky I could put 
the Xilinx right beside the connector in all cases.

Longer than a few meters, or faster than 3G would make me more nervous. You 
will have to be even more careful and do lots of testing and eye 
measurements.

Cheers,
MikeJ

vt2001cpe wrote:
> Anyone have experience with directly driving a cable with RocketIO? I
> am interested in any information/experiences/advice regarding linking
> two FPGAs via RocketIO over a cable. I have seen some signal
> characterization information for high-speed links over copper, but
> usually less than 800Mhz. I believe my implementation would use a a
> less than 1 meter, but would like to know it works at 3, 5,
> 10...meters. Ideally I would like to run the link at 10gbits, but
> 6gbits could work. How feasible is this, or is it back to the drawing
> board?

Running RocketIO over a cable isn't a problem, but of course it depends
on the cable characteristics, the data rate and acceptable losses.  We
have a number of reports on tests that were done with Virtex-II Pro
RocketIO links:

Infiniband - http://direct.xilinx.com/bvdocs/reports/ug043.pdf
SATA       - http://direct.xilinx.com/bvdocs/reports/rpt005.pdf
CAT5/5e/6  - http://direct.xilinx.com/bvdocs/reports/rpt004.pdf

Out of all of these the Infiniband style cables are the best and come in
a variety of lengths and number of channels.  I don't have any experience
with PCI Express cables, but that they should be just as good since they
use the same internal cable technology as the Infiniband cables.

http://www.molex.com/cgi-bin/bv/molex/jsp/family/intro.jsp?superFamOID=-16583&pageTitle=Introduction&channel=Products&familyOID=-16641&chanName=family&frellink=Introduction

Ed McGettigan
--
Xilinx Inc.

Thank you everyone for your quick and informative responses! In the
event of a large number of bit errors, is it possible for a comma
character to be incorrectly introduced? What would the effect be? Will
the CRC fail and put the state machine into an error state, or will
data continue to transmit, unaware?

Ed McGettigan wrote:
> vt2001cpe wrote:
> > Anyone have experience with directly driving a cable with RocketIO? I
> > am interested in any information/experiences/advice regarding linking
> > two FPGAs via RocketIO over a cable. I have seen some signal
> > characterization information for high-speed links over copper, but
> > usually less than 800Mhz. I believe my implementation would use a a
> > less than 1 meter, but would like to know it works at 3, 5,
> > 10...meters. Ideally I would like to run the link at 10gbits, but
> > 6gbits could work. How feasible is this, or is it back to the drawing
> > board?
>
> Running RocketIO over a cable isn't a problem, but of course it depends
> on the cable characteristics, the data rate and acceptable losses.  We
> have a number of reports on tests that were done with Virtex-II Pro
> RocketIO links:
>
> Infiniband - http://direct.xilinx.com/bvdocs/reports/ug043.pdf
> SATA       - http://direct.xilinx.com/bvdocs/reports/rpt005.pdf
> CAT5/5e/6  - http://direct.xilinx.com/bvdocs/reports/rpt004.pdf
>
> Out of all of these the Infiniband style cables are the best and come in
> a variety of lengths and number of channels.  I don't have any experience
> with PCI Express cables, but that they should be just as good since they
> use the same internal cable technology as the Infiniband cables.
>
> http://www.molex.com/cgi-bin/bv/molex/jsp/family/intro.jsp?superFamOID=-16583&pageTitle=Introduction&channel=Products&familyOID=-16641&chanName=family&frellink=Introduction
> 
> Ed McGettigan
> --
> Xilinx Inc.

vt2001cpe wrote:
> Thank you everyone for your quick and informative responses! In the
> event of a large number of bit errors, is it possible for a comma
> character to be incorrectly introduced? What would the effect be? Will
> the CRC fail and put the state machine into an error state, or will
> data continue to transmit, unaware?

It is possible to generate more than a single bit error in the same
8b10b word, but with a well designed link this is very unlikely to
occur.  In any case your transmitter is completely unaware that an
error actually occurred and will happily continue transmitting. It's
your receiver logic needs to be able to handle any errors that happen.

The worst case that could happen is that the error generates a comma
character that is out of alignment with the previous alignment.  This
would cause a constant stream of errors until a correct comma character
is received and the link realigns to the correct 8b10b word. All of
this is fairly standard stuff and has been around for decades in
many different protocols.

Since it doesn't look like you have a particular protocol in mind. I
would suggest that you take a look at the light weight Aurora protocol
http://www.xilinx.com/aurora  Even if you decide to not use it, you
should be able to learn a lot from it.

Ed McGettigan
--
Xilinx Inc.

Ed McGettigan wrote:
> vt2001cpe wrote:
> > Thank you everyone for your quick and informative responses! In the
> > event of a large number of bit errors, is it possible for a comma
> > character to be incorrectly introduced? What would the effect be? Will
> > the CRC fail and put the state machine into an error state, or will
> > data continue to transmit, unaware?
>
> It is possible to generate more than a single bit error in the same
> 8b10b word, but with a well designed link this is very unlikely to
> occur.  In any case your transmitter is completely unaware that an
> error actually occurred and will happily continue transmitting. It's
> your receiver logic needs to be able to handle any errors that happen.
>
> The worst case that could happen is that the error generates a comma
> character that is out of alignment with the previous alignment.  This
> would cause a constant stream of errors until a correct comma character
> is received and the link realigns to the correct 8b10b word. All of
> this is fairly standard stuff and has been around for decades in
> many different protocols.
>
> Since it doesn't look like you have a particular protocol in mind. I
> would suggest that you take a look at the light weight Aurora protocol
> http://www.xilinx.com/aurora  Even if you decide to not use it, you
> should be able to learn a lot from it.
>
> Ed McGettigan
> --
> Xilinx Inc.

As noted, much depends on the cable. I have used a lot of high speed
cables, and the best of the bunch is made by Gore (and whoever they may
have licensed their 'Eye-Opener' self equalising cable to)
http://www.gore.com/en_xx/products/cables/copper/backplane/gore_eye-opener_airmaxvs_cable_assemblies.html

I've run 4x InfiniBand over 11 metres on this type of cable and it was
within the spec (10dB   insertion loss max, amongst other things). That
was the best I ever achieved with 4x cables. I once had a cable that
could do <= 10dB attenuation of a single 2.5Gb/s pair at 17m.

The further you want to run the signal, the more expensive the cable is
going to be. Spectra Strip is usually the basis of most high speed
cable assemblies, which is not particularly cheap.

As always, the loss budget includes the transmitter and receiver specs.
I haven't seen an FPGA that was IB compliant, but could run IB over
limited distances.

Of course, if you want to run faster than that (DDR IB comes to mind ;)
then your cable length will be shortened proportionately.

Cheers

PeteS

Perhaps I did not phrase my question correctly. Assume a hypothetical
system where raw captured data from an ADC is streamed from one box to
another. The two boxes are connected via some sort of RocketIO link
over copper. Ideally, I would like to transmit just the raw samples,
but it seems some sort of overhead will be required to keep the link
established and aligned. In the event that bit errors occur, and a
comma character is falsly detected by the receiver, what happens? For
example, will the receiver try to realign and continue pumping out
data, now misaligned by the false comma, until the real comma is
detected? Or will the receiver raise an error flag, but not try and
realign. I am more interested in maintaining the throughput, and less
concerned that data is "correct". That having been said, I would like
to have correct data, but I do not think I can afford the overhead of
error correction.

Any insight or suggestions?
--Dennis

Ed McGettigan wrote:
> vt2001cpe wrote:
> > Thank you everyone for your quick and informative responses! In the
> > event of a large number of bit errors, is it possible for a comma
> > character to be incorrectly introduced? What would the effect be? Will
> > the CRC fail and put the state machine into an error state, or will
> > data continue to transmit, unaware?
>
> It is possible to generate more than a single bit error in the same
> 8b10b word, but with a well designed link this is very unlikely to
> occur.  In any case your transmitter is completely unaware that an
> error actually occurred and will happily continue transmitting. It's
> your receiver logic needs to be able to handle any errors that happen.
>
> The worst case that could happen is that the error generates a comma
> character that is out of alignment with the previous alignment.  This
> would cause a constant stream of errors until a correct comma character
> is received and the link realigns to the correct 8b10b word. All of
> this is fairly standard stuff and has been around for decades in
> many different protocols.
>
> Since it doesn't look like you have a particular protocol in mind. I
> would suggest that you take a look at the light weight Aurora protocol
> http://www.xilinx.com/aurora  Even if you decide to not use it, you
> should be able to learn a lot from it.
> 
> Ed McGettigan
> --
> Xilinx Inc.

vt2001cpe schrieb:
> Perhaps I did not phrase my question correctly. Assume a hypothetical
> system where raw captured data from an ADC is streamed from one box to
> another. The two boxes are connected via some sort of RocketIO link
> over copper. Ideally, I would like to transmit just the raw samples,
> but it seems some sort of overhead will be required to keep the link
> established and aligned. In the event that bit errors occur, and a
> comma character is falsly detected by the receiver, what happens? For
> example, will the receiver try to realign and continue pumping out
> data, now misaligned by the false comma, until the real comma is
> detected? Or will the receiver raise an error flag, but not try and
> realign. I am more interested in maintaining the throughput, and less

It will detect a code error and raise a flag. Dependig on configuration 
the receiver will try to realign or just continue sampling data. Usually 
a state machine will keep a look at the code error flag and force a 
realign after 3 consecutive errored codes. Iam not sure if this state 
machine is already implemented inside the MGTs, probably you have to 
create it yourself using FPGA logic. Should be no big deal.

Regards
Falk