FPGA on-die LVDS termination issues

Hello,

We use several source-synchronous LVDS-based interfaces in several PCB
designs, such as a unidirectional interface from several multi-channel ADC
devices (TI ADS725x) towards FPGA (Xilinx Virtex4 family, SX subfamily), or
a full-duplex interface between high-end DSP devices (ADI ADSP-TS201S) and
FPGA (Xilinx Virtex4 family, FX subfamily).
In both mentioned cases, we use on-die LVDS termination on the FPGA side,
e.g. the LVDS receiver side. The used I/O standard is LVDS25, with the
attribute DIFF_TERM set to TRUE. In both cases, the (data)bitrate is about
500 MHz, both using DDR techniques, e.g. the clock-rate is half of this
(250 MHz).
The data-clock provided by the ADCs to FPGAs has a continous nature, the
data-clock provided by the DSPs to the FPGAs is non-continuous, e.g. only
active when there are databits (1 or 4 bits) to be transmitted, else in an
idle state.
In all cases, we have performed SI simulations with IBIS models provided by
the vendors (TI/ADI/Xilinx), and the exact transmission line details
extracted from our PCB layout, e.g. microstrip/vias/stripline, etc. Of
course, the nets have been routed according to the requirements, e.g. with
correct differential impedance, matched lengths, etc. etc.
SI simulation was succesful, simulated eye-diagrams etc. were OK.
Note: the on-die termination resistor (100 ohm) was not included in the
IBIS model of the Xilinx LVDS receiver, and needed to be added manually.
Based on the good simulation results and the overall lack of available PCB
area, we decided to use on-die termination.

On the physical target boards however, we encountered difficulties on these
LVDS interfaces, and the measured signals do not look good at all.
In the time-domain, we are measuring (with Lecroy SDA) RC-like curves on
the (data)signals, when they have been stable (either 0 or 1) for a while
and when they start switching again (to 1 or 0 respectively). 
We are aware that the 100 ohm on-die termination "resistor" is in reality
not a physical resistor between the P- and N terminal(s) of the LVDS
pair(s), at least not in case of FPGA devices.
In reality, both terminals have a Thevenin-equivalent circuit, each
consisting of two other "resistors", one from the terminal upto the I/O
supply voltage, and another downto ground. In reality, these "resistors"
are one or more FETs, switched on/off by the FPGA configuration bitstream
to achieve the right termination scheme for the chosen I/O standard.
We are almost running out of things we can still check, e.g. all FPGA
connections, power supply/integrity, decoupling, trace impedances, FPGA
configuration, etc. etc. 
The FPGA vendor is involved since several weeks now, but they seem to be as
flabbergasted by our observations as we are ourselves.... ;-)


My questions are:

- does anybody as a user of FPGA devices (or LVDS in general) recognize the
observations mentioned above, preferably from own experience/measurements?

- are the assumptions regarding the internal FPGA termination circuits
correct?

- can it be than any non-linearity of the Thevenin FETs causes these
problems?   

- any other brilliant ideas, hints, etc.

Any help is welcome.

Best regards,

Jaap

Jaap,

The waveforms will look different at any point OTHER than across the
termination.

You can see this for yourself in the simulation, by placing the
package element in the simulation (i.e. two short -- 10-20mm t-lines
before the termination and IO pin loading model).

If you have already done this, then you are aware of where you look
influences what you see.

Looking directly across the termination is what the receiver sees, so
that is what matters.

The termination is not a carbon resistor, but it is as good as one
when it comes to looking like a resistor, so that is not the issue.
Often, the attribute is not set properly, and the resistor is not
enabled.  Have you checked in FPGA_editor, and do you clearly see the
resistor termination enabled?  Does the receive voltage appear twice
as high as it is indicated in the simulation? (clearly indicating the
resistor is not enabled)

You do not mention the problem:  bad data, occasional incorrect data,
bad data when other IOs switch only, etc.

As a customer, the magic words are "lines down."  If you say this, the
case MUST be escalated.  If unresolved, it must be escalated again and
again, until it gets to the "Fire Marshall"  who reports to the Senior
VP and CEO on unresolved cases, and their status.

Since I invented, implemented, this system, and was the first Fire
Marshall, I am very familiar with the system, and it works really well
-- use it!

A case number is very useful:  if you email it to me, I can check on
its status, and help get it escalated.

Jaap,

You can't just probe in the middle of the trace and expect to see on 
your 'scope what the receiver sees. The 'scope will introduce an 
impedance discontinuity. What probe are you using?

This is particularly the case with FPGAs; the Xilinx Virtex 4 has about 
10pF of input capacitance per pin which is produced by all those other 
IOBs configuration options, specifically 24mA output FETs. High speed 
signals hit those 'caps' and reflect back.

If you really want to probe the signals, you could add an attenuator to 
your design. So, is it working or not. You don't seem to describe your 
problem except the signal doesn't look good on your 'scope.

HTH., Syms.

p.s. Brian Davis posted on CAF about using attenuators for probing, you 
might be able to google for it.

>Jaap,
>
>The waveforms will look different at any point OTHER than across the
>termination.
>
>You can see this for yourself in the simulation, by placing the
>package element in the simulation (i.e. two short -- 10-20mm t-lines
>before the termination and IO pin loading model).
>
>If you have already done this, then you are aware of where you look
>influences what you see.
>
>Looking directly across the termination is what the receiver sees, so
>that is what matters.

OK, clear from a simulation point of view. We are measuring as close as
possible to the receiver e.g. the termination, but this is not always
possible, due to the location of via's (micro-strip to stripline and v.v.),
which may not always be as close to the termination as you would like them
to be.
Anyhow, what we see is not a "normal" reflection, but a RC-curve on top of
the reflection. This RC-curve is "killing", the reflection itself is more
or less as expected. If we manually place a 100 ohm resistor on the board,
and disable the on-die termination, the RC-curve has disappeared, and the
signal looks as neat as you can possibly expect. In this case, the
reflection is there but very OK, and the eye-diagram is well open. 
The problem is that we need about 40 of these extra 100 ohm resistors, on a
board which is already loaded....

>
>The termination is not a carbon resistor, but it is as good as one
>when it comes to looking like a resistor, so that is not the issue.
>Often, the attribute is not set properly, and the resistor is not
>enabled.  Have you checked in FPGA_editor, and do you clearly see the
>resistor termination enabled?  

Yes, we did this check, the attribute was set both in the VHDL as in the
UCF, and we checked the results using the FPGA Editor, and the pinning
file.

>Does the receive voltage appear twice
>as high as it is indicated in the simulation? (clearly indicating the
>resistor is not enabled)
>
>You do not mention the problem:  bad data, occasional incorrect data,
>bad data when other IOs switch only, etc.

Basically all of the items mentioned above. Bad data could be the result of
many issues (internal FPGA I/O timing, SSN, clock jitter requirements,
etc.).
We are trying to eliminate them one by one, to narrow down on the real
cause(s). 

>
>As a customer, the magic words are "lines down."  If you say this, the
>case MUST be escalated.  If unresolved, it must be escalated again and
>again, until it gets to the "Fire Marshall"  who reports to the Senior
>VP and CEO on unresolved cases, and their status.
>
>Since I invented, implemented, this system, and was the first Fire
>Marshall, I am very familiar with the system, and it works really well
>-- use it!
>
>A case number is very useful:  if you email it to me, I can check on
>its status, and help get it escalated.

OK, thanks. As I said, Xilinx is already involved, a RMA procedure was
started but temporary halted on our request because we needed more time to
do our homework. Until now, we haven't been able to find anything that we
have done wrong ourselves. Today, I have been working with some collegues
to connect a DSP evaluation board (ADI EZ-Lite TS201) with LVDS-based DSP
Link Ports direcly to a FPGA evaluation board (Xilinx ML403), e.g. without
any hardware being designed by ourselves (only interconnect consisting of
two RJ45/UTP cables). The FPGA only will contain a loopback (TX to RX and
v.v.), elimination any possible internal timing issues.
If we encounter the same RC-curves on this design, we can stop our attempts
to find the error in our own designs (PCB/board/FPGA), and
redesign/relayout all our boards, which will be extremly costly and also
time consuming. We have already started these redesigns (in parralel), to
reduce leadtime and risk.

We are finishing a report on our measurements, simulations, etc this week,
and we will forward it to our Xilinx representatives.


>
>	   
					
---------------------------------------		
This message was sent using the comp.arch.fpga web interface on
http://www.FPGARelated.com

>Jaap,
>
>You can't just probe in the middle of the trace and expect to see on 
>your 'scope what the receiver sees. The 'scope will introduce an 
>impedance discontinuity. What probe are you using?

We are aware of that. It is impossible to probe at the ideal location, due
to the layout, vias positions, etc.
We are using a LeCroy SDA 6000A with a D600A differential probe. I believe
these devices suitable (if not over-qualified) for these measurements....
;-)

>
>This is particularly the case with FPGAs; the Xilinx Virtex 4 has about 
>10pF of input capacitance per pin which is produced by all those other 
>IOBs configuration options, specifically 24mA output FETs. High speed 
>signals hit those 'caps' and reflect back.

Is this 10 pF input capacitance per pin based on having on-die termination
enabled, disabled or perhaps both?

>
>If you really want to probe the signals, you could add an attenuator to 
>your design. So, is it working or not. You don't seem to describe your 
>problem except the signal doesn't look good on your 'scope.

The problem is bad/corrupted data (occasionally), for more details see my
follow-up to the Austin in this same thread.

>
>HTH., Syms.
>
>p.s. Brian Davis posted on CAF about using attenuators for probing, you 
>might be able to google for it.

OK, thanks, we will do that.
>
>
>	   
					
---------------------------------------		
This message was sent using the comp.arch.fpga web interface on
http://www.FPGARelated.com

On Tue, 13 Oct 2009 14:10:34 -0500, "dc207" <jaap.mol@planet.nl> wrote:


>Anyhow, what we see is not a "normal" reflection, but a RC-curve on top of
>the reflection. This RC-curve is "killing", the reflection itself is more
>or less as expected. If we manually place a 100 ohm resistor on the board,
>and disable the on-die termination, the RC-curve has disappeared, and the
>signal looks as neat as you can possibly expect.

... but do the data errors disappear?

I may have a suspicious mind but it looks possible that the visible reflections
are a red herring, and the internal termination works just fine ... but
something else is causing data corruption.

The best time to discover that is _not_ when you get the revised board layout
back...

- Brian

dc207 wrote:
>> Jaap,
>>
>> You can't just probe in the middle of the trace and expect to see on 
>> your 'scope what the receiver sees. The 'scope will introduce an 
>> impedance discontinuity. What probe are you using?
> 
> We are aware of that. It is impossible to probe at the ideal location, due
> to the layout, vias positions, etc.
> We are using a LeCroy SDA 6000A with a D600A differential probe. I believe
> these devices suitable (if not over-qualified) for these measurements....
> ;-)
> 
>> This is particularly the case with FPGAs; the Xilinx Virtex 4 has about 
>> 10pF of input capacitance per pin which is produced by all those other 
>> IOBs configuration options, specifically 24mA output FETs. High speed 
>> signals hit those 'caps' and reflect back.
> 
> Is this 10 pF input capacitance per pin based on having on-die termination
> enabled, disabled or perhaps both?
> 
>> If you really want to probe the signals, you could add an attenuator to 
>> your design. So, is it working or not. You don't seem to describe your 
>> problem except the signal doesn't look good on your 'scope.
> 
> The problem is bad/corrupted data (occasionally), for more details see my
> follow-up to the Austin in this same thread.
> 
>> HTH., Syms.
>>
>> p.s. Brian Davis posted on CAF about using attenuators for probing, you 
>> might be able to google for it.
> 
> OK, thanks, we will do that.
>>
>> 	   
> 					
> ---------------------------------------		
> This message was sent using the comp.arch.fpga web interface on
> http://www.FPGARelated.com

Hi Jaap,
OK, it sounds like you are more than a little experienced at this, 
apologies for teaching you to suck eggs with your probing! Also, 
apologies in advance if the following is all obvious!

So, each pin has 10pF loading on it no matter what the IOB is set up 
for. Every single mode. There are a bunch of FET's connected to the 
signal to drive signals out if required by the configuration loaded into 
the device. Even if these are turned off, for example when the pin is an 
input, the capacitance is still there. It is the capacitance of the 
FETs' structure. Look at a datasheet for a discrete FET to see why this is.

This should be modelled in the IBIS file. You could try simulating with 
the IBIS model replaced by a 10pF cap to ground, along with the 100 ohm 
resistive termination between pins. If this sim looks the 'scope 
picture, maybe the IBIS model isn't accurate, but I doubt this.

FWIW, I've used the on chip termination for Virtex 4 FX devices many, 
many times without problems, within these limits:-

< 667MBps data rate
< 6 inches trace length.

How good are your power supplies? Are your pairs tightly coupled? Are 
they routed away from any crosstalk hazards? Are the pairs routed on a 
layer next to a ground plane? Do they swap layers a lot?

HTH, Symon.

Jaap,
Not much time to type tonight, but here's something:

-----------------

A. "RC curves" on LVDS signals

>
>  In the time-domain, we are measuring (with Lecroy SDA)
> RC-like curves on the (data)signals, when they have been
> stable (either 0 or 1) for a while and when they start
> switching again (to 1 or 0 respectively).
>

 This sounds _exactly_ like what you would see if the on board
FPGA terminator was not enabled.

 Many of these high speed A/D parts have LVDS outputs without
any internal back termination, so the A/D driver current sources
will run off into the rails and saturate on a long string of
1's or 0's without an external termination of some sort- this
sounds just like what you are seeing.

 I would concentrate on verifying, in hardware, that the
terminations are REALLY enabled, see section C.

 IIRC the ADS527x does not have the switchable internal back
terminations like other TI A/D parts, but it does have the
double-strength LVDS drive setting.

 If you respin the board, I would highly recommend enabling
that control bit and implementing either the back termination
at the A/D or a differential attenuator (see links) to provide
better matching when driving a V4 FPGA; this will also make
it easier to make meaningful measurements with an external probe.

-----------------

B. DIFF_TERM attribute problems

>
> the attribute was set both in the VHDL as in the UCF, and we
>checked the results using the FPGA Editor, and the pinning file.
>
 What version of software are you using?
 What part are you targeting?

 A bug later on in the flow ( e.g. Bitgen ) could have this effect
on the hardware yet still show the terminations in the FPGA editor.

 Do both sides of the differential buffer ( DIFFM/DIFFS ) show the
termination attribute enabled in the FPGA editor?

 There have been a number of past bugs with the DIFF_TERM's getting
dropped somewhere in the flow unless they are done with exactly
the right incantation.

 This is particularly true for the DIFF_OUT variants of the LVDS
input buffers, which IIRC are used in some of the high speed
serializer app notes to provide two copies of the internal data
for better routing delays.

 Despite a 6-7 year old CR reporting this problem, the last time
I checked the bug was still there if the DIFF_TERM attributes were
applied directly to a DIFF_OUT buffer in the HDL source code.

-----------------

C. DIFF_TERM implementation

>
>  In reality, both terminals have a Thevenin-equivalent circuit,
> each consisting of two other "resistors", one from the terminal
> up to the I/O supply voltage, and another downto ground.
>
 Note, the _original_ V2 LCDS_25_DCI termination hack worked
like this, but the DIFF_TERM's are implemented differently, and
look more like ~100 ohm across the pair for signals within the
specified common mode input voltage range.

 I have some half-finished LVDS notes that show a quasi-differential
sweep of a V4 DIFF_TERM done with a single-ended curve tracer and a
couple resistors, I'll try to post them in a few days; see the link
"DIFF_TERM measurements" link below for a written description.

 You can do something similar with a voltage source and a couple
resistors to confirm the presence of the internal termination
in the hardware.

-----------------

D. IBIS modeling of Xilinx differential terminations

>
>Note: the on-die termination resistor (100 ohm) was not included in the
>IBIS model of the Xilinx LVDS receiver, and needed to be added manually.
>

 If you are using an old version of HyperLynx, you may need a fresher
copy.

 The Xilinx IBIS terminator models have been broken in HyperLynx on
both occasions that I've looked into it ( see link below); simulating
the termination as an external resistor will not be an exact model,
but it is probably close enough to spot many problems.

-----------------

Past LVDS Posts and Links ( also see surrounding threads ):

Note that my file links referenced in the following posts
have moved, as AOL silently axed their FTP site last fall :

 http://fpgastuff.googlepages.com/oldaolfiles

 http://fpgastuff.googlepages.com/lvds_current.pdf
 http://fpgastuff.googlepages.com/lvds_current.zip

ADS527x and Xilinx LVDS inputs:
  http://groups.google.com/group/comp.arch.fpga/msg/95809af82ccbb550
  http://groups.google.com/group/comp.arch.fpga/msg/ab999f47d42e50f8
  http://groups.google.com/group/comp.arch.fpga/msg/5a8720eec942612e
  http://groups.google.com/group/comp.arch.fpga/msg/8ad1d05fa4d14e77

DIFF_TERM lab measurements
  http://groups.google.com/group/comp.arch.fpga/msg/dd70f9f0bea2cbe0

S3E DIFF_TERM tool quirks:
  http://groups.google.com/group/comp.arch.fpga/msg/45050b8239cecc56

Xilinx/Hyperlnyx differential termination bugs:
  http://groups.google.com/group/comp.arch.fpga/msg/aa97cb00c9aa721a
  http://groups.google.com/group/comp.arch.fpga/msg/c6e28cb7cc0ce3d0

good luck,
Brian

Jaap,

An RMA is NOT the right way to go.

Often people request an RMA when it is a technical problem, not an
issue of wanting to return a part for failure analysis.

GET OUT OF THE RMA queue, and get into the "solve the problem" queue
by entering a webcase.

We can't help you if you ask us to do the wrong things.

RMA process is to find a supposed fault on a part.  Given that faults
on V5 parts are running less than 1 ppm right now, unless this is just
one part with this problem, and you are sure that it is bad, do not
request an RMA.  Different people, different skills:  they can not
find the problem on your board, nor do they even know how to.


Email me at austin@xilinx.com with the details (company, names, any
case numbers) and I will help get this properly assigned.  If you are
in the RMA queue, I doubt that you will get any resolution until they
reassign this to the right people.

You never answered if the voltage swing is the same with the resistor,
and also with the LVDS termination.

You also never answered if you did the simulation of the resistor at
the receiver, or 19-20mm away from the receiver (always observing
10-20mm away from the receiver).  It could be you are just chasing
this small reflection, that isn't the problem at all!

>On Tue, 13 Oct 2009 14:10:34 -0500, "dc207" <jaap.mol@planet.nl> wrote:
>
>
>>Anyhow, what we see is not a "normal" reflection, but a RC-curve on top
of
>>the reflection. This RC-curve is "killing", the reflection itself is
more
>>or less as expected. If we manually place a 100 ohm resistor on the
board,
>>and disable the on-die termination, the RC-curve has disappeared, and
the
>>signal looks as neat as you can possibly expect.
>
>... but do the data errors disappear?

Yes, the data errors disappear like "snow in the sun" with the external
(on-board) termination resistor....


>
>I may have a suspicious mind but it looks possible that the visible
reflections
>are a red herring, and the internal termination works just fine ... but
>something else is causing data corruption.

Today, we managed to get the ADI TS201 EZ-lite DSP eval board connected to
the Xilinx ML403 board running without data errors, using the LVDS-based
DSP Link Port, two RJ45/UTP cable to interconnect the DSP to the FPGA, and
a simple loopback implemented in the Virtex4 (FX) FPGA. 
This is a confirmation that we are doing something wrong, but we have
(re)verified so many possible causes already, and are running out of
options.
Tomorrow we will be testing a completly stripped-down FPGA, e.g. only with
loopbacks implemented on multiple DSP Link Ports and absolutelly nothing
else. As if we were connecting 6 UTP loopback cables to each DSP Link
Port...
We will see what will we measure.
Meanwhile, somebody is performing Power Integrity simulations (Hyperlynx
PI) on our PCB design, but until now the tool crashes due to shortage of
memory, e.g. the design seems too big even for high-end (memory-rich) PC
platforms (?).
Sending the design to Mentor requires a signed NDA, which will probably
take weeks to complete (based on our experiences with several NDA
contracts)... ;-(

>
>The best time to discover that is _not_ when you get the revised board
layout
>back...
>
>- Brian
>
>	   
					
---------------------------------------		
This message was sent using the comp.arch.fpga web interface on
http://www.FPGARelated.com