same RTL on two same boards giving different behaviour

Hi,
I am using spartan3 xc3s4000 custom board in my design interfaced with a
national PHY DP83865, xilinx 12.3 for synthesis and implementation and i'm
facing a strange problem. I run the same RTL on two boards and it behaves
differently on both of them. Has anyone faced this issue before ? Does
anyone know why it's happening ?



Thanks

Regards
Salimbaba	   
					
---------------------------------------		
Posted through http://www.FPGARelated.com

On Apr 17, 11:06=A0am, "salimbaba"
<a1234573@n_o_s_p_a_m.n_o_s_p_a_m.owlpic.com> wrote:
> Hi,
> I am using spartan3 xc3s4000 custom board in my design interfaced with a
> national PHY DP83865, xilinx 12.3 for synthesis and implementation and i'=
m
> facing a strange problem. I run the same RTL on two boards and it behaves
> differently on both of them. Has anyone faced this issue before ? Does
> anyone know why it's happening ?
>
> Thanks
>
> Regards
> Salimbaba =A0 =A0 =A0 =A0 =A0
>
> --------------------------------------- =A0 =A0 =A0 =A0
> Posted throughhttp://www.FPGARelated.com

The most common reason is timing failures in the design due to:
- Lack of any timing constraints
- Missing IO timing constraints for device-to-device timing
- Data transfers between asynchronous clocks without synchronization
- Use of asynchronous resets
- Latches in the design due to HDL errors

I would start with reviewing all of the WARNINGs created by the
synthesizer and ISE tools and then move on to the unconstrained paths
reported by the timing analyzer.

Ed McGettigan
--
Xilinx Inc.

>Hi,
>I am using spartan3 xc3s4000 custom board in my design interfaced with a
>national PHY DP83865, xilinx 12.3 for synthesis and implementation and
i'm
>facing a strange problem. I run the same RTL on two boards and it behaves
>differently on both of them. Has anyone faced this issue before ? Does
>anyone know why it's happening ?

Do you know that both boards are built correctly?
Could it be a board hardware fault?
Does either board behave 'correctly'?
	   
					
---------------------------------------		
Posted through http://www.FPGARelated.com

>>Hi,
>>I am using spartan3 xc3s4000 custom board in my design interfaced with a
>>national PHY DP83865, xilinx 12.3 for synthesis and implementation and
>i'm
>>facing a strange problem. I run the same RTL on two boards and it
behaves
>>differently on both of them. Has anyone faced this issue before ? Does
>>anyone know why it's happening ?
>
>Do you know that both boards are built correctly?
>Could it be a board hardware fault?
>Does either board behave 'correctly'?
>	   
>					
>---------------------------------------		
>Posted through http://www.FPGARelated.com


Yes, the boards are built correctly i suppose. I haven't been able to find
the fault so far. I have changed the timing constraints, as tight as they
can get, and as loose as they can get, still no success. The RTL works
smoothly on one board, and my packets are transferred fine. Whereas same
RTL doesn't give the same behavior on other 3 boards. It is basically a
Gigabit MAC RTL i have written.

What happens on other boards is that packets are transferred smoothly and
after a minute or so, they start dropping. Had it been buffer overflow
issue, it should have occurred on the 1st board as well since i have tested
the first board with full load.

I changed the PHYs today, but to no avail. I am clueless now. Any pointers
will be of great help.

And about timing constraints, i have done many iterations and almost all of
them work on the FIRST board and fail to deliver same performance on other
boards.

Kindly help me out.
Thanks..

Regards
	   
					
---------------------------------------		
Posted through http://www.FPGARelated.com

On Mon, 25 Apr 2011 15:26:25 -0500, salimbaba wrote:

>>>Hi,
>>>I am using spartan3 xc3s4000 custom board in my design interfaced with
>>>a national PHY DP83865, xilinx 12.3 for synthesis and implementation
>>>and
>>i'm
>>>facing a strange problem. I run the same RTL on two boards and it
> behaves
>>>differently on both of them. Has anyone faced this issue before ? Does
>>>anyone know why it's happening ?
>>
>>Do you know that both boards are built correctly? Could it be a board
>>hardware fault?
>>Does either board behave 'correctly'?
>>	   
>>	   
>>--------------------------------------- Posted through
>>http://www.FPGARelated.com
> 
> 
> Yes, the boards are built correctly i suppose. I haven't been able to
> find the fault so far. I have changed the timing constraints, as tight
> as they can get, and as loose as they can get, still no success. The RTL
> works smoothly on one board, and my packets are transferred fine.
> Whereas same RTL doesn't give the same behavior on other 3 boards. It is
> basically a Gigabit MAC RTL i have written.
> 
> What happens on other boards is that packets are transferred smoothly
> and after a minute or so, they start dropping. Had it been buffer
> overflow issue, it should have occurred on the 1st board as well since i
> have tested the first board with full load.
> 
> I changed the PHYs today, but to no avail. I am clueless now. Any
> pointers will be of great help.
> 
> And about timing constraints, i have done many iterations and almost all
> of them work on the FIRST board and fail to deliver same performance on
> other boards.
> 
> Kindly help me out.
> Thanks..

A Gb Ethernet MAC/PCS/PHY combination is likely to involve clock domain 
crossings.  The PHYs negotiate with their link partner to decide on one 
end as being the master clock.  The MACs, OTOH, will usually be clocked 
by the local source.  There will be typically a few (+/-200 in the worst 
case) ppm difference between the clocks.  This will be compensated for by 
either adding or deleting words of interframe gap.  IIRC this happens in 
the PCS.

The boards are supposedly identical, but the clock frequencies will be 
slightly different.  I suggest looking for a bug in your rate adaptation 
circuit.  Alternatively, you could add frequency counters to your design 
(that measure the PHY clock w.r.t. the local clock) and see if the 
direction of clock error correlates with the errors you see).

Regards,
Allan

On Apr 25, 1:26=A0pm, "salimbaba"
<a1234573@n_o_s_p_a_m.n_o_s_p_a_m.owlpic.com> wrote:
> >>Hi,
> >>I am using spartan3 xc3s4000 custom board in my design interfaced with =
a
> >>national PHY DP83865, xilinx 12.3 for synthesis and implementation and
> >i'm
> >>facing a strange problem. I run the same RTL on two boards and it
> behaves
> >>differently on both of them. Has anyone faced this issue before ? Does
> >>anyone know why it's happening ?
>
> >Do you know that both boards are built correctly?
> >Could it be a board hardware fault?
> >Does either board behave 'correctly'?
>
> >--------------------------------------- =A0 =A0 =A0 =A0 =A0 =A0
> >Posted throughhttp://www.FPGARelated.com
>
> Yes, the boards are built correctly i suppose. I haven't been able to fin=
d
> the fault so far. I have changed the timing constraints, as tight as they
> can get, and as loose as they can get, still no success. The RTL works
> smoothly on one board, and my packets are transferred fine. Whereas same
> RTL doesn't give the same behavior on other 3 boards. It is basically a
> Gigabit MAC RTL i have written.
>
> What happens on other boards is that packets are transferred smoothly and
> after a minute or so, they start dropping. Had it been buffer overflow
> issue, it should have occurred on the 1st board as well since i have test=
ed
> the first board with full load.
>
> I changed the PHYs today, but to no avail. I am clueless now. Any pointer=
s
> will be of great help.
>
> And about timing constraints, i have done many iterations and almost all =
of
> them work on the FIRST board and fail to deliver same performance on othe=
r
> boards.
>
> Kindly help me out.
> Thanks..
>
> Regards
>
> --------------------------------------- =A0 =A0 =A0 =A0
> Posted throughhttp://www.FPGARelated.com- Hide quoted text -
>
> - Show quoted text -

It could be due to PPM differences between the clock sources on your
boards where the two that are exhibiting a problem are either faster
or slower than the one that works well generating overruns or
underruns.  If your code isn't sending idles or recognizing idles and
removing them this could be a problem.

Or it could be due to two other issues that I described earlier that
you haven't mentioned addressing:

- Data transfers between asynchronous clocks without synchronization
- Latches in the design due to HDL errors

Ed McGettigan
--
Xilinx Inc.

>
>It could be due to PPM differences between the clock sources on your
>boards where the two that are exhibiting a problem are either faster
>or slower than the one that works well generating overruns or
>underruns.  If your code isn't sending idles or recognizing idles and
>removing them this could be a problem.
>
>Or it could be due to two other issues that I described earlier that
>you haven't mentioned addressing:
>
>- Data transfers between asynchronous clocks without synchronization
>- Latches in the design due to HDL errors
>
>Ed McGettigan
>--
>Xilinx Inc.
>

Hey, data transfers between asynchronous clocks are synced and there are no
latches in the design. The behavior of the RTL is a little weird on the
boards on which it is not working, let me explain if i may, i program a
fresh FPGA (FPGA that hasn't been turned on for a while), packets seem to
go smoothly, but after few seconds, packets start to drop, and eventually
within a minute from start, packets stop going altogether =\ 

Since i have 2 interfaces on the board (interface A and interface B), and
whatever i receive on A, i transmit it on B, this thing works fine. But the
packets i receive on B and transmit on A, that's where the problem exists.

kindly, give me some pointers so that i can debug the problem asap as it
has started to get on my nerves =\


Thanks a lot

regards,	   
					
---------------------------------------		
Posted through http://www.FPGARelated.com

On Apr 27, 11:59=A0am, "salimbaba"
<a1234573@n_o_s_p_a_m.n_o_s_p_a_m.owlpic.com> wrote:
> >It could be due to PPM differences between the clock sources on your
> >boards where the two that are exhibiting a problem are either faster
> >or slower than the one that works well generating overruns or
> >underruns. =A0If your code isn't sending idles or recognizing idles and
> >removing them this could be a problem.
>
> >Or it could be due to two other issues that I described earlier that
> >you haven't mentioned addressing:
>
> >- Data transfers between asynchronous clocks without synchronization
> >- Latches in the design due to HDL errors
>
> >Ed McGettigan
> >--
> >Xilinx Inc.
>
> Hey, data transfers between asynchronous clocks are synced and there are =
no
> latches in the design. The behavior of the RTL is a little weird on the
> boards on which it is not working, let me explain if i may, i program a
> fresh FPGA (FPGA that hasn't been turned on for a while), packets seem to
> go smoothly, but after few seconds, packets start to drop, and eventually
> within a minute from start, packets stop going altogether =3D\
>
> Since i have 2 interfaces on the board (interface A and interface B), and
> whatever i receive on A, i transmit it on B, this thing works fine. But t=
he
> packets i receive on B and transmit on A, that's where the problem exists=
.
>
> kindly, give me some pointers so that i can debug the problem asap as it
> has started to get on my nerves =3D\
>
> Thanks a lot
>
> regards, =A0 =A0 =A0 =A0 =A0
>
> --------------------------------------- =A0 =A0 =A0 =A0
> Posted throughhttp://www.FPGARelated.com- Hide quoted text -
>
> - Show quoted text -

How many levels of register synchronization did you use for the
asynchronous signals?
Please post the Device Utilization Summary portion of the PAR report
file.
How many WARNING messages are present in your synthesis report file?


You said that a "fresh FPGA" works at the beginning and then starts to
fail.  Does this mean that if you reprogram the part after it has been
running for a while it fails immediately?

Ed McGettigan
--
Xilinx Inc.

On Apr 27, 2:59=A0pm, "salimbaba"
<a1234573@n_o_s_p_a_m.n_o_s_p_a_m.owlpic.com> wrote:
>
> Hey, data transfers between asynchronous clocks are synced and there are =
no
> latches in the design.

- Are the clocks all directly from either the input pin receiving an
oscillator or the output of an internal PLL/DLL?  (The only correct
answer is 'yes')
- Are any clocks generated internally to the design using either logic
of flip flops?  (The only correct answer is 'no')

If your design does not have the correct answers as indicated above,
then that is the source of your timing problem, you must find and fix
it.

> The behavior of the RTL is a little weird on the
> boards on which it is not working, let me explain if i may, i program a
> fresh FPGA (FPGA that hasn't been turned on for a while), packets seem to
> go smoothly, but after few seconds, packets start to drop, and eventually
> within a minute from start, packets stop going altogether =3D\
>

When the story begins 'It works for a while and then stops working' or
'When I spray cold spray (or maybe a heat gun) it starts (or stops)
working' then the story will always end the same...you have a timing
problem, end of story.

If you want to prolong the drama, try spraying your FPGA with cold
spray to cool it off and watch it start working again...for the same
few seconds (maybe longer)...then watch it stop working again.  If you
don't believe me, try it.  At that point, you can continue the
curiousity, or simply accept that the root cause is indeed a timing
problem.

> Since i have 2 interfaces on the board (interface A and interface B), and
> whatever i receive on A, i transmit it on B, this thing works fine. But t=
he
> packets i receive on B and transmit on A, that's where the problem exists=
.
>

Nothing that you've described will give anyone enough information to
debug your problem for you so only general guidelines can be
suggested...here are mine

- Say to yourself, there is a timing problem with the design.
- Say it again...and again if necessary until you fully accept that
the problem is timing and you have to find it.
- You have not correctly performed static timing analysis
- Look at the timing report for the list of clock signals.  Get rid of
any internally generated clocks if there are any.  Do not proceed any
further until you have completed this task.
- Look at the report for signals that asynchronously reset a flip
flop.  For each such signal, verify that the reset signal is the
output of a flip flop that is clocked by the same clock signal that
clocks the flip flop that is getting asynchronously reset.  Repeat
this until you have rid the design of all such reset signals.  Note:
This is not to say that you cannot use async reset signals, what it
says is that the reset signal must first be synchronized to the clock
that is used with the flop.  That also implies that you cannot have an
async reset signal that is used to clear flip flops in more than one
clock domain.
- Verify that the static timing analysis *is* analyzing signals that
cross clock domains.  If not enabled, it can't report such errors.
Verify that each and every clock domain transfer is handled
correctly.  Do these transfers happen through a dual clock fifo part
that you did *not* write the code for and is widely used?  Or do some
(all?) transfers happen through code that you wrote?  If there are any
that happen through code that you wrote...that is a likely source of
the design error.
- Verify that there are no signals that cross clock domains and end up
at two different flip flops.  The outputs of those flip flops will at
some point be different even though you think they are both sampling
the 'same' signal (which they're not).  Rid your design of any such
signals.

> kindly, give me some pointers so that i can debug the problem asap as it
> has started to get on my nerves =3D\
>

Good luck on your search.  Doggedly follow the guidelines posted above
and you'll find the error.

Kevin Jennings

>On Apr 27, 2:59=A0pm, "salimbaba"
><a1234573@n_o_s_p_a_m.n_o_s_p_a_m.owlpic.com> wrote:
>>
>> Hey, data transfers between asynchronous clocks are synced and there are
=
>no
>> latches in the design.
>
>- Are the clocks all directly from either the input pin receiving an
>oscillator or the output of an internal PLL/DLL?  (The only correct
>answer is 'yes')
>- Are any clocks generated internally to the design using either logic
>of flip flops?  (The only correct answer is 'no')
>
>If your design does not have the correct answers as indicated above,
>then that is the source of your timing problem, you must find and fix
>it.
>
>> The behavior of the RTL is a little weird on the
>> boards on which it is not working, let me explain if i may, i program a
>> fresh FPGA (FPGA that hasn't been turned on for a while), packets seem
to
>> go smoothly, but after few seconds, packets start to drop, and
eventually
>> within a minute from start, packets stop going altogether =3D\
>>
>
>When the story begins 'It works for a while and then stops working' or
>'When I spray cold spray (or maybe a heat gun) it starts (or stops)
>working' then the story will always end the same...you have a timing
>problem, end of story.
>
>If you want to prolong the drama, try spraying your FPGA with cold
>spray to cool it off and watch it start working again...for the same
>few seconds (maybe longer)...then watch it stop working again.  If you
>don't believe me, try it.  At that point, you can continue the
>curiousity, or simply accept that the root cause is indeed a timing
>problem.
>
>> Since i have 2 interfaces on the board (interface A and interface B),
and
>> whatever i receive on A, i transmit it on B, this thing works fine. But
t=
>he
>> packets i receive on B and transmit on A, that's where the problem
exists=
>.
>>
>
>Nothing that you've described will give anyone enough information to
>debug your problem for you so only general guidelines can be
>suggested...here are mine
>
>- Say to yourself, there is a timing problem with the design.
>- Say it again...and again if necessary until you fully accept that
>the problem is timing and you have to find it.
>- You have not correctly performed static timing analysis
>- Look at the timing report for the list of clock signals.  Get rid of
>any internally generated clocks if there are any.  Do not proceed any
>further until you have completed this task.
>- Look at the report for signals that asynchronously reset a flip
>flop.  For each such signal, verify that the reset signal is the
>output of a flip flop that is clocked by the same clock signal that
>clocks the flip flop that is getting asynchronously reset.  Repeat
>this until you have rid the design of all such reset signals.  Note:
>This is not to say that you cannot use async reset signals, what it
>says is that the reset signal must first be synchronized to the clock
>that is used with the flop.  That also implies that you cannot have an
>async reset signal that is used to clear flip flops in more than one
>clock domain.
>- Verify that the static timing analysis *is* analyzing signals that
>cross clock domains.  If not enabled, it can't report such errors.
>Verify that each and every clock domain transfer is handled
>correctly.  Do these transfers happen through a dual clock fifo part
>that you did *not* write the code for and is widely used?  Or do some
>(all?) transfers happen through code that you wrote?  If there are any
>that happen through code that you wrote...that is a likely source of
>the design error.
>- Verify that there are no signals that cross clock domains and end up
>at two different flip flops.  The outputs of those flip flops will at
>some point be different even though you think they are both sampling
>the 'same' signal (which they're not).  Rid your design of any such
>signals.
>
>> kindly, give me some pointers so that i can debug the problem asap as
it
>> has started to get on my nerves =3D\
>>
>
>Good luck on your search.  Doggedly follow the guidelines posted above
>and you'll find the error.
>
>Kevin Jennings
>

Hi Kevin,
That's exactly what's happening, when the FPGA cools down, it again starts
to transmit packets for a while and then eventually stops when the FPGA is
hot.

One thing i forgot to mention was that the packets that are sent out of the
interface, when i capture them on wireshark, it shows some of them in CRC
errors, some in receive errors and the rest nowhere. I tried to capture the
packets on CHIPSCOPE, like on the transmitting end just before the packets
exits the FPGA, the packet is fine till that point. So i placed another
FPGA to capture it and it showed me a garbage packet, like data_valid
signal going low right after SFD.

Anyway that's just some information. I will follow the guidelines as there
were NO in them. 

Thanks a lot 	   
					
---------------------------------------		
Posted through http://www.FPGARelated.com