Real examples of metastability causing bugs

On Wed, 09 Jan 2008 12:49:48 GMT, "KJ" <kkjennings@sbcglobal.net>
wrote:

>
>"Allan Herriman" <allanherriman@hotmail.com> wrote in message 
>news:gdf9o3dsdk8l2p9954gnfga9bcsp5to7ki@4ax.com...
><snip>
>> What you may find surprising is that 0% of the clock domain crossing
>> bugs had anything to do with metastability.  Glitches and races were
>> the cause.
>>
>> My interpretation:
>> I think that most designers have heard of metastability, so they put
>> retiming flip flops everywhere.  Consequently, metastability related
>> problems don't occur often.
>>
>
>Your last paragraph directly contradicts the previous one.

Quite the contrary.  Please make the distinction between actual
metastability and other clock domain crossing issues, such as sampling
the same async signal with two different flip flops.

In the rest of your post (snipped) you didn't even seem to acknowledge
that there are other types of clock domain crossing issues that aren't
related to metastability.


The first point of my post was that metastability is easy to deal with
- just add another flip flop and leave enough timing slack to get
whatever MTBF you want.  (I know from some of my designs that unless
you have clock rates of more than a few hundred MHz, it isn't hard to
get failure rates of < 1 metastability-related failure in the lifetime
of the product.)

The other point of my post was that because everyone has heard of
metastability and that it's usually easy to deal with - just add flip
flops and some timing slack - it gets taken into account in designs
and doesn't actually cause a lot of bugs.  Instead, actual bugs
related to clock domain crossings are mostly caused by things not
related to metastability.  (I'm sure you could list a few.  I'll see
if I can dig up a copy of the training material I wrote which
describes the most common errors.)


Also, please bear in mind that I was quoting the results of actual
research on fielded product designs.

Regards,
Allan

> "Allan Herriman" <allanherriman@hotmail.com> wrote in message 
> news:gdf9o3dsdk8l2p9954gnfga9bcsp5to7ki@4ax.com...
> <snip>
>> What you may find surprising is that 0% of the clock domain crossing
>> bugs had anything to do with metastability.  Glitches and races were
>> the cause.
>>
>> My interpretation:
>> I think that most designers have heard of metastability, so they put
>> retiming flip flops everywhere.  Consequently, metastability related
>> problems don't occur often.

KJ wrote:
> Your last paragraph directly contradicts the previous one.

Maybe Allan's point is that
lack of proper synchronization is a much more
common problem than metastable events are
in properly synchronized designs.

   -- Mike Treseler

"Allan Herriman" <allanherriman@hotmail.com> wrote in message 
news:veh9o3t1s270peip0bqvatsa9g2mc99ql3@4ax.com...
> On Wed, 09 Jan 2008 12:49:48 GMT, "KJ" <kkjennings@sbcglobal.net>
> wrote:
>
>>
>>"Allan Herriman" <allanherriman@hotmail.com> wrote in message
>>news:gdf9o3dsdk8l2p9954gnfga9bcsp5to7ki@4ax.com...
>><snip>
>>> What you may find surprising is that 0% of the clock domain crossing
>>> bugs had anything to do with metastability.  Glitches and races were
>>> the cause.
>>>
>>> My interpretation:
>>> I think that most designers have heard of metastability, so they put
>>> retiming flip flops everywhere.  Consequently, metastability related
>>> problems don't occur often.
>>>
>>
>>Your last paragraph directly contradicts the previous one.
>
> Quite the contrary.  Please make the distinction between actual
> metastability and other clock domain crossing issues, such as sampling
> the same async signal with two different flip flops.
>
Yes, but sampling an async signal with two different flip flops would not be 
an example of properly adding retiming flip flops.

> In the rest of your post (snipped) you didn't even seem to acknowledge
> that there are other types of clock domain crossing issues that aren't
> related to metastability.
>
Actually I do agree with most of what you said (although re-reading my post 
I can see it probably didn't come across that way...sorry).

I certainly agree that darn near every timing problem that I've investigated 
as well has to do with moving a signal either from a completely asynchronous 
domain or some other clock domain.  Whether or not there was actual 
metastability or not was irrelevant since the solution to the design error 
was to properly move the signal into the sampling clock domain.

Fundamentally the design error was violation of setup/hold requirements 
and/or sampling that signal in more than one place.  Whether or not that 
caused actual metastability or not I didn't investigate.

My only disagreement was when you said "...so they put retiming flip flops 
everywhere.  Consequently, metastability related problems don't occur 
often.".  But if they properly put in the retiming flops, then they wouldn't 
have any timing issues, let alone a low probability (but not non-existent) 
one such as metastability.  But my point probably hinges on the work 
'properly' as well.

>
> The first point of my post was that metastability is easy to deal with
> - just add another flip flop and leave enough timing slack to get
> whatever MTBF you want.  (I know from some of my designs that unless
> you have clock rates of more than a few hundred MHz, it isn't hard to
> get failure rates of < 1 metastability-related failure in the lifetime
> of the product.)
I agree as long as that signal from async or other domain goes only into 
exactly one flip flop than it has been properly taken care of.  I'll also 
add that even when apparently coded properly, I've found that synthesis 
tools can sometimes defeat this by replicating a flop to improve its result 
which means you have to add vendor specific attributes to try to guard 
against this.

>
> The other point of my post was that because everyone has heard of
> metastability and that it's usually easy to deal with - just add flip
> flops and some timing slack - it gets taken into account in designs
> and doesn't actually cause a lot of bugs.
I think you're overestimating new designers ability to properly add these 
flops based on postings in this and other newsgroups even when the poster 
seems to have knowledge of metastability.

> Instead, actual bugs
> related to clock domain crossings are mostly caused by things not
> related to metastability.  (I'm sure you could list a few.  I'll see
> if I can dig up a copy of the training material I wrote which
> describes the most common errors.)
>
Violating setup and/or hold time requirements covers darn near everything. 
Sampling with only one flip flop into the new domain covers most other 
cases.  Never combinatorially generate a signal that will be used to sample 
another signal covers the only other things that I can recall at the moment.

> Also, please bear in mind that I was quoting the results of actual
> research on fielded product designs.
>
And posting those results is appreciated...like I said, I think we're 
solidly in agreement about the solution although I might not have come 
across as such in my earlier posting.

Kevin Jennings 

Eli Bendersky wrote:
>>> 2) If such an input is sampled by two different FFs for different
>>> purposes, they may end up with different results.
>> This is the case of the *missing* synchronizer.
>> This is often confused with metastability,
>> but it is really a design error.
>> I don't have to wait nearly as long to
>> observe an error in this case.
>>
> 
> Yes, and my question was about the "missing synchronizer", which is a
> design error as you said.
> 
> I just wanted an example of real code from real life doing something
> useful that is susceptible to this design error.
> 
> Eli

A real life situation for a missing synchronizer would be the "duh" 
moment I had in one of my first processor interfaced designs.  I wrote a 
value to the FPGA but the register write wasn't related to the system 
clock.  Occasionally some of my logic got one part of the word on one 
cycle before all the logic got the full register value on the second 
cycle.  This schism where values were *supposed* to change 
simultaneously but didn't caused me problems.

Just adding a synchronizing flop DOES NOT get rid of metastability 
troubles.  Just adding two consecutive synchronizing flops DOES NOT get 
rid of metastability.  Luckily most of the time the path between the two 
flops ends up being short but too often the designer DOES NOT properly 
constrain the path between those two flops.

The effect of hitting the metastability window is that the logic takes a 
moment longer to decide if it's high or low.  While the static timing 
analyzer will guarantee your results if you meet setup and hold, the 
synchronizing flop specifically violates the setup and hold in order to 
rarely hit that metastability window.  In this case, the constraints 
MUST be changed to guarantee the metastability related errors will be in 
the 1k or 1M year kind of range.

The timing constraint from a synchronizing flop to the next flop in the 
sequence MUST be constrained to a time that's shorter than the 
prevailing system clock period.  If the signal takes up to 2 ns longer 
to decide what signal level it is, the following logic (or second 
synchronizing flop) must have that additional headroom built-in through 
timing constraints.

- John_H

On Wed, 09 Jan 2008 05:47:51 -0800, Mike Treseler
<mike_treseler@comcast.net> wrote:

>
>> "Allan Herriman" <allanherriman@hotmail.com> wrote in message 
>> news:gdf9o3dsdk8l2p9954gnfga9bcsp5to7ki@4ax.com...
>> <snip>
>>> What you may find surprising is that 0% of the clock domain crossing
>>> bugs had anything to do with metastability.  Glitches and races were
>>> the cause.
>>>
>>> My interpretation:
>>> I think that most designers have heard of metastability, so they put
>>> retiming flip flops everywhere.  Consequently, metastability related
>>> problems don't occur often.
>
>KJ wrote:
>> Your last paragraph directly contradicts the previous one.
>
>Maybe Allan's point is that
>lack of proper synchronization is a much more
>common problem than metastable events are
>in properly synchronized designs.

Yes!  Thanks Mike.

I couldn't find the actual figures I wanted, but from memory it went
something like this (most commonly encountered clock domain crossing
or async logic bug listed first):

- (race) Passing vectors (i.e. multiple signals) from clock domain A
to clock domain B and expecting all the bits to arrive on the same B
clock.

- (race) As above, but adding multiple banks of retiming flip flops in
the B clock domain, which fixed the (non-existent) metastability issue
but did nothing about the race.

- (race) Passing a signal in clock domain A to multiple flip flops in
clock domain B, and expecting the B flip flops to get the same value
on the same clock.

- (race) As above, but created when the tools replicate the B logic to
manage fanout.

- (glitch) Multiple signals in clock domain A hit some combinatorial
logic producing a single signal which is sampled by a flip flop in
clock domain B.  Sometimes there may be a glitch which gets sampled by
the B flip flop.
It can be difficult to design combinatorial logic with good glitch
coverage (and if you do, the tools will often remove it).  (See XAPP
024, btw.)

- (glitch) Clock multiplexers made out of combinatorial logic with
inadequate glitch coverage (or adequate glitch coverage removed by the
tools).



I think the significant thing about that list is that even if flip
flops were infinitely fast and had 0 chance of ever entering a
metastable state, all of those bugs would still exist.


Regards,
Allan

On Thu, 10 Jan 2008 01:38:32 +1100, Allan Herriman
<allanherriman@hotmail.com> wrote:

>On Wed, 09 Jan 2008 05:47:51 -0800, Mike Treseler
><mike_treseler@comcast.net> wrote:
>
>>
>>> "Allan Herriman" <allanherriman@hotmail.com> wrote in message 
>>> news:gdf9o3dsdk8l2p9954gnfga9bcsp5to7ki@4ax.com...
>>> <snip>
>>>> What you may find surprising is that 0% of the clock domain crossing
>>>> bugs had anything to do with metastability.  Glitches and races were
>>>> the cause.
>>>>
>>>> My interpretation:
>>>> I think that most designers have heard of metastability, so they put
>>>> retiming flip flops everywhere.  Consequently, metastability related
>>>> problems don't occur often.
>>
>>KJ wrote:
>>> Your last paragraph directly contradicts the previous one.
>>
>>Maybe Allan's point is that
>>lack of proper synchronization is a much more
>>common problem than metastable events are
>>in properly synchronized designs.
>
>Yes!  Thanks Mike.
>
>I couldn't find the actual figures I wanted, but from memory it went
>something like this (most commonly encountered clock domain crossing
>or async logic bug listed first):
>
>- (race) Passing vectors (i.e. multiple signals) from clock domain A
>to clock domain B and expecting all the bits to arrive on the same B
>clock.
>
>- (race) As above, but adding multiple banks of retiming flip flops in
>the B clock domain, which fixed the (non-existent) metastability issue
>but did nothing about the race.
>
>- (race) Passing a signal in clock domain A to multiple flip flops in
>clock domain B, and expecting the B flip flops to get the same value
>on the same clock.
>
>- (race) As above, but created when the tools replicate the B logic to
>manage fanout.
>
>- (glitch) Multiple signals in clock domain A hit some combinatorial
>logic producing a single signal which is sampled by a flip flop in
>clock domain B.  Sometimes there may be a glitch which gets sampled by
>the B flip flop.
>It can be difficult to design combinatorial logic with good glitch
>coverage (and if you do, the tools will often remove it).  (See XAPP
>024, btw.)
>
>- (glitch) Clock multiplexers made out of combinatorial logic with
>inadequate glitch coverage (or adequate glitch coverage removed by the
>tools).
>
>
>
>I think the significant thing about that list is that even if flip
>flops were infinitely fast and had 0 chance of ever entering a
>metastable state, all of those bugs would still exist.


A few more.  These are still async logic, but not related to clock
domain crossings.

- Using async reset or set inputs on flip flops to implement a logic
function (rather than just using them for initialisation).  I can
remember a case where a design would fail even when we could prove
mathematically that it couldn't fail.  Rewriting it to avoid the use
of async resets fixed the problem.

- Gating clocks to create a logic function.  I know this sort of thing
is done in ASICs to save power, but it just doesn't seem to work too
well in FPGAs sometimes.

"Allan Herriman" <allanherriman@hotmail.com> wrote in message 
news:t7o9o3tvvs7jn0gpa56uf2cjng1239729u@4ax.com...
>
> - Using async reset or set inputs on flip flops to implement a logic
> function (rather than just using them for initialisation).  I can
> remember a case where a design would fail even when we could prove
> mathematically that it couldn't fail.  Rewriting it to avoid the use
> of async resets fixed the problem.
>
Hi Allan,
Can you remember what that was? I'm fairly sure synthesis tools use this 
type of trick, and I've never seen a problem with it, providing timing is 
met.
>
> - Gating clocks to create a logic function.  I know this sort of thing
> is done in ASICs to save power, but it just doesn't seem to work too
> well in FPGAs sometimes.
>
Right, you get runts or glitches on the clock which might clock some FFs but 
not others. Very bad!
Cheers, Syms. 

I do not see how metastability could ever be a "useful" feature. My
app note is an analysis tool that measures the statistical probability
of the metastable delay by random testing.
As such it is useful, and I think it shows the only practical way to
really get quantitative data.
Peter Alfke, Xilinx Applications


On Jan 8, 10:22=A0pm, Eli Bendersky <eli...@gmail.com> wrote:
> On Jan 8, 9:41=A0pm, Peter Alfke <pe...@xilinx.com> wrote:
>
> > On Jan 8, 6:20=A0am, Eli Bendersky <eli...@gmail.com> wrote:> Hello,
>
> > > Suppose that I'm sampling an asynchronous signal with a FF, without
> > > using any synchronizers before it. This FF will become metastable from=

> > > time to time with a MTBF depending on the device's parameters, the
> > > clock rate and the input signal change rate.
>
> > Eli, Look at XAPP094 (you can easily google it) It shows the circuit I
> > have used to quantify metastable delay.
> > The delay is short, so you have to be quick to catch it...
> > Peter Alfke
>
> Hi Peter,
> I downloaded this application note a couple of weeks ago and went
> through it. Would you say that your metastability-catching circuit
> could be useful for some real application ?

On Jan 9, 9:33 am, "Symon" <symon_bre...@hotmail.com> wrote:
> "Allan Herriman" <allanherri...@hotmail.com> wrote in message
>
> news:t7o9o3tvvs7jn0gpa56uf2cjng1239729u@4ax.com...
>
> > - Using async reset or set inputs on flip flops to implement a logic
> > function (rather than just using them for initialisation).  I can
> > remember a case where a design would fail even when we could prove
> > mathematically that it couldn't fail.  Rewriting it to avoid the use
> > of async resets fixed the problem.
>
> Hi Allan,
> Can you remember what that was? I'm fairly sure synthesis tools use this
> type of trick, and I've never seen a problem with it, providing timing is
> met.

<snip>


I know that at least XST will use these inputs to implement logic
functions.  They have comparatively long setup times, and I see them
frequently when looking at static timing analysis reports.

Regards,

John McCaskill
www.FasterTechnology.com

On Jan 9, 8:25=A0am, John McCaskill <jhmccask...@gmail.com> wrote:
> On Jan 9, 9:33 am, "Symon" <symon_bre...@hotmail.com> wrote:
>
> > "Allan Herriman" <allanherri...@hotmail.com> wrote in message
>
> >news:t7o9o3tvvs7jn0gpa56uf2cjng1239729u@4ax.com...
>
> > > - Using async reset or set inputs on flip flops to implement a logic
> > > function (rather than just using them for initialisation). =A0I can
> > > remember a case where a design would fail even when we could prove
> > > mathematically that it couldn't fail. =A0Rewriting it to avoid the use=

> > > of async resets fixed the problem.
>
> > Hi Allan,
> > Can you remember what that was? I'm fairly sure synthesis tools use this=

> > type of trick, and I've never seen a problem with it, providing timing i=
s
> > met.
>
> <snip>
>
> I know that at least XST will use these inputs to implement logic
> functions. =A0They have comparatively long setup times, and I see them
> frequently when looking at static timing analysis reports.
>
> Regards,
>
> John McCaskillwww.FasterTechnology.com

While XST may use the inputs in implementing the logic, it uses the
synchronous set/reset rather than the asynchronous clear/preset
equivalent.  Any synchronous implementation should be covered by the
timing analysis.