Handling overflow in a self-repeating frequency counter

Hi guys.

I'm currently working on the HDL code for an open-source project which 
images 'strange and unusual' magnetic media (mainly floppy discs but also 
MFM and RLL hard disc drives).

To do this, I measure the time between incoming pulses on the RD_DATA 
line. One pulse indicates a magnetic transition. The time between a given 
pair of pulses encodes (in a roundabout way) a data bit. This is a gross 
oversimplification, but serves the purpose for this explanation.

After measuring the time between two pulses, it re-arms and measures 
between the second pulse and the one following it. Rinse, repeat.

The captured timing values are stored in an on-FPGA FIFO buffer, then 
passed onto an off-chip Cypress 100MHz 512K*8bit high speed static RAM 
(this thing needs A LOT of storage). The period counter's reference clock 
is 100MHz. 10 nanoseconds per count. The most significant bit is reserved 
for a 'flag' bit (it's used by other parts of the design), leaving a 
range of 0 to 127 for the counter.

The problem is, a lot of the timing values are about 800 counts long. To 
work around this, I implemented logic to store a 'special' value in the 
stream if the counter overflows from 127 to 0. This value is "8'd0" -- 8 
bits, all zeroes.

This raises a problem: if any two incoming pulses are measured at exactly 
128 clocks apart (or a modulo thereof), the HDL stores a carry (0x00)... 
immediately followed by the value of the counter, which is 0x00 (because 
the count of clock pulses is a multiple of 128). This is messing up the 
datastream, and I only caught it because a friend mentioned it, I added a 
test case to the testbench, and I got a truckload of assertion failures.

The Verilog code is here:
  http://hg.discferret.com/microcode/file/1b4ea3919380/DiscReader.v
  http://hg.discferret.com/microcode/file/1b4ea3919380/
Flag_Delay1tcy_OneCycle.v
  (other modules are here if you need them: http://hg.discferret.com/
microcode/file/1b4ea3919380 )

I have also rewritten DiscReader.v to use a free-running counter (which 
seems like a better solution) -- that version is here:
  http://pastebin.com/evMtjXWj

Feel free to offer comments on either of these... the rewrite is later 
code, and as far as I can tell fully synchronous to the clock (the older 
code has some other corner cases -- hence the rewrite).


Effectively the byte-stream format is:
CARRY starts at 0.
  0x00 or 0x80 -- add 128 to CARRY.
  Anything else -- the timing value is CARRY + (this_byte AND 0x7F). 
Clear CARRY to zero afterwards.


The best idea I've come up with thus far is to scrap the overflow logic, 
cause the acquisition to abort with an error state if the timer 
overflows, and use a 12-bit or larger counter (which would allow for a 
maximum count of 40us). This would of course mean I'd have to rewrite the 
RAM controller logic to handle a 16-bit stream and chunk it into 
bytes... :(


Can anyone think of a good way to resolve this, ideally without 
introducing timing jitter (although the acquisition clock is PLL'd off a 
20MHz TTL oscillator, so there's probably a bit of that already)? I'm not 
overly attached to the bitstream format, so feel free to propose changes.


Thanks,
-- 
Phil.
philpem@philpem.me.uk
http://www.philpem.me.uk/

Philip Pemberton <philpem@philpem.me.uk> wrote:

> I'm currently working on the HDL code for an open-source project which 
> images 'strange and unusual' magnetic media (mainly floppy discs but also 
> MFM and RLL hard disc drives).

> To do this, I measure the time between incoming pulses on the RD_DATA 
> line. One pulse indicates a magnetic transition. The time between a given 
> pair of pulses encodes (in a roundabout way) a data bit. This is a gross 
> oversimplification, but serves the purpose for this explanation.

> After measuring the time between two pulses, it re-arms and measures 
> between the second pulse and the one following it. Rinse, repeat.

> The captured timing values are stored in an on-FPGA FIFO buffer, then 
> passed onto an off-chip Cypress 100MHz 512K*8bit high speed static RAM 
> (this thing needs A LOT of storage). The period counter's reference clock 
> is 100MHz. 10 nanoseconds per count. The most significant bit is reserved 
> for a 'flag' bit (it's used by other parts of the design), leaving a 
> range of 0 to 127 for the counter.

The bit rate for floppies is 125000b/s at the low end, and, as far as
I know, 1000000b/s at the high end.  To decode FM or MFM data, you
don't need a hugh amount of resolution.  There are only two different
times between transitions, at least ideally.

The GCR code used by older Apple formats has more different times, but
not all that many, and the bit rate is on the low side.

To write a floppy from data, you need to consider precompensation,
which needs a little finer timing.  Note that writing a floppy from
your data stream, you will also need to consider precompensation.

Anyway, my suggestion is to use a slower clock.  Then it should only
overflow at splice points, or other places where the data doesn't
matter.

> The problem is, a lot of the timing values are about 800 counts long. To 
> work around this, I implemented logic to store a 'special' value in the 
> stream if the counter overflows from 127 to 0. This value is "8'd0" -- 8 
> bits, all zeroes.

The design of codes that can be uniquely decoded is well known.
One answer is to store one more than the overflow value.  
That is, a 128 count should be X'0001', so X'00' always means overflow,
and never data.  X'007f' then will be 254, and X'000001' will be 255.

The logic is a little more complicated, but shouldn't be hard to
fit into an FPGA.

-- glen

On Fri, 06 Jan 2012 02:49:54 +0000, Philip Pemberton wrote:

> Hi guys.
> 
> I'm currently working on the HDL code for an open-source project which
> images 'strange and unusual' magnetic media (mainly floppy discs but
> also MFM and RLL hard disc drives).
> 
> To do this, I measure the time between incoming pulses on the RD_DATA
> line. One pulse indicates a magnetic transition. The time between a
> given pair of pulses encodes (in a roundabout way) a data bit. 
...

> The problem is, a lot of the timing values are about 800 counts long. To
> work around this, I implemented logic to store a 'special' value in the
> stream if the counter overflows from 127 to 0. This value is "8'd0" -- 8
> bits, all zeroes.

Consider A-law (or mu-law) encoding instead.

- Brian

Brian Drummond <brian@shapes.demon.co.uk> wrote:
> On Fri, 06 Jan 2012 02:49:54 +0000, Philip Pemberton wrote:

>> I'm currently working on the HDL code for an open-source project which
>> images 'strange and unusual' magnetic media (mainly floppy discs but
>> also MFM and RLL hard disc drives).
 
>> To do this, I measure the time between incoming pulses on the RD_DATA
>> line. One pulse indicates a magnetic transition. The time between a
>> given pair of pulses encodes (in a roundabout way) a data bit. 

>> The problem is, a lot of the timing values are about 800 counts long. To
>> work around this, I implemented logic to store a 'special' value in the
>> stream if the counter overflows from 127 to 0. This value is "8'd0" -- 8
>> bits, all zeroes.

> Consider A-law (or mu-law) encoding instead.

That should work.  I think, though, that it doesn't matter much.

The gaps between sectors, and between the before/after the index address
mark, all have data bits in them as formatted.  

The problem comes with write splice.  (Put "write splice" floppy into
google, and some hits will be for floppy controllers.  At least the
one for the 279x describes the track format.)

Each sector has a header with the track/sector/side/length bytes.
When writing, the controller finds the header for the appropriate
sector, counts off a specific number of bytes (different for FM and
MFM), then turns on the write current.  It then writes some binary
zeros, an address mark, followed by the data, CRC, a few more bytes,
then turns the write current off.  The point where it turns on and
off is the write splice, and I believe that is where you see the
long counts.  The exact length doesn't matter much, and will vary
between drives, and with the motor speed, with a tolerance of maybe
about 10%.  

The gap after the sector and before the next sector header is long
enough such that within tolerance the write splice won't overwrite
the next header.  The gaps are generously large (but with data bits
in them) to allow for a variation in motor speeds and controller
clocks.  (Much more motor speed than clock.)

There are enough data bits after the write splice and before the
address mark to sync up the PLL.  

What is the variation in length for these long counts, and about
how many do you find on a track?

I presume you start reading on the index pulse, and stop at the
next index pulse.  If you want to convert a track image back to
a floppy, you will start writing at the index pulse, and either stop
at the next (if you don't run out of data), or continue writing
binary zeros (in FM or MFM form) until the index pulse.

-- glen

On Fri, 06 Jan 2012 03:18:38 +0000, glen herrmannsfeldt wrote:

> Philip Pemberton <philpem@philpem.me.uk> wrote:
> 
>> I'm currently working on the HDL code for an open-source project which
>> images 'strange and unusual' magnetic media (mainly floppy discs but
>> also MFM and RLL hard disc drives).
> 
>> To do this, I measure the time between incoming pulses on the RD_DATA
>> line. One pulse indicates a magnetic transition. The time between a
>> given pair of pulses encodes (in a roundabout way) a data bit. This is
>> a gross oversimplification, but serves the purpose for this
>> explanation.
> 
>> After measuring the time between two pulses, it re-arms and measures
>> between the second pulse and the one following it. Rinse, repeat.
> 
>> The captured timing values are stored in an on-FPGA FIFO buffer, then
>> passed onto an off-chip Cypress 100MHz 512K*8bit high speed static RAM
>> (this thing needs A LOT of storage). The period counter's reference
>> clock is 100MHz. 10 nanoseconds per count. The most significant bit is
>> reserved for a 'flag' bit (it's used by other parts of the design),
>> leaving a range of 0 to 127 for the counter.
> 
> The bit rate for floppies is 125000b/s at the low end, and, as far as I
> know, 1000000b/s at the high end.  To decode FM or MFM data, you don't
> need a hugh amount of resolution.  There are only two different times
> between transitions, at least ideally.
> 
> The GCR code used by older Apple formats has more different times, but
> not all that many, and the bit rate is on the low side.
> 
> To write a floppy from data, you need to consider precompensation, which
> needs a little finer timing.  Note that writing a floppy from your data
> stream, you will also need to consider precompensation.
> 
> Anyway, my suggestion is to use a slower clock.  Then it should only
> overflow at splice points, or other places where the data doesn't
> matter.
> 
>> The problem is, a lot of the timing values are about 800 counts long.
>> To work around this, I implemented logic to store a 'special' value in
>> the stream if the counter overflows from 127 to 0. This value is "8'd0"
>> -- 8 bits, all zeroes.
> 
> The design of codes that can be uniquely decoded is well known. One
> answer is to store one more than the overflow value. That is, a 128
> count should be X'0001', so X'00' always means overflow, and never data.
>  X'007f' then will be 254, and X'000001' will be 255.
> 
> The logic is a little more complicated, but shouldn't be hard to fit
> into an FPGA.
> 
> -- glen

If you want the data to fit neatly into seven bits, give your counter a 
period of 127, and assign X'00' to overflow, with X'01' through X'7f' 
denoting a bit hit at that count.  That'll insure unique data, without 
blowing out your bit-count budget.

You'll complicate the decoding a bit, but that's an easy task for a 
computer to carry out.

-- 
My liberal friends think I'm a conservative kook.
My conservative friends think I'm a liberal kook.
Why am I not happy that they have found common ground?

Tim Wescott, Communications, Control, Circuits & Software
http://www.wescottdesign.com

On Fri, 06 Jan 2012 12:18:50 -0600, Tim Wescott wrote:

> If you want the data to fit neatly into seven bits, give your counter a
> period of 127, and assign X'00' to overflow, with X'01' through X'7f'
> denoting a bit hit at that count.  That'll insure unique data, without
> blowing out your bit-count budget.
> 
> You'll complicate the decoding a bit, but that's an easy task for a
> computer to carry out.

That's more or less what I'm doing now.

The problem is, there's a race condition -- if there's an overflow at the 
same time as an incoming data pulse, then the overflow is stored, but the 
transition is ignored. In the example you gave, that's when the count is 
*exactly* 127, or a modulo thereof.

What I need to do is eliminate this race condition...


Here's my code --

module DiscReader(CLOCK, CLKEN, RUN, FD_RDDATA_IN, FD_INDEX_IN, RESET, 
DATA, WRITE);
	parameter BITS = 8;		// data bits

	input					
CLOCK;				// counter clock
	input					
CLKEN;				// counter clock enable
	input					
RUN;				// enable input -- 1 to acquire
	input					FD_RDDATA_IN;		// 
read data
	input					FD_INDEX_IN;		// 
index pulse
	input					
RESET;				// asynchronous reset

	output reg	[BITS-1:0]	DATA;				// 
data output to RAM
	output reg				
WRITE;				// write output to RAM


/////////////////////////////////////////////////////////////////////////////
// Input synchronisation
	wire FD_RDDATA_IN_tcysync, FD_INDEX_IN_tcysync;
	Flag_Delay1tcy_OneCycle _fcd_rddata	(CLOCK,	FD_RDDATA_IN,	
FD_RDDATA_IN_tcysync);
	Flag_Delay1tcy_OneCycle _fcd_index	(CLOCK,	FD_INDEX_IN,	
FD_INDEX_IN_tcysync);

/////////////////////////////////////////////////////////////////////////////
// Free-running counter for frequency measurement
	reg [BITS-2:0] counter, last_count;
	always @(posedge CLOCK) begin
		if (RESET) begin
			// reset active -- clear counter
			counter <= 'd0;
		end else if (CLKEN) begin
			// otherwise increment
			counter <= counter + 'd1;
		end
	end

/////////////////////////////////////////////////////////////////////////////
// Frequency counter

	// current counter value
	wire [BITS-2:0] cur_count = (last_count <= counter) ?
		counter - last_count :					// 
last count <= counter
		('b1 << BITS) - last_count + counter;	// last count >  
counter

	always @(posedge CLOCK) begin
		// don't write to memory!
		WRITE <= 1'b0;

		if (RESET) begin
			// reset active -- clear last-count
			last_count <= 'd0;
		end else begin
			if (FD_RDDATA_IN_tcysync | FD_INDEX_IN_tcysync | 
(cur_count == (('d1 << BITS-1)-1))) begin
				// Index hit, flux transition or overflow
				DATA <= {FD_INDEX_IN_tcysync ? 1'b1 : 
1'b0, cur_count};

				// write to memory
				WRITE <= 1'b1;

				// Update 'previous count value' register
				last_count <= counter;
			end
		end
	end

endmodule


module Flag_Delay1tcy_OneCycle(clk, in, out);
	input		clk, in;
	output	out;

	reg[1:0] in_Delayed;
	always @(posedge clk) in_Delayed <= {in_Delayed[0], in};
	assign out = (in_Delayed[0] && !in_Delayed[1]);
endmodule


// Testbench -- provokes race condition in DiscReader module

// 5ns time reference with 1ns precision
`timescale 1 ns / 1 ns

module DiscReaderTB;
	reg clock, run, rddata, index, reset;
	wire clock_enable = 1'b1;

	parameter N = 8;

	wire [N-1:0] data_r;
	wire ram_write;

	// instantiate the disc reader
	DiscReader U0 (
		.CLOCK			(clock),
		.CLKEN			(clock_enable),
		.RUN			(run),
		.FD_RDDATA_IN	(rddata),
		.FD_INDEX_IN	(index),
		.RESET			(reset),

		.DATA			(data_r),
		.WRITE			(ram_write)
	);

	// set initial state
	initial begin
		clock = 0;
		run = 0;
		rddata = 0;
		index = 0;
		reset = 0;
	end

	// generate a 100MHz clock
	always
		#1 clock = !clock;

	// set up for variable dumping
	initial  begin
		$dumpfile ("counter.vcd"); 
		$dumpvars; 
	end

	integer k;
	// main loop
	initial begin
		#10
			reset = 1;
		#10
			reset = 0;
			run = 1;

		for (k=0; k<=16; k=k+1) begin
				rddata = 1;
			#2
				rddata = 0;
			#252
				;
		end

		#1000 $finish;
	end

endmodule

Thanks.
-- 
Phil.
philpem@philpem.me.uk
http://www.philpem.me.uk/

Tim Wescott <tim@seemywebsite.com> wrote:

(snip)
>>> The problem is, a lot of the timing values are about 800 counts long.
>>> To work around this, I implemented logic to store a 'special' value in
>>> the stream if the counter overflows from 127 to 0. This value is "8'd0"
>>> -- 8 bits, all zeroes.
 
>> The design of codes that can be uniquely decoded is well known. One
>> answer is to store one more than the overflow value. That is, a 128
>> count should be X'0001', so X'00' always means overflow, and never data.
>>  X'007f' then will be 254, and X'000001' will be 255.
 
>> The logic is a little more complicated, but shouldn't be hard to fit
>> into an FPGA.

> If you want the data to fit neatly into seven bits, give your counter a 
> period of 127, and assign X'00' to overflow, with X'01' through X'7f' 
> denoting a bit hit at that count.  That'll insure unique data, without 
> blowing out your bit-count budget.

That is what I was thinking, but probably didn't get quite right.

> You'll complicate the decoding a bit, but that's an easy task for a 
> computer to carry out.

Well, easy for a computer, slightly harder to write as a state
machine in verilog, but not that much harder.

-- glen

Philip Pemberton <philpem@philpem.me.uk> wrote:
> On Fri, 06 Jan 2012 12:18:50 -0600, Tim Wescott wrote:

>> If you want the data to fit neatly into seven bits, give your counter a
>> period of 127, and assign X'00' to overflow, with X'01' through X'7f'
>> denoting a bit hit at that count.  That'll insure unique data, without
>> blowing out your bit-count budget.
 
>> You'll complicate the decoding a bit, but that's an easy task for a
>> computer to carry out.

> That's more or less what I'm doing now.

> The problem is, there's a race condition -- if there's an overflow at the 
> same time as an incoming data pulse, then the overflow is stored, but the 
> transition is ignored. In the example you gave, that's when the count is 
> *exactly* 127, or a modulo thereof.

> What I need to do is eliminate this race condition...

No, it is worse than a race condition, you have one state that
has two meanings.  

OK, the easy way is to do it modulo 64, and use the 64 value to mean 64,
and 0 to mean 0.  (It is easier to write modulo 64 than 127.)

For 5.25in FM, the data rate is 125000b/s, the transitions are either
at 8us or 4us, so there will be a lot of 80s.  Not so good.

For 5.25in MFM, the data rate is 250000b/s, there is no clock pulse
if either the cell before or after has a data pulse.  I believe that
means either 4us or 6us, except for an address mark where it
would be 8us.  (I haven't thought about this for a while.
Find a reference to the IBM standard.)

OK, do it modulo 127 with 127 as the overflow code which allows
for a zero code to mean zero.  (Most likely only after a 127.)
It isn't that hard to test for 127, and you have plenty of
time to do it.

-- glen

On 01/05/2012 10:18 PM, glen herrmannsfeldt wrote:

> The bit rate for floppies is 125000b/s at the low end, and, as far as
> I know, 1000000b/s at the high end.  To decode FM or MFM data, you
> don't need a hugh amount of resolution.  There are only two different
> times between transitions, at least ideally.
>
> The GCR code used by older Apple formats has more different times, but
> not all that many, and the bit rate is on the low side.
>
> To write a floppy from data, you need to consider precompensation,
> which needs a little finer timing.  Note that writing a floppy from
> your data stream, you will also need to consider precompensation.
>
> Anyway, my suggestion is to use a slower clock.  Then it should only
> overflow at splice points, or other places where the data doesn't
> matter.

What Philip failed to mention was that the product in question is also 
intended to read data streams from MFM hard drives.  A slower clock may not be 
an option.

Steven Hirsch <snhirsch@gmail.com> wrote:

(snip, I wrote)
>> To write a floppy from data, you need to consider precompensation,
>> which needs a little finer timing.  Note that writing a floppy from
>> your data stream, you will also need to consider precompensation.

>> Anyway, my suggestion is to use a slower clock.  Then it should only
>> overflow at splice points, or other places where the data doesn't
>> matter.

> What Philip failed to mention was that the product in question is also 
> intended to read data streams from MFM hard drives.  A slower clock may not be 
> an option.

I was assuming that you knew which type of drive you were working
with and could select accordingly.  

Interestingly, though, the IBM PC/AT disk controller uses a 24MHz
crystal for the floppy drive, and, if I remember right, 10MHz 
for the MFM hard drive.

-- glen