400 Mb/s ADC

We are building a new radio telescope called PAST
(http://astrophysics.phys.cmu.edu/~jbp/past6.pdf)
which we will install at the South Pole or in Western China.

To make this work, will need to sample (6 to 8 bit precision) dozens
of analog voltages at 400 Msample/sec and feed these data streams into
PCs. One PC per sampler.

The flash ADCs we need are available (Maxim), but we are finding it
difficult to get the data into the PC.

One simple way would be to use SCSI ultra640, but so far I have not
found any 640 adapters on the market.  Is any 640 adapter available?
anything coming soon?

or we could go right into a PCI-X bus.  has anyone out there
done this at 400 Mb/s?  is this hard to do? FPGA core liscense
for this seems expensive ($9K), with no guarentee of 400 mByte rates.

is there a better way?

thanks 

-Jeff Peterson

Jeff Peterson wrote:
> We are building a new radio telescope called PAST
> (http://astrophysics.phys.cmu.edu/~jbp/past6.pdf)
> which we will install at the South Pole or in Western China.
>
> To make this work, will need to sample (6 to 8 bit precision) dozens
> of analog voltages at 400 Msample/sec and feed these data streams into
> PCs. One PC per sampler.

How big is a sample?

>
> The flash ADCs we need are available (Maxim), but we are finding it
> difficult to get the data into the PC.
>
> One simple way would be to use SCSI ultra640, but so far I have not
> found any 640 adapters on the market.  Is any 640 adapter available?
> anything coming soon?
>
> or we could go right into a PCI-X bus.  has anyone out there
> done this at 400 Mb/s?  is this hard to do? FPGA core liscense
> for this seems expensive ($9K), with no guarentee of 400 mByte rates.
>
> is there a better way?


Not clear from this whether you mean Mbit/s or MBytes/sec. If you mean
Mbit/s then obviously that's not a hard problem to solve. If as I suspect
you do mean Mbytes/sec then a PC (by the conventional definition) isn't
going to cut it because typical PC motherboards don't support PCI-X at any
frequency, they are still limited to 33MHz/32bit PCI which just isn't good
enough.

So the first step will be identifying a motherboard (probably with a
workstation or server classification) that supports PCI-X at least 100MHz,
which gives a peak theoretical throughput of 800MB/s, but a sustain probably
closer to 400MB/s. Then you need to define what you are doing with the data,
for example you could be:

1. Just capturing the data performing some operation on it, storing the
results and throwing away the sample

2. You might be actually planning to capture to disk 400MB/s for a sustained
period which has some pretty hairy implications for storage capacity.

I do know of one site doing something on a similar scale, and that's a US
Airforce project called Starfire Optical Range
(http://www.sor.plk.af.mil/SOR/) at Kirkland AFB. I don't believe this
project is heavily classified (I certainly didn't have to sign anything
before helping them on the storage subsystem in 2000) so it might be worth
contacting them to see if they can help you spec out a system.


-- 
Nik Simpson

Do you really need to transfer raw data? Can you do some front-end
processing to bring the speed down? If answers are 'yes' and 'no'
respectively, think of repacking. You can pack 8 bytes into one 64-bit word.
This brings the speed down to 50 MW/s, which should fit into regular 64/66
PCI.


/Mikhail



"Jeff Peterson" <jbp@cmu.edu> wrote in message
news:369b6e8b.0311190715.4d66f38f@posting.google.com...
> We are building a new radio telescope called PAST
> (http://astrophysics.phys.cmu.edu/~jbp/past6.pdf)
> which we will install at the South Pole or in Western China.
>
> To make this work, will need to sample (6 to 8 bit precision) dozens
> of analog voltages at 400 Msample/sec and feed these data streams into
> PCs. One PC per sampler.
>
> The flash ADCs we need are available (Maxim), but we are finding it
> difficult to get the data into the PC.
>
> One simple way would be to use SCSI ultra640, but so far I have not
> found any 640 adapters on the market.  Is any 640 adapter available?
> anything coming soon?
>
> or we could go right into a PCI-X bus.  has anyone out there
> done this at 400 Mb/s?  is this hard to do? FPGA core liscense
> for this seems expensive ($9K), with no guarentee of 400 mByte rates.
>
> is there a better way?
>
> thanks
>
> -Jeff Peterson

Jeff Peterson wrote:

> We are building a new radio telescope called PAST
> (http://astrophysics.phys.cmu.edu/~jbp/past6.pdf)
> which we will install at the South Pole or in Western China.
> 
> To make this work, will need to sample (6 to 8 bit precision) dozens
> of analog voltages at 400 Msample/sec and feed these data streams into
> PCs. One PC per sampler.
> 
> The flash ADCs we need are available (Maxim), but we are finding it
> difficult to get the data into the PC.
> 
> One simple way would be to use SCSI ultra640, but so far I have not
> found any 640 adapters on the market.  Is any 640 adapter available?
> anything coming soon?
> 
> or we could go right into a PCI-X bus.  has anyone out there
> done this at 400 Mb/s?  is this hard to do? FPGA core liscense
> for this seems expensive ($9K), with no guarentee of 400 mByte rates.

The first thing I would think of would be to buffer it and then read it 
later.  You don't say how long this data stream will be, or if this is a 
peak rate with a much lower average rate.  Does it have to go to disk at 
that rate?

You could collect the samples into 64 bit words and write then into 
SDRAM at 40 or 50 MHz.

If it has to go to disk at that rate, I would work on the hardware to 
get it onto the disk without a processor in between.

-- glen

"Nik Simpson" <n_simpson@bellsouth.net> wrote in message news:<DiMub.3510$gU2.827@bignews6.bellsouth.net>...
> Jeff Peterson wrote:
> > We are building a new radio telescope called PAST
> > (http://astrophysics.phys.cmu.edu/~jbp/past6.pdf)
> > which we will install at the South Pole or in Western China.
> >
> > To make this work, will need to sample (6 to 8 bit precision) dozens
> > of analog voltages at 400 Msample/sec and feed these data streams into
> > PCs. One PC per sampler.
> 
> How big is a sample?
8 bits.
> 
> >
> > The flash ADCs we need are available (Maxim), but we are finding it
> > difficult to get the data into the PC.
> >
> > One simple way would be to use SCSI ultra640, but so far I have not
> > found any 640 adapters on the market.  Is any 640 adapter available?
> > anything coming soon?
> >
> > or we could go right into a PCI-X bus.  has anyone out there
> > done this at 400 Mb/s?  is this hard to do? FPGA core liscense
> > for this seems expensive ($9K), with no guarentee of 400 mByte rates.
> >
> > is there a better way?
> 
> 
> Not clear from this whether you mean Mbit/s or MBytes/sec. If you mean
> Mbit/s then obviously that's not a hard problem to solve. If as I suspect
> you do mean Mbytes/sec then a PC (by the conventional definition) isn't
> going to cut it because typical PC motherboards don't support PCI-X at any
> frequency, they are still limited to 33MHz/32bit PCI which just isn't good
> enough.
i do mean 400Mbytes/sec. and yeah, PCI 33/32 wont cut it.

> 
> So the first step will be identifying a motherboard (probably with a
> workstation or server classification) that supports PCI-X at least 100MHz,
> which gives a peak theoretical throughput of 800MB/s, but a sustain probably
> closer to 400MB/s. Then you need to define what you are doing with the data,
> for example you could be:
> 
> 1. Just capturing the data performing some operation on it, storing the
> results and throwing away the sample
we accumualte averages (of cross products of fourier tranforms)

> 
> 2. You might be actually planning to capture to disk 400MB/s for a sustained
> period which has some pretty hairy implications for storage capacity.
> 
we wont store the raw data, just a very much reduced set.

> I do know of one site doing something on a similar scale, and that's a US
> Airforce project called Starfire Optical Range
> (http://www.sor.plk.af.mil/SOR/) at Kirkland AFB. I don't believe this
> project is heavily classified (I certainly didn't have to sign anything
> before helping them on the storage subsystem in 2000) so it might be worth
> contacting them to see if they can help you spec out a system.

> we accumualte averages (of cross products of fourier tranforms)

What are you going to be using for calculations? If you are planning on
using DSP cards, then you don't need to go through PCI. You coud transfer
data directly from your A/D card to a DSP card using for example FPDP
interface.

There is a number of companies who do similar things for radar and sonar.
Look at the products by Pentek, ICS, Gage, etc...

BTW, I think this discussion drifted away from the FPGA topic, so it
probably doesn't belong here...

/Mikhail

Jeff Peterson wrote:
>> 1. Just capturing the data performing some operation on it, storing
>> the results and throwing away the sample
>
> we accumualte averages (of cross products of fourier tranforms)
>

So the basic problem is getting 400MB/s of data into memory and processing
it, but are you reading 400MB every second, or sampling say once every ten
seconds. If it's every second, then you've got a bigger problem because I'd
be surprised if you can process it fast enough to get the job done before
the next sample comes along.

>>
>> 2. You might be actually planning to capture to disk 400MB/s for a
>> sustained period which has some pretty hairy implications for
>> storage capacity.
>>
> we wont store the raw data, just a very much reduced set.

So disk output bandwidth is not going to be a problem, what you are looking
for is a way of getting 400MB/s of data into memory for post-processing,
correct. Is it possible to break-up the input stream, so for example instead
of reading a single stream of 400MB/s, you've five devices reading 80MB/s in
parralel? Is the design of the device capturing the data set in stone or can
it be "parrallelized" if so it would make the problem much simpler and any
solution more scalable and less expensive.


-- 
Nik Simpson

"Nik Simpson" <n_simpson@bellsouth.net> wrote in message news:<vQPub.41$zi3.40@bignews3.bellsouth.net>...
> Jeff Peterson wrote:
> >> 1. Just capturing the data performing some operation on it, storing
> >> the results and throwing away the sample
> >
> > we accumualte averages (of cross products of fourier tranforms)
> >
> 
> So the basic problem is getting 400MB/s of data into memory and processing
> it, but are you reading 400MB every second, or sampling say once every ten
> seconds. If it's every second, then you've got a bigger problem because I'd
> be surprised if you can process it fast enough to get the job done before
> the next sample comes along.
we will take about 64K samples, then can pause while processing...
however all the time we are pausing we are losing data.  so we do want to
keep the duty cycle up.  50% dudty cyle is not a problem. 5% would be.


> 
> >>
> >> 2. You might be actually planning to capture to disk 400MB/s for a
> >> sustained period which has some pretty hairy implications for
> >> storage capacity.
> >>
> > we wont store the raw data, just a very much reduced set.
> 
> So disk output bandwidth is not going to be a problem, what you are looking
> for is a way of getting 400MB/s of data into memory for post-processing,
> correct. Is it possible to break-up the input stream, so for example instead
> of reading a single stream of 400MB/s, you've five devices reading 80MB/s in
> parralel? Is the design of the device capturing the data set in stone or can
> it be "parrallelized" if so it would make the problem much simpler and any
> solution more scalable and less expensive.
this could work.  for example we have considered using 2 x scsi 320 
interfaces.  might work but its a bit of a kludge, and if we got the
two interfaces out of sync we would have a real mess.

"MM" <mbmsv@yahoo.com> wrote in message news:<bpg42l$1o09pi$1@ID-204311.news.uni-berlin.de>...
> Do you really need to transfer raw data? Can you do some front-end
> processing to bring the speed down? If answers are 'yes' and 'no'
> respectively, think of repacking. You can pack 8 bytes into one 64-bit word.
> This brings the speed down to 50 MW/s, which should fit into regular 64/66
> PCI.
> 
> 
> /Mikhail
> 

i dont believe we can reduce the data rate by pre-processing.

yes, repacking might allow a 64/66 PCI to accept the data. i worry
that we will spend lots of time and money, but the margin will be
insufficient for it to actually work.  i have heard that some PCI
cores are not too efficient.

-Jeff

"Jeff Peterson" <jbp@cmu.edu> wrote in message
news:369b6e8b.0311191050.44b781b6@posting.google.com...
> we accumualte averages (of cross products of fourier tranforms)
>

Jeff,
    I think you need to do the 'we accumualte averages (of cross products of
fourier tranforms)' in your FPGA. That way you dramatically reduce your data
rate down to something sensible. It sounds tricky, but not as tricky as
getting 400M x 8 bits x 50% duty cycle = 1.6 Gbps into a PC and processing
it!
            cheers, Syms.