video buffering scheme, nonsequential access (no spatial locality)

"wallge" <wallge@gmail.com> wrote in message 
news:1169753051.309748.114380@q2g2000cwa.googlegroups.com...
>I am not doing any image filtering.

Yes you are.

> This is not a filtering operation.

Yes it is.

> It is an interpolation operation
> typically bilinear or bicubic
> to do image transformations.

And that's a filtering operation.
So the maximum kernel size is 4 x 4, though
you might use 2 x 2. The kernel size could have a substantail
bearing on the traffic to/from on-chip RAM.

I'm still not sure of your limitations on off-chip RAM.
You have a buffer on the input or output (or both?)
Do you have enough bandwidth to have an
intermediate buffer for a two-pass operation? 

Can you write out the FIR filter coeffs for
a bilinear interpolation "filter kernel"?
How about a bicubic interpolator filter kernel
what are its filter coeffs?

arguing semantics was not the purpose of my post.

I will probably wind up doing bilinear interpolation or
"filtering". Which means I need 4 pixels of the input frame to
determine
1 pixel of output warped frame.

By the way what is the Freq response of the bilinear interpolation
"filter"?



On Jan 25, 5:16 pm, "Pete Fraser" <pfra...@covad.net> wrote:
> "wallge" <wal...@gmail.com> wrote in messagenews:1169753051.309748.114380@q2g2000cwa.googlegroups.com...
>
> >I am not doing any image filtering.Yes you are.
>
> > This is not a filtering operation.Yes it is.
>
> > It is an interpolation operation
> > typically bilinear or bicubic
> > to do image transformations.And that's a filtering operation.
> So the maximum kernel size is 4 x 4, though
> you might use 2 x 2. The kernel size could have a substantail
> bearing on the traffic to/from on-chip RAM.
>
> I'm still not sure of your limitations on off-chip RAM.
> You have a buffer on the input or output (or both?)
> Do you have enough bandwidth to have an
> intermediate buffer for a two-pass operation?

"wallge" <wallge@gmail.com> wrote in message 
news:1169766685.898182.155950@v45g2000cwv.googlegroups.com...
> Can you write out the FIR filter coeffs for
> a bilinear interpolation "filter kernel"?
> How about a bicubic interpolator filter kernel
> what are its filter coeffs?

I'm happy to, but we're getting away from FPGA stuff,
so let's do that off line. Let me know how many phases you
need, and the coefficient format you'd like. I usually
use a minor 4x4 variation on cubic, but it's all set up in
Mathematica, so I could do cubic also.

>
> arguing semantics was not the purpose of my post.
>
> I will probably wind up doing bilinear interpolation or
> "filtering". Which means I need 4 pixels of the input frame to
> determine
> 1 pixel of output warped frame.

So you don't really need coefficient tables for this.
You can just use the fractional phase directly.

>
> By the way what is the Freq response of the bilinear interpolation
> "filter"?

It depends on the position of output relative to input pixel, but
for a central output pixel the frequency response would be
Cosusoidal.

Getting back to FPGA stuff though, what are your off-chip
RAM bandwidth limitations, and could you consider a two-pass approach?

>> I'm still not sure of your limitations on off-chip RAM.
>> You have a buffer on the input or output (or both?)
>> Do you have enough bandwidth to have an
>> intermediate buffer for a two-pass operation?
> 

On Jan 24, 11:36 am, "wallge" <wal...@gmail.com> wrote:
> I am doing some embedded video processing, where I store an incoming
> frame of video, then based on some calculations in another part of the
> system, I warp that buffered frame of video. Now when the frame goes
> into the buffer
> (an off-FPGA SDRAM chip), it is simply written in one pixel at a time
> in row major ordering.
>
> The problem with this is that I will not be accessing it in this way. I
> may want to do some arbitrary image rotation. This means
>  the first pixel I want to access is not the first one I put in the
> buffer, It might actually be the last one in the buffer. If I am doing
> full page reads, or even burst reads, I will get a bunch of pixels that
> I will not need to determine the output pixel value. If i just do
> single reads, this waists a bunch of clock cycles setting up the SDRAM,
> telling it which row to activate and which column to read from. After
> the read is done, you then have to issue the precharge command to close
> the row. There is a high degree of inefficiency to this. It takes 5,
> maybe 10 clock cycles just to retrieve one
> pixel value.
>
> Does anyone know a good way to organize a frame buffer to be more
> friendly (and more optimal) to nonsequential access (like the kind we
> might need if we wanted to warp the input image via some
> linear/nonlinear transformation)?

A fairly simple technique, reasonably well-known among video system
designers, is to use what is sometimes called tiling the image into the
((DDR)S)DRAM columns.
E.g., assume a 1Kx1K image with vertical and horizontal address bits
(V9..V0) and (H9..H0), and also DRAM with row and column address bits
(R9..R0) and (C9..C0). Do _not_ use the straight mapping of:
(V9..V0) <=> (R9..R0) and (H9..H0) <=> (C9..C0)
Instead, map the H/V LSBs into the DRAM column address, and the H/V
MSBs to the DRAM row address:
(V4..V0,H4..H0) <=> (C9..C0) and (V9..V5,H9..H5) <=> (R9..R0)
When warping, the image sample addresses are pipelined out to DRAM,
with time designed into the pipeline to examine the addresses for DRAM
row boundary crossings, and only stall when it's necessary to re-RAS.
Stalls only occur when the sampling area overlaps the edge of a tile,
instead of with every 2x2 or 3x3 fetch.
(You posed your question so well, I'll bet this already occurred to
you)

Caching can also be used to bypass the external RAM access pipeline
when the required pixels are already in the FPGA. There are lots of
different caching techniques, I haven't looked at that in a while.

Block processing is a kind of variant of caching, reading a tile from
external DRAM into BRAM, warping from that BRAM into another BRAM, then
sending the results back out, but border calculations get messy for
completely arbitrary warps.

HTH
Just John

I am not sure what you mean by two pass approach.
The max (theoretical) bandwidth I have available  to/from the SDRAM
is about
16 bits * 100 Mhz = 1.6 Gbit/sec

This is not an achievable estimate of course, even if I only did full
page
reads and writes, since there is overhead associated with each. I also
have to refresh periodically.

My pixel bit width could be brought down to 8 bits. That way I could
store 2
pixels per address if need be.



On Jan 25, 7:23 pm, "Pete Fraser" <pfra...@covad.net> wrote:
> "wallge" <wal...@gmail.com> wrote in messagenews:1169766685.898182.155950@v45g2000cwv.googlegroups.com...
>
> > Can you write out the FIR filter coeffs for
> > a bilinear interpolation "filter kernel"?
> > How about a bicubic interpolator filter kernel
> > what are its filter coeffs?I'm happy to, but we're getting away from FPGA stuff,
> so let's do that off line. Let me know how many phases you
> need, and the coefficient format you'd like. I usually
> use a minor 4x4 variation on cubic, but it's all set up in
> Mathematica, so I could do cubic also.
>
>
>
> > arguing semantics was not the purpose of my post.
>
> > I will probably wind up doing bilinear interpolation or
> > "filtering". Which means I need 4 pixels of the input frame to
> > determine
> > 1 pixel of output warped frame.So you don't really need coefficient tables for this.
> You can just use the fractional phase directly.
>
>
>
> > By the way what is the Freq response of the bilinear interpolation
> > "filter"?It depends on the position of output relative to input pixel, but
> for a central output pixel the frequency response would be
> Cosusoidal.
>
> Getting back to FPGA stuff though, what are your off-chip
> RAM bandwidth limitations, and could you consider a two-pass approach?
>
> >> I'm still not sure of your limitations on off-chip RAM.
> >> You have a buffer on the input or output (or both?)
> >> Do you have enough bandwidth to have an
> >> intermediate buffer for a two-pass operation?

On Jan 26, 3:15 pm, "wallge" <wal...@gmail.com> wrote:
> I am not sure what you mean by two pass approach.
> The max (theoretical) bandwidth I have available  to/from the SDRAM
> is about
> 16 bits * 100 Mhz = 1.6 Gbit/sec
>
> This is not an achievable estimate of course, even if I only did full
> page
> reads and writes, since there is overhead associated with each. I also
> have to refresh periodically.
>
> My pixel bit width could be brought down to 8 bits. That way I could
> store 2
> pixels per address if need be.
>

You may be missing an important feature of SDRAM.  You don't need to
use full-page reads or writes to keep data streaming at 100% of the
available bandwidth (if you don't change direction) or very nearly 100%
(if you switch from read to write infrequently).  This is due to the
ability
to set up another block operation on one bank while another bank is
transferring data.  When I use SDRAM for relatively random operations
like this I like to think of the minimum data unit as one minimal burst
(two words in a single-data-rate SDRAM) to each of the four banks.
Any number of these data units can be strung one after another
with no break in the data flow.  Then if you wanted to internally
buffer
a square section of the image in internal blockRAM the width
of the minimum block (allowing 100% data rate) would only be
16 8-bit pixels or 8 16-bit pixels in your case.  If the area can
cover the required computational core (4 x 4?) for several pixels
at a time, you can reduce overall bandwidth.  This was the point
of suggesting an internal cache memory.

HTH,
Gabor

> You may be missing an important feature of SDRAM.  You don't need to
> use full-page reads or writes to keep data streaming at 100% of the
> available bandwidth (if you don't change direction) or very nearly 100%
> (if you switch from read to write infrequently).  This is due to the
> ability
> to set up another block operation on one bank while another bank is
> transferring data.  

Hi Gabor,

I've missed this too, what happens at the end of the burst?  Do you 
have another "cas" otherwise the burst will stop (unless it's full 
page)  Anyway, if it works like that then we can save bunch of cycles 
ras to cas and data transfer can be seamless

I don't know it's possible to activate another bank while one bank is 
read/written, please clarify me on this, must be my big miss :)

Thanks,

"wallge" <wallge@gmail.com> writes:

> I am not sure what you mean by two pass approach.
> The max (theoretical) bandwidth I have available  to/from the SDRAM
> is about
> 16 bits * 100 Mhz = 1.6 Gbit/sec
>
> This is not an achievable estimate of course, even if I only did full
> page
> reads and writes, since there is overhead associated with each. I also
> have to refresh periodically.
>

Don't forget that for video apps, you often don't need to refresh, as
you are reading and writing the SDRAM rows in a regular fashion which
means you can guarantee that each gets touched often enough.

Indeed for some video applications, like output framebuffers, all you
need to do is ensure that you read the row out for display soon enough
after the write, which is often easy to achieve.

Cheers,
Martin

-- 
martin.j.thompson@trw.com 
TRW Conekt - Consultancy in Engineering, Knowledge and Technology
http://www.conekt.net/electronics.html

   

Gabor,

Are you saying that I don't need to activate/precharge the bank
when switching to another?
I am kind of unclear on this. When do activate and precharge commands
need to be issued? I thought when switching to a new row or bank you 
had
to precharge (close) the previously active one, then activate the new 
row/bank before
actually reading from or writing to it. Where am I going wrong here?

Also to the notion that I don't need to refresh since I am doing video
buffering: I am actually buffering multiple frames of video and then 
reading
out several frames later. In other words, there may be a significant 
fraction
of a second (say 1/8~1/4 sec) of delay between writing data into a
particular page of memory and actually reading it back out.
Is this too much time to expect my pixel data to still be valid 
without refreshing?



On Jan 26, 6:03 pm, "Gabor" <g...@alacron.com> wrote:
> On Jan 26, 3:15 pm, "wallge" <wal...@gmail.com> wrote:
>
> > I am not sure what you mean by two pass approach.
> > The max (theoretical) bandwidth I have available  to/from the SDRAM
> > is about
> > 16 bits * 100 Mhz = 1.6 Gbit/sec
>
> > This is not an achievable estimate of course, even if I only did full
> > page
> > reads and writes, since there is overhead associated with each. I also
> > have to refresh periodically.
>
> > My pixel bit width could be brought down to 8 bits. That way I could
> > store 2
> > pixels per address if need be.You may be missing an important feature of SDRAM.  You don't need to
> use full-page reads or writes to keep data streaming at 100% of the
> available bandwidth (if you don't change direction) or very nearly 100%
> (if you switch from read to write infrequently).  This is due to the
> ability
> to set up another block operation on one bank while another bank is
> transferring data.  When I use SDRAM for relatively random operations
> like this I like to think of the minimum data unit as one minimal burst
> (two words in a single-data-rate SDRAM) to each of the four banks.
> Any number of these data units can be strung one after another
> with no break in the data flow.  Then if you wanted to internally
> buffer
> a square section of the image in internal blockRAM the width
> of the minimum block (allowing 100% data rate) would only be
> 16 8-bit pixels or 8 16-bit pixels in your case.  If the area can
> cover the required computational core (4 x 4?) for several pixels
> at a time, you can reduce overall bandwidth.  This was the point
> of suggesting an internal cache memory.
>
> HTH,
> Gabor

On Jan 29, 7:50 am, "wallge" <wal...@gmail.com> wrote:
> Are you saying that I don't need to activate/precharge the bank
> when switching to another?
> I am kind of unclear on this. When do activate and precharge commands
> need to be issued? I thought when switching to a new row or bank you
> had
> to precharge (close) the previously active one, then activate the new
> row/bank before
> actually reading from or writing to it. Where am I going wrong here?

Each bank is closed or open on a given row. As long as your accesses 
go to an already open row, you needn't precharge. Beware though you 
must close the bank after tRAS MAX ~= 70 us.

AFAICT, ppl here have been suggesting data layout schemes what will 
increase your likelyhood of hitting an open row.

> Also to the notion that I don't need to refresh since I am doing video
> buffering: I am actually buffering multiple frames of video and then
> reading
> out several frames later. In other words, there may be a significant
> fraction
> of a second (say 1/8~1/4 sec) of delay between writing data into a
> particular page of memory and actually reading it back out.
> Is this too much time to expect my pixel data to still be valid
> without refreshing?

Yes that too much. tREFI (Average periodic refresh interval) is ~ 7  
us.

Tommy