Video scaler for Spartan 3E?

Hi
Just want to do a feasibility study on whether it is possible to design and 
implement a video scaler on a Spartan 3E?
Well my tutor kind of came up with this proposal for a project of mine but 
then on my tutor's side, he has experience with fpga but I'm not sure 
whether he had designed video scaling so may not know the complexity of it.

So, feasible? And with which chip as well? (in the best of best case, i hope 
can use just a spartan starter kit :) ) 

Oh if for what images to scale..
hmm maybe videos from a camera?
or would it possible to scale video from a vcd player?


"Ken Soon" <csoon@xilinx.com> wrote in message 
news:f1bf8c$t11@cnn.xsj.xilinx.com...
> Hi
> Just want to do a feasibility study on whether it is possible to design 
> and implement a video scaler on a Spartan 3E?
> Well my tutor kind of came up with this proposal for a project of mine but 
> then on my tutor's side, he has experience with fpga but I'm not sure 
> whether he had designed video scaling so may not know the complexity of 
> it.
>
> So, feasible? And with which chip as well? (in the best of best case, i 
> hope can use just a spartan starter kit :) )
> 

On May 3, 3:49 am, "Ken Soon" <c...@xilinx.com> wrote:
> Hi
> Just want to do a feasibility study on whether it is possible to design and
> implement a video scaler on a Spartan 3E?
> Well my tutor kind of came up with this proposal for a project of mine but
> then on my tutor's side, he has experience with fpga but I'm not sure
> whether he had designed video scaling so may not know the complexity of it.
>
> So, feasible? And with which chip as well? (in the best of best case, i hope
> can use just a spartan starter kit :) )


For that you need to sample a 250kB image into RAM and read it back in
a different order.
You need to do that 25 or 30 times a second.
As long as you have 500kB of SRAM that can handle 16MB/S random access
this should be easy.
For better results you need to interpolate multiple pixels on readback
which increases the bandwidth
requirement.
Rotating the image requires not much more hardware than scaling. Both
is done with two bresenham units.

Kolja Sulimma

On 3 Mai, 03:49, "Ken Soon" <c...@xilinx.com> wrote:
> Hi
> Just want to do a feasibility study on whether it is possible to design and
> implement a video scaler on a Spartan 3E?
> Well my tutor kind of came up with this proposal for a project of mine but
> then on my tutor's side, he has experience with fpga but I'm not sure
> whether he had designed video scaling so may not know the complexity of it.
>
> So, feasible? And with which chip as well? (in the best of best case, i hope
> can use just a spartan starter kit :) )

as a starter you may look at "rotazoom" IP included in demo designs of
spartan-3A kit.

it takes bitmap image from memory and display it zoomed and rotated on
display.

Antti

On a sunny day (3 May 2007 01:20:54 -0700) it happened "comp.arch.fpga"
<ksulimma@googlemail.com> wrote in
<1178180454.117412.95440@h2g2000hsg.googlegroups.com>:

>On May 3, 3:49 am, "Ken Soon" <c...@xilinx.com> wrote:
>> Hi
>> Just want to do a feasibility study on whether it is possible to design and
>> implement a video scaler on a Spartan 3E?
>> Well my tutor kind of came up with this proposal for a project of mine but
>> then on my tutor's side, he has experience with fpga but I'm not sure
>> whether he had designed video scaling so may not know the complexity of it.
>>
>> So, feasible? And with which chip as well? (in the best of best case, i hope
>> can use just a spartan starter kit :) )
>
>
>For that you need to sample a 250kB image into RAM and read it back in
>a different order.
>You need to do that 25 or 30 times a second.
>As long as you have 500kB of SRAM that can handle 16MB/S random access
>this should be easy.
>For better results you need to interpolate multiple pixels on readback
>which increases the bandwidth
>requirement.
>Rotating the image requires not much more hardware than scaling. Both
>is done with two bresenham units.
>
>Kolja Sulimma

I dunno, I think you can do that with much less memory, say we scale 1 to 2,
then you only need to strore 2 picture lines of the original:

ABCDEFG
HIJKLMN


Now you start averaging first pixel out = (A+B+H+I) / 4,
second pixel out = (B+C+I+J) / 4.

etc, In this case you only need to store 2 lines.
For other factors you need some more lines.
Do not forget the aliasing factor, you are sub sampling, and need to lowpass
at below f/2.
 

Ken,

  Just out of curiosity, what is your timeline on this project??
You've been asking about this for months on here! (That's what all
your DDR interface questions were about, no?)

   There's all sorts of considerations here...

   What is your input data format? Composite video? RGB? etc.. Some
sort of computer driven RGB input would probably be your simplest as
you'll be receiving your three colors seperately and you can just
sample them as 3 input streams.  But you're gonna have lots of
learning and educating yourself to do on sync timing and all that fun
stuff - Keep in mind, most analog video signals were designed to be
fairly crude signals to drive a big honking electromagnetic coil that
certainly didnt respond anything close to the clock rate of your
spartan.  Therefore, putting them into the digital world in some sorta
of useful form can be a bit tricky - basically you need to look for
your syncs and count out to where your first pixel is supposed to
be.

   What are your input & output resolutions? frame rates?  If you're
going 10x10 to 20x20 at 5 fps, I'd bet you could do that in a
Spartan3... 640x480 input at a 60hz refresh, I don't really know -
that's where you're going to have to sit down and run some
calcuations.  Go to your school's library and take out some books on
image processing or multi-dimensional DSP... you're gonna need to
learn some basic image processing first to get an idea of what
operations need to be done before you can begin to grasp the size of
the part you need.

   How accurate do your interpolations need to be?  Think basic 1-
dimensional DSP.. how do you upsample a signal?  insert zeros between
samples and LPF, right?  Or do the same thing in the F-domain by zero
padding the ends of the FFT and inverse-FFT'ing.  Either of these
methods in 2 dimensions is going to be a lot of computation and a lot
of taking stuff in and out of memory in different orders.   DDR
memories like you to take stuff out in the same order you put them
in... they slow down big time when you try to jump around..  So if you
do this, you will need to some up with some clever methods of read out
pieces from DDR in the incorrect order, and then re-reading from a
local, smaller, block ram in the order you actually want.  This will
take careful planning and a lot of simulation, even an experienced
designer would have a tricky time with this and probably get it wrong
in simulation the first shot.

   If your up/down sampling doesn't need to be terribly accurate, this
simplifies things.  Consider a simple first order interpolation...
look left, look right and figure out what goes in between.  You would
do the row dimension as the data flows in, just put two rows into
memory at a time and then do your column dimension.  Obviously, you
will need to calculate bandwidth here and make sure you've got plenty
enough (with margin!) on your memory.

   Also, remember that multiplication & division are tricky in the
binary world.  The spartan3 does have some built in multipliers.  But
for a school project consider limiting your scope to make your life a
little easier.  Maybe you only scale by powers of 2, so everything
becomes a Left/Right shift instead of mult/div?  Maybe you limit it
even more, you take 640x480 60Hz in and you ouptut 1280x960 60Hz, and
do nothing else.  The fewer possible cases you need to accoutn for the
more likely it will be to fit in a smaller part.

  Lastly, are you trying to do this on the spartan3 starter kit?
becuase that has no video in, and only a crude video out (no DAC, you
can just turn on and off the RGB pixels in combinations.. so 2^3 = 8
colors, of which 1 is black and 1 is white)...   If you are building
your own board you are going to be limitted by the size FPGA you can
get in a quad flat pack part - i assume you dont have the facilities
to do a BGA - and those QFPs are still VERY fine pitch to solder by
hand.  Let me know if you've found an eval board already that's
suitable for this application, I;m curious - I don't think any of the
digilent spartan boards have video input.

  Good luck



On May 2, 9:49 pm, "Ken Soon" <c...@xilinx.com> wrote:
> Hi
> Just want to do a feasibility study on whether it is possible to design and
> implement a video scaler on a Spartan 3E?
> Well my tutor kind of came up with this proposal for a project of mine but
> then on my tutor's side, he has experience with fpga but I'm not sure
> whether he had designed video scaling so may not know the complexity of it.
>
> So, feasible? And with which chip as well? (in the best of best case, i hope
> can use just a spartan starter kit :) )

On May 2, 9:53 pm, "Ken Soon" <c...@xilinx.com> wrote:
> Oh if for what images to scale..
> hmm maybe videos from a camera?
> or would it possible to scale video from a vcd player?
>
> "Ken Soon" <c...@xilinx.com> wrote in message
>
> news:f1bf8c$t11@cnn.xsj.xilinx.com...
>
> > Hi
> > Just want to do a feasibility study on whether it is possible to design
> > and implement a video scaler on a Spartan 3E?
> > Well my tutor kind of came up with this proposal for a project of mine but
> > then on my tutor's side, he has experience with fpga but I'm not sure
> > whether he had designed video scaling so may not know the complexity of
> > it.
>
> > So, feasible? And with which chip as well? (in the best of best case, i
> > hope can use just a spartan starter kit :) )


I have 2 data points for you nased on existing designs with Spartan 2e
and Virtex 2.  The latter is more closely related to the Spartan 3.
In
that design I fit a scaler / combiner taking 4 RGB inputs each 800 x
600
at up to 120 fps.  The inputs were not synchronized so they needed
to be stored in SDRAM first.  The output was scaled between 1:2
and 2:1 (50 - 200%) and mixed into a single 800 x 600 output frame
DMA'd into a PC using a PLX9656 local bus adapter.  This fit pretty
handily in a Virtex 2 1000 using DDR SDRAM (1 64-bit SO-DIMM).

The other design was standalone using a Spartan 2e 400.  This design
takes two 640 x 480 inputs stores them in single-data-rate SDRAM
and then combines them into a scaled 1024 by 768 output at
60 fps.

Both designs use a low-pass filter followed by linear interpolation
in each dimension (horizontal and vertical).  For the horizontal
filtering I used distributed arithmetic filters to save logic.

So, given enough time and design expertise I would think you
could easily come up with a scaler for TV (even HDTV) video
in a Spartan 3 400 or larger assuming you have the other
required resources.  Usually you'll find that you run out of
block RAM before you run out of fabric in this sort of design.
Also without the DA FIR, you could run out of multipliers.

Have fun,
Gabor

"Paul" <pauljbennett@gmail.com> wrote in message 
news:1178208971.772258.206100@h2g2000hsg.googlegroups.com...
> Ken,
>
>  Just out of curiosity, what is your timeline on this project??
> You've been asking about this for months on here! (That's what all
> your DDR interface questions were about, no?)

You remembered my posts!
Ok, anyway, well actually I had started with this project for my attachment 
which lasted for about 2-3 months and it nearing its end and don't forsee 
completing it.
And next year, I had another full-fledged project assignment which will take 
about ,taking on the safe side, about 10months? So I thought of trying to 
progressing on for on my current project design (trying to learn how it 
works) or maybe even create my own video scaler myself on a cheaper 3E 
starter kit than the 1600E inrevium board that i'm using now.

Hmm, well, not sure on the input data format. I would like to scale a 
composite video but if i can scale RGB format, I would be satisfied as I'm 
still really new in this field. Sync timing, hmm I did some tiny demo using 
other people's design which had a hv sync generator, maybe have to study it 
more.

Input & output resolutions and frame rates? I would like to start with 
640x480 and 60Hz since it seems basic enough to start with this (though i 
would like to cater to lower resolution with progress)
Gabor mentioned that he did it with Spartan 2e 400 which took in two 640 x 
480 inputs, stores them in single-data-rate SDRAM and then combines them 
into a scaled 1024 by 768 output at 60 fps.

>   How accurate do your interpolations need to be?  Think basic 1-
> dimensional DSP.. how do you upsample a signal?  insert zeros between
> samples and LPF, right?  Or do the same thing in the F-domain by zero
> padding the ends of the FFT and inverse-FFT'ing.  Either of these
> methods in 2 dimensions is going to be a lot of computation and a lot
> of taking stuff in and out of memory in different orders.   DDR
> memories like you to take stuff out in the same order you put them
> in... they slow down big time when you try to jump around..  So if you
> do this, you will need to some up with some clever methods of read out
> pieces from DDR in the incorrect order, and then re-reading from a
> local, smaller, block ram in the order you actually want.  This will
> take careful planning and a lot of simulation, even an experienced
> designer would have a tricky time with this and probably get it wrong
> in simulation the first shot.

Woah, ok I'm totally lost in here. But no problem, I will take note of these 
points and consult my profs or someone

 First step i got to take will just be a simple fixed resolution scaling of 
one resolution set, like you said 1280x960.
Well I'm not sure if i will be lent a expensive board with the 1600E on it 
or the starter kit.
Oh I saw that the starter kit does have a DAC and also an ADC.
Oh and it does have a Hirose FX2 connector then maybe I can add the digilent 
video decoder for S video,

Thanks alot Paul for providing the considerations that I will need to take 
note and the immense amount of advices.
Btw, just a side note though, for my current project, I have reduced the 
Htaps and Vtaps to 4 & 3. Minimum frequency is about 177Mhz, above the 
minimum requirement os 167Mhz. Now i'm just stuck with how to assign the 
pins (headers, h v coefficients, seq lut address,....) and how to actually 
use this scaler.

Hmm Gabor, even a scaler for HDTV on a Spartan 3e?!...
oh btw, combining two 640x480 to a scaled 1024x768, isn't the usual way to 
interpolate from a single 640x480 source to a 1024x768?

oh and regarding SDRAM, i should have enough on even a starter kit with 
512Mbit SDRAM available.

And aliasing factor, sub-sampling, and lowpass at below f/2...hmm ok will 
take notes of these too, Jan

Thanks alot guys for the help!~ much appreciated :) 

On 4 Mai, 10:01, "Ken Soon" <c...@xilinx.com> wrote:
> "Paul" <pauljbenn...@gmail.com> wrote in message

> Input & output resolutions and frame rates? I would like to start with
> 640x480 and 60Hz since it seems basic enough to start with this (though i
> would like to cater to lower resolution with progress)
> Gabor mentioned that he did it with Spartan 2e 400 which took in two 640 x
> 480 inputs, stores them in single-data-rate SDRAM and then combines them
> into a scaled 1024 by 768 output at 60 fps.

As another poster mentioned: If input and output run at the same
resolution and
frame rate you only need to store a small number of video lines. You
can do that
in internal memory.
Start with a project that samples the video input and directly outputs
it in the desired
format. Than do the same with one video line delay.
Once you have that working your scaler is more or less done.

Kolja Sulimma

OK... think time domain signal...  say your signal x = [1 10 5 3] and
you want to upsample this to twice the sampling frequency....  if you
did a simple linear interpolation you'd get x_upsampled = [1 5.5 10
7.5 5 4 3 ?]  that last datapoint{?} is kinda arbitrary, presumably
you'd just set it to 3.   However, signals are defined as a sum of
sinusoids, and generally they fit that model fairly well - when you
have something that doesn't, it takes more sinusoids to fit it.. thats
the general theory behind a fourier transform, right?  So linear
interpolation isn't really you're best solution.  You can
mathematically prove that if you had an infinitely long signal, and an
infinitely perfect low pass filter, the "perfect" upsampling is done
by inserting zeros  x_zero = [1 0 10 0 5 0 3 0], and low pass
fitlering it.  Obviously, neither the infinite sequence or the perfect
LPF is realistic.  If you do the math (or plug it into matlab), the
FFT of x (shfited to place zero frequency in the center), X=[-7 -4+7j
19 -4-7j]  If you take the FFT of x_zero, X_ZERO = [19   -4-7j   -7
-4+7j  19   -4-7j   -7   -4+7j]    Note, this is simply X repeated.
So how do you get back your correct frequency spectrum?  An ideal LPF
returns X_UPSAMPLE=[0    0   -7 -4+7j 19 -4-7j   0   0].   Now note
this... if you use matlab and take the inverse FFT of this (the
shifted version of this actually)  you get [1.0000    5.8107
10.0000    8.6391    5.0000    3.6893    3.0000    0.8609]
Actually... you get this divided by 2  (you need to multiply by your
upsampling facor, that comes out in the math) with a risidual
imaginary part due to the fact that it's an even length sequence, so
you just look at the real part.   Point being however, note that the
interpolated values are NOT what you get using straight linear
interpolation.  this is because this method takes into account the
curvature of the rest of the sequence and stuff.

That being said.... for this project, I would imagine that simple
linear interpolation is plenty adequate!  A 2 dimensional version of
this example above would be quite tricky and take a lot of processing
power.  But read the chapter in your DSP book on upsampling - it will
cover all this :-)  Actually, not positive, but the stock, every
college in the world uses, oppenheim & shaeffer signals & systems book
MIGHT even go into it in one of the later chapters.


> >   How accurate do your interpolations need to be?  Think basic 1-
> > dimensional DSP.. how do you upsample a signal?  insert zeros between
> > samples and LPF, right?  Or do the same thing in the F-domain by zero
> > padding the ends of the FFT and inverse-FFT'ing.  Either of these
> > methods in 2 dimensions is going to be a lot of computation and a lot
> > of taking stuff in and out of memory in different orders.   DDR
> > memories like you to take stuff out in the same order you put them
> > in... they slow down big time when you try to jump around..  So if you
> > do this, you will need to some up with some clever methods of read out
> > pieces from DDR in the incorrect order, and then re-reading from a
> > local, smaller, block ram in the order you actually want.  This will
> > take careful planning and a lot of simulation, even an experienced
> > designer would have a tricky time with this and probably get it wrong
> > in simulation the first shot.
>
> Woah, ok I'm totally lost in here. But no problem, I will take note of these
> points and consult my profs or someone