Debugging SDRAM interfaces

Hi guys,
I could really use some help from an SDRAM / FPGA guru here...

I've got an SDRAM controller IP core -- specifically, the sdram_wb core 
by Stephen Williams, available from the Git repository
<git://icarus.com/~steve-icarus/sdram_wb.git>. This core works fine on 
the Altera DE1, with a 16-bit-wide 64Mbit (1M*16*4bank) PowerChip SDRAM, 
P/N A2V64S40CTP. The FPGA is a Cyclone II 2C20, and all I had to do to 
make the core work was add a PLL to shift the SDRAM clock by 2ns.

Unfortunately (for me), I hit the limits of what the DE1 could do -- I 
got the SoC working, but ran out of space for the LCD controller 
(framebuffer) and audio controller. So I've started playing with the 
Enterpoint Drigmorn2 board, which uses a Xilinx Spartan3A XC3S700A FPGA, 
and a 32Mbyte (2M*32*4bank) 32-bit-wide ISSI SDRAM chip, the IS42S32800B.

The problem is, while all the other SoC logic (CPU, etc.) seems to work 
fine, the SDRAM interface just plain won't work. The WISHBONE interface 
works great (as in, I can R/W the cache, which stores 32 words, split 
into 4 blocks), but if I do a write/readback that requires a RAM access, 
the readback returns garbage:

----8<----

SDRAM controller configuration:
        Status register:  0x00000833
                CAS latency         = 3
                Column address bits = 9
                Enabled and ready
        Refresh Timer:    390 clock cycles
        CONFIG1:          0x00001010
                Trp_clk  = 0
                Trfc_clk = 1
                Trcd_clk = 0
                Tras_clk = 1
Cmd> w
Write to memory
Enter address (0x40000000 is added automatically): 0x00000000
Value: 0x55AABBCC
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000000
Value: 0x55AABBCC
Cmd> w
Write to memory
Enter address (0x40000000 is added automatically): 0x00000040
Value: 0xBBCC55AA
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000040
Value: 0xBBCC55AA
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000000
Value: 0x55AABBCC

Cmd> w
Write to memory
Enter address (0x40000000 is added automatically): 0x00000080
Value: 0x12345678
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000000
Value: 0x7AEDB9CC
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000080
Value: 0x7AFFBDFE

----8<----

Interestingly, as long as I only write to data in the address range 
[0x00..0x80], the RAM works fine. As soon as I write to address 0x80, all 
bets are off. The data is not the same each time, but it does remain 
static:

----8<----
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000000
Value: 0xBD2015AA
Cmd> w
Write to memory
Enter address (0x40000000 is added automatically): 0x00000000
Value: 0x12345678
Cmd> w
Write to memory
Enter address (0x40000000 is added automatically): 0x00000080
Value: 0x89ABCDEF
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000000
Value: 0xBD301068
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000000
Value: 0xBD301068
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000080
Value: 0x8830BDFF
Cmd> q
Read from memory
Enter address (0x40000000 is added automatically): 0x00000000
Value: 0xBD301068
----8<----

Ordinarily I'd start by checking for timing issues by soldering wires to 
the pins on the RAM IC, then hooking up the logic analyser. This isn't an 
option in this case, because the chip is a BGA -- I physically can't get 
to the pins/pads/balls/whatever. I'd rather not remove the chip because I 
don't have much experience with BGA rework and I'd probably ruin the 
(rather expensive) development board.

I have tried routing the clocks and SDRAM data lines onto the LHS and RHS 
header pins, then captured the data with my logic analyser -- this shows 
that the timing is fine, and also shows that the data being returned is 
not the same as that written to the SDRAM. I can do a few screen captures 
and put these online if it helps, although I'll openly admit that timing 
on the headers =/= timing at the SDRAM (for a start, they're going to be 
going through different I/O buffers).

I've also checked the refresh timing -- it's about 15.6 microseconds 
(with a 25MHz input clock). I've tried reducing the refresh-timer value 
to ~12us but this has no effect on the failure symptoms.

This is how I have the clocking set up:
// Core clock DCM
dcm_core_clock DCM_CoreClockGen (
	.CLKIN_IN			(CLOCK), 
	.RST_IN				(DCM_RESET), 
	.CLKFX_OUT			(MCLK), 
	.CLKIN_IBUFG_OUT	(CLOCK_B),
	.CLK0_OUT			(), 
	.LOCKED_OUT			(MCLK_LOCKED)
	);

// SDRAM clock DCM -- skews the master clock for the SDRAM
dcm_sdram DCM_SDRAMClockGen (
	.CLKIN_IN			(MCLK),
	// Hold SDRAM DCM in reset until master DCM has locked
	.RST_IN				(DCM_RESET || !MCLK_LOCKED),
	.CLK0_OUT			(),
	.CLK90_OUT			(),
	.CLK180_OUT			(),
	.CLK270_OUT			(SDRAM_CLK),
	.LOCKED_OUT			(SDRAM_CLK_LOCKED)
	);

CLOCK is a 25MHz clock input from a crystal oscillator. The Core Clock 
DCM is a throwback to an earlier design which ran at 50MHz, but is 
currently set up as a "1:1" multiplier (if you can call it that).

The SDRAM DCM is set up to mirror the input clock on the output, with 
different phase shifts. At the moment I'm using the 270deg output, but 
I've also tried the 0deg and 90deg outputs with no success.

SDRAM_CLK is the output pin for the SDRAM clock.

Can anyone offer any suggestions for other things I could try (or check)?
I'm happy to re-synth the logic or provide logic analyser traces, HDL 
source, etc. to help out -- but bear in mind I'm limited to 40 logic 
analyser 'bits' at a time (the Drigmorn2 only has 40 user I/Os).

Thanks,
-- 
Phil.
usenet10@philpem.me.uk
http://www.philpem.me.uk/
If mail bounces, replace "10" with the last two digits of the current year

Hi,

Check system memory map. Maybe somthing is covered with something else.

Adam
> Hi guys,
> I could really use some help from an SDRAM / FPGA guru here...
> 
> I've got an SDRAM controller IP core -- specifically, the sdram_wb core 
> by Stephen Williams, available from the Git repository
> <git://icarus.com/~steve-icarus/sdram_wb.git>. This core works fine on 
> the Altera DE1, with a 16-bit-wide 64Mbit (1M*16*4bank) PowerChip SDRAM, 
> P/N A2V64S40CTP. The FPGA is a Cyclone II 2C20, and all I had to do to 
> make the core work was add a PLL to shift the SDRAM clock by 2ns.
> 
> Unfortunately (for me), I hit the limits of what the DE1 could do -- I 
> got the SoC working, but ran out of space for the LCD controller 
> (framebuffer) and audio controller. So I've started playing with the 
> Enterpoint Drigmorn2 board, which uses a Xilinx Spartan3A XC3S700A FPGA, 
> and a 32Mbyte (2M*32*4bank) 32-bit-wide ISSI SDRAM chip, the IS42S32800B.
> 
> The problem is, while all the other SoC logic (CPU, etc.) seems to work 
> fine, the SDRAM interface just plain won't work. The WISHBONE interface 
> works great (as in, I can R/W the cache, which stores 32 words, split 
> into 4 blocks), but if I do a write/readback that requires a RAM access, 
> the readback returns garbage:
> 
> ----8<----
> 
> SDRAM controller configuration:
>         Status register:  0x00000833
>                 CAS latency         = 3
>                 Column address bits = 9
>                 Enabled and ready
>         Refresh Timer:    390 clock cycles
>         CONFIG1:          0x00001010
>                 Trp_clk  = 0
>                 Trfc_clk = 1
>                 Trcd_clk = 0
>                 Tras_clk = 1
> Cmd> w
> Write to memory
> Enter address (0x40000000 is added automatically): 0x00000000
> Value: 0x55AABBCC
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000000
> Value: 0x55AABBCC
> Cmd> w
> Write to memory
> Enter address (0x40000000 is added automatically): 0x00000040
> Value: 0xBBCC55AA
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000040
> Value: 0xBBCC55AA
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000000
> Value: 0x55AABBCC
> 
> Cmd> w
> Write to memory
> Enter address (0x40000000 is added automatically): 0x00000080
> Value: 0x12345678
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000000
> Value: 0x7AEDB9CC
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000080
> Value: 0x7AFFBDFE
> 
> ----8<----
> 
> Interestingly, as long as I only write to data in the address range 
> [0x00..0x80], the RAM works fine. As soon as I write to address 0x80, all 
> bets are off. The data is not the same each time, but it does remain 
> static:
> 
> ----8<----
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000000
> Value: 0xBD2015AA
> Cmd> w
> Write to memory
> Enter address (0x40000000 is added automatically): 0x00000000
> Value: 0x12345678
> Cmd> w
> Write to memory
> Enter address (0x40000000 is added automatically): 0x00000080
> Value: 0x89ABCDEF
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000000
> Value: 0xBD301068
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000000
> Value: 0xBD301068
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000080
> Value: 0x8830BDFF
> Cmd> q
> Read from memory
> Enter address (0x40000000 is added automatically): 0x00000000
> Value: 0xBD301068
> ----8<----
> 
> Ordinarily I'd start by checking for timing issues by soldering wires to 
> the pins on the RAM IC, then hooking up the logic analyser. This isn't an 
> option in this case, because the chip is a BGA -- I physically can't get 
> to the pins/pads/balls/whatever. I'd rather not remove the chip because I 
> don't have much experience with BGA rework and I'd probably ruin the 
> (rather expensive) development board.
> 
> I have tried routing the clocks and SDRAM data lines onto the LHS and RHS 
> header pins, then captured the data with my logic analyser -- this shows 
> that the timing is fine, and also shows that the data being returned is 
> not the same as that written to the SDRAM. I can do a few screen captures 
> and put these online if it helps, although I'll openly admit that timing 
> on the headers =/= timing at the SDRAM (for a start, they're going to be 
> going through different I/O buffers).
> 
> I've also checked the refresh timing -- it's about 15.6 microseconds 
> (with a 25MHz input clock). I've tried reducing the refresh-timer value 
> to ~12us but this has no effect on the failure symptoms.
> 
> This is how I have the clocking set up:
> // Core clock DCM
> dcm_core_clock DCM_CoreClockGen (
> 	.CLKIN_IN			(CLOCK), 
> 	.RST_IN				(DCM_RESET), 
> 	.CLKFX_OUT			(MCLK), 
> 	.CLKIN_IBUFG_OUT	(CLOCK_B),
> 	.CLK0_OUT			(), 
> 	.LOCKED_OUT			(MCLK_LOCKED)
> 	);
> 
> // SDRAM clock DCM -- skews the master clock for the SDRAM
> dcm_sdram DCM_SDRAMClockGen (
> 	.CLKIN_IN			(MCLK),
> 	// Hold SDRAM DCM in reset until master DCM has locked
> 	.RST_IN				(DCM_RESET || !MCLK_LOCKED),
> 	.CLK0_OUT			(),
> 	.CLK90_OUT			(),
> 	.CLK180_OUT			(),
> 	.CLK270_OUT			(SDRAM_CLK),
> 	.LOCKED_OUT			(SDRAM_CLK_LOCKED)
> 	);
> 
> CLOCK is a 25MHz clock input from a crystal oscillator. The Core Clock 
> DCM is a throwback to an earlier design which ran at 50MHz, but is 
> currently set up as a "1:1" multiplier (if you can call it that).
> 
> The SDRAM DCM is set up to mirror the input clock on the output, with 
> different phase shifts. At the moment I'm using the 270deg output, but 
> I've also tried the 0deg and 90deg outputs with no success.
> 
> SDRAM_CLK is the output pin for the SDRAM clock.
> 
> Can anyone offer any suggestions for other things I could try (or check)?
> I'm happy to re-synth the logic or provide logic analyser traces, HDL 
> source, etc. to help out -- but bear in mind I'm limited to 40 logic 
> analyser 'bits' at a time (the Drigmorn2 only has 40 user I/Os).
> 
> Thanks,

On Fri, 21 May 2010 13:40:04 +0200, Górski Adam wrote:
> Check system memory map. Maybe somthing is covered with something else.

Nope -- replacing it with a Block RAM (the same WISHBONE RAM core I use 
for the "safe zone" boot RAM at 0x30000000) makes the RAM test work ("RAM 
OK, 8KiB detected"). That would seem to suggest that the problem is 
either:
  - in the SDRAM controller logic
  - in the SDRAM clock drive logic / IOBs
  - on the PCB itself

:-/

-- 
Phil.
usenet10@philpem.me.uk
http://www.philpem.me.uk/
If mail bounces, replace "10" with the last two digits of the current year

Philip Pemberton pisze:
> On Fri, 21 May 2010 13:40:04 +0200, Górski Adam wrote:
>> Check system memory map. Maybe somthing is covered with something else.
> 
> Nope -- replacing it with a Block RAM (the same WISHBONE RAM core I use 
> for the "safe zone" boot RAM at 0x30000000) makes the RAM test work ("RAM 
> OK, 8KiB detected"). That would seem to suggest that the problem is 
> either:
>   - in the SDRAM controller logic
>   - in the SDRAM clock drive logic / IOBs
>   - on the PCB itself
> 
> :-/
> 
Check timings on both boards in result log file and then make proper 
constraints.

Best regards

Adam Górski

On 21 May 2010 11:15:11 GMT, Philip Pemberton <usenet10@philpem.me.uk>
wrote:

>Hi guys,
>I could really use some help from an SDRAM / FPGA guru here...
>
>I've got an SDRAM controller IP core -- specifically, the sdram_wb core 
>by Stephen Williams, available from the Git repository
><git://icarus.com/~steve-icarus/sdram_wb.git>. This core works fine on 
>the Altera DE1, with a 16-bit-wide 64Mbit (1M*16*4bank) PowerChip SDRAM, 
>P/N A2V64S40CTP. The FPGA is a Cyclone II 2C20, and all I had to do to 
>make the core work was add a PLL to shift the SDRAM clock by 2ns.

>Unfortunately (for me), I hit the limits of what the DE1 could do -- I 
>got the SoC working, but ran out of space for the LCD controller 
>(framebuffer) and audio controller. So I've started playing with the 
>Enterpoint Drigmorn2 board, which uses a Xilinx Spartan3A XC3S700A FPGA, 
>and a 32Mbyte (2M*32*4bank) 32-bit-wide ISSI SDRAM chip, the IS42S32800B.

What happens in simulation?

Simulating SDRAM should be OK - use the simulation model for the nearest
equivalent chip from Micron if you can't get one from ISSI.

Simulating DDR devices gets messy because the vendors don't supply VHDL
models, so you need a mixed-language capable sim, unless you're using
Verilog.

If it's as simple as clock phase timing you may get away with
experiments on the hardware, using the DCM variable phase shift, but I
would recommend simulation first.

- Brian

On Fri, 21 May 2010 19:48:42 +0100, Brian Drummond wrote:

> What happens in simulation?

I don't have a full simulation model of the CPU, SDRAM controller and so 
on. There is a testbench included with sdram_wb, but it uses a completely 
different chip (Micron MT48LC2M32B2, 512K*32*4banks) and I haven't 
managed to find a model of the ISSI SDRAM. Good ol' ISSI don't release 
Verilog models of their chips so I'm having to make do... :-/

The nearest I've found in Micron's portfolio is the MT48LC8M32B2, which 
matches for bank size but doesn't check the refresh timing. I've run the 
testbench with that model, and it passed all five test runs (it tests 
five times with different refresh timings).

Do I need to set up any timing constraints in ISE for the SDRAM? If so, 
what do I need to set up?

I've got a UCF that specifies the pin parameters (DRIGMORN2.UCF), but 
doesn't specify any timing requirements (not even the frequency of the 
input clock). The Xilinx documentation is as clear as mud on this, and I 
can't find any good application notes on the subject (but I found tons of 
DDR/DDR2 SDRAM appnotes)...

-- 
Phil.
usenet10@philpem.me.uk
http://www.philpem.me.uk/
If mail bounces, replace "10" with the last two digits of the current year

OK, this is nuts...

With ISE Synthesizer set up like this:
  Optimisation Goal:   AREA
  Optimisation Effort: NORMAL

The core works fine (the timing is a little out, but not bad enough to 
pooch the whole thing). If I set it up like this:
  Optimisation Goal:   SPEED
  Optimisation Effort: NORMAL

Then the whole thing stops working -- it outright fails to read/write the 
SDRAM. I can access the SDRAM controller's cache (32 bytes of the current 
page), but accessing an out-of-page address returns garbage.

If I do the same thing on Quartus? Well, the timing looks better in SPEED 
mode, but it still works fine on the DE1.

What the *bleep* is going on?

-- 
Phil.
usenet10@philpem.me.uk
http://www.philpem.me.uk/
If mail bounces, replace "10" with the last two digits of the current year

The constraints you should have in the ucf are the input clock fequency,
the pin constraints and the IO types you are using for those pins. If the
design passes simulation and meets timing once you have run P&R then you
should be ok. 

Jon
	   
					
---------------------------------------		
Posted through http://www.FPGARelated.com

Philip Pemberton <usenet10@philpem.me.uk> wrote:

>OK, this is nuts...
>
>With ISE Synthesizer set up like this:
>  Optimisation Goal:   AREA
>  Optimisation Effort: NORMAL
>
>The core works fine (the timing is a little out, but not bad enough to 
>pooch the whole thing). If I set it up like this:
>  Optimisation Goal:   SPEED
>  Optimisation Effort: NORMAL
>
>Then the whole thing stops working -- it outright fails to read/write the 
>SDRAM. I can access the SDRAM controller's cache (32 bytes of the current 
>page), but accessing an out-of-page address returns garbage.
>
>If I do the same thing on Quartus? Well, the timing looks better in SPEED 
>mode, but it still works fine on the DE1.
>
>What the *bleep* is going on?

You probably have unconstrained paths which meet timing or not
depending on the routing mode. It is crucial that all paths in an FPGA
have timing constraints (including paths from the inputs to the
flipflops and flipflops to the outputs).

-- 
Failure does not prove something is impossible, failure simply
indicates you are not using the right tools...
nico@nctdevpuntnl (punt=.)
--------------------------------------------------------------

On May 21, 6:19=A0pm, Philip Pemberton <usene...@philpem.me.uk> wrote:
> OK, this is nuts...
>
> With ISE Synthesizer set up like this:
> =A0 Optimisation Goal: =A0 AREA
> =A0 Optimisation Effort: NORMAL
>
> The core works fine (the timing is a little out, but not bad enough to
> pooch the whole thing). If I set it up like this:
> =A0 Optimisation Goal: =A0 SPEED
> =A0 Optimisation Effort: NORMAL
>
> Then the whole thing stops working -- it outright fails to read/write the
> SDRAM. I can access the SDRAM controller's cache (32 bytes of the current
> page), but accessing an out-of-page address returns garbage.
>
> If I do the same thing on Quartus? Well, the timing looks better in SPEED
> mode, but it still works fine on the DE1.
>
> What the *bleep* is going on?
>
> --
> Phil.
> usene...@philpem.me.ukhttp://www.philpem.me.uk/
> If mail bounces, replace "10" with the last two digits of the current yea=
r

As others have mentioned, you probably have some unconstrained
paths causing timing violations.  If you think you have enough
constraints, but still have this problem, try setting up the
post place&route static timing report for Verbose and enter
a good size number like 100 in the option "report unconstrained
paths".  Then if you find an unconstrained path that probably
should be constrained you will know what to add to your
constraints.

As for SPEED vs. AREA, in Xilinx FPGA's you very often
get the best overall timing results using AREA optimization
rather than speed.  This is probably because the route
portion of your total path delay is large.  This shows up
in larger designs and larger parts especially since the
worst case routing delays grow with the design size.

Regards,
Gabor