Hi,

I have a custom made PCIe board with a Virtex 5 FPGA on which I
implemented a DMA unit which uses the PCIe endpoint block plus v1.14.
I also implemented simple read/write operations from the PC to the
board (the board responds with completion TLPs). The read/write
operations are working, DMA is not working (transferring data from
FPGA to PC).

The board is inserted in a pc with Windows 7 64 bits platform. An
application allocates virtual memory and passes a pointer to the
memory block to the driver. The driver locks the memory pages and
creates a scatter-gather list with physical addresses by using the DMA
adapter structure. The physical addresses are written to the FPGA.

When I start a DMA operation by writing a register in the FPGA, I can
see in chipscope the correct physical addresses in the TLP header (of
the memory write requests). However, I do not see the correct values
in the allocated memory at the PC. What can I do to check where it is
going wrong?

In my opinion there are two possibilities: either the TLP is blocked
by a PCIe switch at the main board or the data is available at another
memory location.

I also tried the other direction (send memory read requests TLPs from
the FPGA to the PC and receive completion TLPs as answer at the memory
read request). In the completion TLPs I see the correct data (data I
wrote into the PC memory before starting the DMA operation).

Any suggestions/ideas are welcome.

Thanks in advance,

Frank

Frank van Eijkelenburg
<f...@gmail.com> wrote:
 
> I have a custom made PCIe board with a Virtex 5 FPGA on which I
> implemented a DMA unit which uses the PCIe endpoint block plus v1.14.
> I also implemented simple read/write operations from the PC to the
> board (the board responds with completion TLPs). The read/write
> operations are working, DMA is not working (transferring data from
> FPGA to PC).
(snip)
 
> When I start a DMA operation by writing a register in the FPGA, I can
> see in chipscope the correct physical addresses in the TLP header (of
> the memory write requests). However, I do not see the correct values
> in the allocated memory at the PC. What can I do to check where it is
> going wrong?

Not having tried to do DMA through PCI before, is data being
written, but the wrong data?

I would try writing all zeros or all ones and see if those come
through fine.  It could be timing between the FPGA and PCI such
that the wrong data is being latched.

Then try slightly less predictable data and see what gets through.

-- glen

On Aug 2, 5:28=A0pm, glen herrmannsfeldt
<g...@ugcs.caltech.edu> wrote:
> Frank van Eijkelenburg <fei.technolut...@gmail.com> wrote:
>
> > I have a custom made PCIe board with a Virtex 5 FPGA on which I
> > implemented a DMA unit which uses the PCIe endpoint block plus v1.14.
> > I also implemented simple read/write operations from the PC to the
> > board (the board responds with completion TLPs). The read/write
> > operations are working, DMA is not working (transferring data from
> > FPGA to PC).
>
> (snip)
>
> > When I start a DMA operation by writing a register in the FPGA, I can
> > see in chipscope the correct physical addresses in the TLP header (of
> > the memory write requests). However, I do not see the correct values
> > in the allocated memory at the PC. What can I do to check where it is
> > going wrong?
>
> Not having tried to do DMA through PCI before, is data being
> written, but the wrong data?

That is what I do not know. Yes the correct data is send to the PC,
but if I readout the memory the values are unchanged.

>
> I would try writing all zeros or all ones and see if those come
> through fine. =A0It could be timing between the FPGA and PCI such
> that the wrong data is being latched.
>
> Then try slightly less predictable data and see what gets through.
>
> -- glen


If it was timing, I expect the other way around also problems (which I
don't have). Also single memory read/write requests send from the PC
are working correctly.

On Aug 3, 3:26 am, Frank van Eijkelenburg
<fei.technolut...@gmail.com>
wrote:
> On Aug 2, 5:28 pm, glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
>
>
>
> > Frank van Eijkelenburg <fei.technolut...@gmail.com> wrote:
>
> > > I have a custom made PCIe board with a Virtex 5 FPGA on which I
> > > implemented a DMA unit which uses the PCIe endpoint block plus v1.14.
> > > I also implemented simple read/write operations from the PC to the
> > > board (the board responds with completion TLPs). The read/write
> > > operations are working, DMA is not working (transferring data from
> > > FPGA to PC).
>
> > (snip)
>
> > > When I start a DMA operation by writing a register in the FPGA, I can
> > > see in chipscope the correct physical addresses in the TLP header (of
> > > the memory write requests). However, I do not see the correct values
> > > in the allocated memory at the PC. What can I do to check where it is
> > > going wrong?
>
> > Not having tried to do DMA through PCI before, is data being
> > written, but the wrong data?
>
> That is what I do not know. Yes the correct data is send to the PC,
> but if I readout the memory the values are unchanged.
>
>
>
> > I would try writing all zeros or all ones and see if those come
> > through fine.  It could be timing between the FPGA and PCI such
> > that the wrong data is being latched.
>
> > Then try slightly less predictable data and see what gets through.
>
> > -- glen
>
> If it was timing, I expect the other way around also problems (which I
> don't have). Also single memory read/write requests send from the PC
> are working correctly.

Does CPU caches get snooped when DMA wrote into host memory?  If not,
then you may not see right data by normal read.

AFAIK, in x86 architectures, io writes will be
cache coherent, unless you're setting SNR (no snoop) in the write header by mistakeBut to me this sounds like a Windows problem. Typically garden variety PC's don't block
PCIE accesses, all of the address space is open to a peripheral.  Acid test : if you can
write to a given physical address from your FPGA, and read the value, you're golden in the
physical address domain. So maybe what you think you're reading in your code is not
actually getting to the correct physical address. try : write from FPGA card to a physical addressread back using FPGA card, see if good. write some known data from windows softwareread back using FPGA card.  

On Aug 3, 6:03=A0pm, alpha
<zhg....@gmail.com> wrote:
> On Aug 3, 3:26 am, Frank van Eijkelenburg <fei.technolut...@gmail.com>
> wrote:
>
>
>
> > On Aug 2, 5:28 pm, glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
>
> > > Frank van Eijkelenburg <fei.technolut...@gmail.com> wrote:
>
> > > > I have a custom made PCIe board with a Virtex 5 FPGA on which I
> > > > implemented aDMAunit which uses the PCIe endpoint block plus v1.14.
> > > > I also implemented simple read/write operations from the PC to the
> > > > board (the board responds with completion TLPs). The read/write
> > > > operations are working,DMAis not working (transferring data from
> > > > FPGA to PC).
>
> > > (snip)
>
> > > > When I start aDMAoperation by writing a register in the FPGA, I can
> > > > see in chipscope the correct physical addresses in the TLP header (=
of
> > > > the memory write requests). However, I do not see the correct value=
s
> > > > in the allocated memory at the PC. What can I do to check where it =
is
> > > > going wrong?
>
> > > Not having tried to doDMAthrough PCI before, is data being
> > > written, but the wrong data?
>
> > That is what I do not know. Yes the correct data is send to the PC,
> > but if I readout the memory the values are unchanged.
>
> > > I would try writing all zeros or all ones and see if those come
> > > through fine. =A0It could be timing between the FPGA and PCI such
> > > that the wrong data is being latched.
>
> > > Then try slightly less predictable data and see what gets through.
>
> > > -- glen
>
> > If it was timing, I expect the other way around also problems (which I
> > don't have). Also single memory read/write requests send from the PC
> > are working correctly.
>
> Does CPU caches get snooped whenDMAwrote into host memory? =A0If not,
> then you may not see right data by normal read.

Finally, I solved the problem. For those who want to learn from
mistakes of others, here comes the cause of the problem:

The packets which were transmitted to the pc were too large (more than
the maximum payload size of the receiver). In that case, the packets
are simply dropped (no errors). Of course I have to read the maximum
payload size from the device control register in the PCI Express
Capability structure.

best regards,

Frank