On Jul 2, 9:04=A0pm, Charles Gardiner <charles.gardi...@invalid.invalid> wrote:> Hi Frank, > > > > > I am not sure if we understand each other. > > Yes, it certainly sounds like that. > > > What do you mean by > > completing the request with IoCompleteRequest? There is no request > > from software point of view. > > I think this might clear up the reason why your data is missing. (See als=o below> about the type of DMA). I don't think the S/G list you are getting is des=cribing> your application buffer. This is best done by specifying DO_DIRECT_IO as =the DMA> method for your device. If you specify DO_BUFFERED_IO you will get an S/G=List> describing an intermediate buffer in kernel space and this probably never=gets> copied over to your application space buffer unless you terminate the req=uest.> I've never done the 'neither' method myself and from what I hear, it's a > complicated beast. > > > The FPGA will do a DMA write (data from > > FPGA to PC memory) at its own initiative. The allocated memory is used > > as long as the software is running. I do not allocate new memory for > > each new DMA transfer, but at startup a large piece of memory is > > allocated and the physical addresses are written to the FPGA by the > > driver software. > > Sounds like you are doing something like a circular buffer in memory whic=h stays> alive as long as your device does? > > > > > And yes, we use a DMA adapter in combination with the > > GetScatterGatherList method. We already used this in another project > > but that was PCI and DMA read (data from PC memory to FPGA). > > > By the way, where can I set the type of DMA? > > Typically, you set the DMA buffering method in your AddDevice function af=ter you> create your device object. Quoting from Oney's book, > > NTSTATUS AddDevice(..) { > =A0 =A0PDEVICE_OBJECT =A0 =A0fdo; > > =A0 =A0IoCreateDevice(....., &fdo); > =A0 =A0fdo->Flags |=3D DO_BUFFERED_IO; > =A0 =A0 =A0 =A0 =A0 =A0 <or> > =A0 =A0fdo->Flags |=3D DO_DIRECT_IO; > =A0 =A0 =A0 =A0 =A0 =A0 <or> > =A0 =A0fdo->Flags |=3D 0; =A0// i.e. neither Direct nor Buffered > > And, you can't change your mind afterwards. > > By the way if my assumption about the circular buffer in your design is c=orrect,> there is a slightly more standard solution (standard in the sense that ev=erybody> on the microsoft drivers newgroup seems to do it). It however requires tw=o threads> in your application. The first one requests a buffer (using new or malloc=) and> sets up an I/O Request ReadFile, WriteFile or DeviceIoControl referencing=this> buffer. This is performed as an asynchronous request. >There is absolutely no need for a second thread. Just issue "submit buffer" overlapped I/O request in a first thread and don't wait for completion.> The driver recognizes this request and pends it indefinitely, (typically =terminate> it when your driver is shutting down, otherwise windows will probably han=g). That's pretty bad advice. As a minimum, you should install the cancel handler and complete the request on cancellation. Besides, normally you should stop all DMA activities and complete the request in CLEANUP routine. Also it is always a good idea to have custom "stop DMA and withdraw circular buffer" IOCTL for a normal user-initiated termination.> Pending the request has the nice side effect that the buffer now becomes =locked> down permanently. > > Assuming you have set up your driver to use DO_DIRECT_IO DMA, you can get=the S/G> list describing the application space buffer as you are currently doing a=nd feed> this to your FPGA. > > Using the second thread in your application you can constantly read data =from the> locked down pages (you app. space buffer) that are being written by your =FPGA.> > Assuming the DO_DIRECT_IO solves your problem (I think there is a good ch=ance), I> would however still consider migrating to a KMDF based driver, particular=ily if> you are writing a new one. It's much easier to maintain and is probably m=ore> portable for future MS versions. > > > > > best regards, > > > Frank > > best regards, > CharlesYes, KMDF MUCH is easier than plain WDM.
DMA operation to 64-bits PC platform
Started by ●July 1, 2010
Reply by ●July 3, 20102010-07-03
Reply by ●July 3, 20102010-07-03
Michael S schrieb:> > Anyway, the discussion doesn't belong here. I recommend > http://groups.google.com/group/microsoft.public.development.device.drivers >No it doesn't. It belongs just where it is. If you followed from the beginning, the OP has two problems/questions 1) How to do 64-Bit addressing in PCIe. This is definitely FPGA related. 2) Why are his data not appearing as expected. It only became clear in the course of discussion that this may be due to his driver DMA type setting. (He hasn't confirmed yet) Nit-picking when everything is (maybe) solved is easy when you haven't really been contributing constructively.
Reply by ●July 4, 20102010-07-04
On Jul 4, 2:01=A0am, Charles Gardiner <charles.gardi...@invalid.invalid> wrote:> Michael S schrieb: > > > > > Anyway, the discussion doesn't belong here. I recommend > >http://groups.google.com/group/microsoft.public.development.device.dr... > > No it doesn't. It belongs just where it is. If you followed from the begi=nning,> the OP has two problems/questions >I insist that subthread started by Nico Coesel could be discussed in mpddd with better depth and precision.> 1) How to do 64-Bit addressing in PCIe. This is definitely FPGA related.You mean, 32-bit addressing.> > 2) Why are his data not appearing as expected. It only became clear in th=e course> of discussion that this may be due to his driver DMA type setting. (He ha=sn't> confirmed yet)Microsoft calls them "data buffer access methods". DMA type (or, by KMDF terminology, profile) is something else. I hope you realize how important it is to use the same terminology as the rest of the world. http://msdn.microsoft.com/en-us/library/ff554436%28v=3DVS.85%29.aspx> > Nit-picking when everything is (maybe) solved is easy when you haven't re=ally been> =A0contributing constructively.If you followed from the beginning, you would realize that I constructively contributed the answer to the first question 4 minutes ahead of you ;) Also, IMHO, pointer to the most appropriate newsgroup is very constructive.
Reply by ●July 4, 20102010-07-04
Michael S schrieb:> I insist that subthread started by Nico Coesel could be discussed in > mpddd with better depth and precision. >You insist? Do I care?> > If you followed from the beginning, you would realize that I > constructively contributed the answer to the first question 4 minutes > ahead of you ;)4 mins.? WOW. Incredible. Impressive, Awesome..... What I see in this posting is a terse reply to the easy part of the problem with not a single thought spent on the more complex part, even though this is really the OPs show stopper. For the rest of your postings, I mainly see blowholed personal seals of approval assigned to what others have said (aka nit-picking). All coming in "posthumously" (still assuming that specifying DO_DIRECT_IO actually solves the problem) I really love attending meetings when guys like you are involved, especially in the last phase when those who havn't contributed much insist on getting a chance to lift their leg. The german comedian Karl Valentin had perhaps a more literary way of putting it: "I suppose everything has been said then, but not yet by everybody". Oh, don't take my personal aversion too much to heart. I'm sure somebody on this planet recognises your contribution, even if it's only Dilbert
Reply by ●July 4, 20102010-07-04
On Jul 4, 11:59=A0am, Charles Gardiner <charles.gardi...@invalid.invalid> wrote:> Michael S schrieb: > > > I insist that subthread started by Nico Coesel could be discussed =A0in > > mpddd with better depth and precision. > > You insist? Do I care? > > > If you followed from the beginning, you would realize that I > > constructively contributed the answer to the first question 4 minutes > > ahead of you ;) > > 4 mins.? WOW. Incredible. Impressive, Awesome..... > > What I see in this posting is a terse reply to the easy part of the probl=em with> not a single thought spent on the more complex part, even though this is =really> the OPs show stopper. > > For the rest of your postings, I mainly see blowholed personal seals of a=pproval> assigned to what others have said (aka nit-picking). All coming in "posth=umously"> (still assuming that specifying DO_DIRECT_IO actually solves the problem)And I am 90% sure that 3DW was his only problem. The following sentence in his original post suggests that he doesn't use BUFFERED_IO: "The driver locks the memory". For me it sounds like he is doing NEITHER I/O. So it's either 3DW or, less likely, he erroneously too early completes his ReadFile() request.> > I really love attending meetings when guys like you are involved, especia=lly in> the last phase when those who havn't contributed much insist on getting a=chance> to lift their leg. The german comedian Karl Valentin had perhaps a more l=iterary> way of putting it: "I suppose everything has been said then, but not yet =by> everybody". > > Oh, don't take my personal aversion too much to heart. I'm sure somebody =on this> planet recognises your contribution, even if it's only DilbertYou are wrong. I never really was obsessed with getting the credit. More like obsessed with precise formulations, although more so in real life than on the Web. Also I am one of those types that don't like to quickly jump to conclusion.
Reply by ●July 6, 20102010-07-06
On Jul 2, 9:04=A0pm, Charles Gardiner <charles.gardi...@invalid.invalid> wrote:> Hi Frank, > > > > > I am not sure if we understand each other. > > Yes, it certainly sounds like that. > > > What do you mean by > > completing the request with IoCompleteRequest? There is no request > > from software point of view. > > I think this might clear up the reason why your data is missing. (See als=o below> about the type of DMA). I don't think the S/G list you are getting is des=cribing> your application buffer. This is best done by specifying DO_DIRECT_IO as =the DMA> method for your device. If you specify DO_BUFFERED_IO you will get an S/G=List> describing an intermediate buffer in kernel space and this probably never=gets> copied over to your application space buffer unless you terminate the req=uest.> I've never done the 'neither' method myself and from what I hear, it's a > complicated beast. > > > The FPGA will do a DMA write (data from > > FPGA to PC memory) at its own initiative. The allocated memory is used > > as long as the software is running. I do not allocate new memory for > > each new DMA transfer, but at startup a large piece of memory is > > allocated and the physical addresses are written to the FPGA by the > > driver software. > > Sounds like you are doing something like a circular buffer in memory whic=h stays> alive as long as your device does? > > > > > And yes, we use a DMA adapter in combination with the > > GetScatterGatherList method. We already used this in another project > > but that was PCI and DMA read (data from PC memory to FPGA). > > > By the way, where can I set the type of DMA? > > Typically, you set the DMA buffering method in your AddDevice function af=ter you> create your device object. Quoting from Oney's book, > > NTSTATUS AddDevice(..) { > =A0 =A0PDEVICE_OBJECT =A0 =A0fdo; > > =A0 =A0IoCreateDevice(....., &fdo); > =A0 =A0fdo->Flags |=3D DO_BUFFERED_IO; > =A0 =A0 =A0 =A0 =A0 =A0 <or> > =A0 =A0fdo->Flags |=3D DO_DIRECT_IO; > =A0 =A0 =A0 =A0 =A0 =A0 <or> > =A0 =A0fdo->Flags |=3D 0; =A0// i.e. neither Direct nor Buffered > > And, you can't change your mind afterwards. > > By the way if my assumption about the circular buffer in your design is c=orrect,> there is a slightly more standard solution (standard in the sense that ev=erybody> on the microsoft drivers newgroup seems to do it). It however requires tw=o threads> in your application. The first one requests a buffer (using new or malloc=) and> sets up an I/O Request ReadFile, WriteFile or DeviceIoControl referencing=this> buffer. This is performed as an asynchronous request. > > The driver recognises this request and pends it indefinitely, (typically =terminate> it when your driver is shutting down, otherwise windows will probably han=g).> Pending the request has the nice side effect that the buffer now becomes =locked> down permanently. > > Assuming you have set up your driver to use DO_DIRECT_IO DMA, you can get=the S/G> list describing the application space buffer as you are currently doing a=nd feed> this to your FPGA. > > Using the second thread in your application you can constantly read data =from the> locked down pages (you app. space buffer) that are being written by your =FPGA.> > Assuming the DO_DIRECT_IO solves your problem (I think there is a good ch=ance), I> would however still consider migrating to a KMDF based driver, particular=ily if> you are writing a new one. It's much easier to maintain and is probably m=ore> portable for future MS versions. > > > > > best regards, > > > Frank > > best regards, > CharlesHi Charles, We tried your suggestion (we were using BUFFERED_IO). Unfortunately it was not the (final) solution. Perhaps there are more causes for the problem. Anyway, thanks for your suggestion. We are almost out of ideas of what we can test. Do you have other ideas or tests we can do to find the cause? I hope to fix the problem before my vacation (only one day left :) best regards, Frank
Reply by ●July 6, 20102010-07-06
On Jul 2, 9:04=A0pm, Charles Gardiner <charles.gardi...@invalid.invalid> wrote:> Hi Frank, > > > > > I am not sure if we understand each other. > > Yes, it certainly sounds like that. > > > What do you mean by > > completing the request with IoCompleteRequest? There is no request > > from software point of view. > > I think this might clear up the reason why your data is missing. (See als=o below> about the type of DMA). I don't think the S/G list you are getting is des=cribing> your application buffer. This is best done by specifying DO_DIRECT_IO as =the DMA> method for your device. If you specify DO_BUFFERED_IO you will get an S/G=List> describing an intermediate buffer in kernel space and this probably never=gets> copied over to your application space buffer unless you terminate the req=uest.> I've never done the 'neither' method myself and from what I hear, it's a > complicated beast. > > > The FPGA will do a DMA write (data from > > FPGA to PC memory) at its own initiative. The allocated memory is used > > as long as the software is running. I do not allocate new memory for > > each new DMA transfer, but at startup a large piece of memory is > > allocated and the physical addresses are written to the FPGA by the > > driver software. > > Sounds like you are doing something like a circular buffer in memory whic=h stays> alive as long as your device does? > > > > > And yes, we use a DMA adapter in combination with the > > GetScatterGatherList method. We already used this in another project > > but that was PCI and DMA read (data from PC memory to FPGA). > > > By the way, where can I set the type of DMA? > > Typically, you set the DMA buffering method in your AddDevice function af=ter you> create your device object. Quoting from Oney's book, > > NTSTATUS AddDevice(..) { > =A0 =A0PDEVICE_OBJECT =A0 =A0fdo; > > =A0 =A0IoCreateDevice(....., &fdo); > =A0 =A0fdo->Flags |=3D DO_BUFFERED_IO; > =A0 =A0 =A0 =A0 =A0 =A0 <or> > =A0 =A0fdo->Flags |=3D DO_DIRECT_IO; > =A0 =A0 =A0 =A0 =A0 =A0 <or> > =A0 =A0fdo->Flags |=3D 0; =A0// i.e. neither Direct nor Buffered > > And, you can't change your mind afterwards. > > By the way if my assumption about the circular buffer in your design is c=orrect,> there is a slightly more standard solution (standard in the sense that ev=erybody> on the microsoft drivers newgroup seems to do it). It however requires tw=o threads> in your application. The first one requests a buffer (using new or malloc=) and> sets up an I/O Request ReadFile, WriteFile or DeviceIoControl referencing=this> buffer. This is performed as an asynchronous request. > > The driver recognises this request and pends it indefinitely, (typically =terminate> it when your driver is shutting down, otherwise windows will probably han=g).> Pending the request has the nice side effect that the buffer now becomes =locked> down permanently. > > Assuming you have set up your driver to use DO_DIRECT_IO DMA, you can get=the S/G> list describing the application space buffer as you are currently doing a=nd feed> this to your FPGA. > > Using the second thread in your application you can constantly read data =from the> locked down pages (you app. space buffer) that are being written by your =FPGA.> > Assuming the DO_DIRECT_IO solves your problem (I think there is a good ch=ance), I> would however still consider migrating to a KMDF based driver, particular=ily if> you are writing a new one. It's much easier to maintain and is probably m=ore> portable for future MS versions. > > > > > best regards, > > > Frank > > best regards, > CharlesHi Charles, We tried your suggestion (we were using BUFFERED_IO). Unfortunately it was not the (final) solution. Perhaps there are more causes for the problem. Anyway, thanks for your suggestion. We are almost out of ideas of what we can test. Do you have other ideas or tests we can do to find the cause? I hope to fix the problem before my vacation (only one day left :) best regards, Frank
Reply by ●July 6, 20102010-07-06
Frank van Eijkelenburg schrieb:> We tried your suggestion (we were using BUFFERED_IO). Unfortunately it > was not the (final) solution.Was there any noticeable change in the behaviour at all? Is it still valid that your FPGA can _read_ data from the buffer when your application writes it there? With DO_DIRECT_IO specified, it's not clear to me off-hand why you are not seeing the memory locations in both directions now. Perhaps there are more causes for the> problem. Anyway, thanks for your suggestion. We are almost out of > ideas of what we can test. Do you have other ideas or tests we can do > to find the cause? I hope to fix the problem before my vacation (only > one day left :)Oops, thats tight. I'm just on the way to a customers so I don't have my usual references at hand. Have you tried the flush (zero length read from FPGA) after a write to memory. Although, to be honest I don't think that's the solution (just a straw to grab for in case your system has some caching behaviour I haven't seen before). My last (KMDF based) design was similar to yours. The FPGA was streaming to memory and the SW application reading from the buffer shared between application memory and kernel memory. I never had any data loss, even without the zero length read. If you can send me as much relevant info as possible, I'll have another look this evening. Regards, Charles
Reply by ●July 6, 20102010-07-06
On Jul 6, 11:00=A0am, Frank van Eijkelenburg <fei.technolut...@gmail.com> wrote:> On Jul 2, 9:04=A0pm, Charles Gardiner <charles.gardi...@invalid.invalid> > wrote: > > > > > Hi Frank, > > > > I am not sure if we understand each other. > > > Yes, it certainly sounds like that. > > > > What do you mean by > > > completing the request with IoCompleteRequest? There is no request > > > from software point of view. > > > I think this might clear up the reason why your data is missing. (See a=lso below> > about the type of DMA). I don't think the S/G list you are getting is d=escribing> > your application buffer. This is best done by specifying DO_DIRECT_IO a=s the DMA> > method for your device. If you specify DO_BUFFERED_IO you will get an S=/G List> > describing an intermediate buffer in kernel space and this probably nev=er gets> > copied over to your application space buffer unless you terminate the r=equest.> > I've never done the 'neither' method myself and from what I hear, it's =a> > complicated beast. > > > > The FPGA will do a DMA write (data from > > > FPGA to PC memory) at its own initiative. The allocated memory is use=d> > > as long as the software is running. I do not allocate new memory for > > > each new DMA transfer, but at startup a large piece of memory is > > > allocated and the physical addresses are written to the FPGA by the > > > driver software. > > > Sounds like you are doing something like a circular buffer in memory wh=ich stays> > alive as long as your device does? > > > > And yes, we use a DMA adapter in combination with the > > > GetScatterGatherList method. We already used this in another project > > > but that was PCI and DMA read (data from PC memory to FPGA). > > > > By the way, where can I set the type of DMA? > > > Typically, you set the DMA buffering method in your AddDevice function =after you> > create your device object. Quoting from Oney's book, > > > NTSTATUS AddDevice(..) { > > =A0 =A0PDEVICE_OBJECT =A0 =A0fdo; > > > =A0 =A0IoCreateDevice(....., &fdo); > > =A0 =A0fdo->Flags |=3D DO_BUFFERED_IO; > > =A0 =A0 =A0 =A0 =A0 =A0 <or> > > =A0 =A0fdo->Flags |=3D DO_DIRECT_IO; > > =A0 =A0 =A0 =A0 =A0 =A0 <or> > > =A0 =A0fdo->Flags |=3D 0; =A0// i.e. neither Direct nor Buffered > > > And, you can't change your mind afterwards. > > > By the way if my assumption about the circular buffer in your design is=correct,> > there is a slightly more standard solution (standard in the sense that =everybody> > on the microsoft drivers newgroup seems to do it). It however requires =two threads> > in your application. The first one requests a buffer (using new or mall=oc) and> > sets up an I/O Request ReadFile, WriteFile or DeviceIoControl referenci=ng this> > buffer. This is performed as an asynchronous request. > > > The driver recognises this request and pends it indefinitely, (typicall=y terminate> > it when your driver is shutting down, otherwise windows will probably h=ang).> > Pending the request has the nice side effect that the buffer now become=s locked> > down permanently. > > > Assuming you have set up your driver to use DO_DIRECT_IO DMA, you can g=et the S/G> > list describing the application space buffer as you are currently doing=and feed> > this to your FPGA. > > > Using the second thread in your application you can constantly read dat=a from the> > locked down pages (you app. space buffer) that are being written by you=r FPGA.> > > Assuming the DO_DIRECT_IO solves your problem (I think there is a good =chance), I> > would however still consider migrating to a KMDF based driver, particul=arily if> > you are writing a new one. It's much easier to maintain and is probably=more> > portable for future MS versions. > > > > best regards, > > > > Frank > > > best regards, > > Charles > > Hi Charles, > > We tried your suggestion (we were using BUFFERED_IO).If you were using BUFFERED_IO why was your driver locking the pages? In case of BUFFERED_IO the pages come from kernel non-paged pool and don't have to be specifically locked. The only case where the driver is responsible for locking/unlocking pages is NEITHER I/O.>Unfortunately it > was not the (final) solution. Perhaps there are more causes for the > problem. Anyway, thanks for your suggestion. We are almost out of > ideas of what we can test. Do you have other ideas or tests we can do > to find the cause? I hope to fix the problem before my vacation (only > one day left :) > > best regards, > > FrankAnother typical mistake is driver forgets to call IoMarkIrpPending(). KMDF does it automatically, but in plain WDM it's responsibility of your driver. However forgotten IoMarkIrpPending() normally shows different symptoms.
Reply by ●July 6, 20102010-07-06
On Jul 6, 11:00=A0am, Frank van Eijkelenburg <fei.technolut...@gmail.com> wrote:> I hope to fix the problem before my vacation (only one day left :) >Something, I most certainly DO NOT RECOMMEND for final solution, but it could help to go to vacation in better mood. Scrap all the schoolbook nice&complex Windows DMA API stuff. Instead, take your Irp->MdlAddress, do MmGetMdlPfnArray() and access physical addresses directly. It's wrong, it's immoral but on simple x86/x64 PC or on small dual-processor server it always work. Just don't forget to bring back the official DMA API when you are back from vocation and have more time than a few hours.
