Anyone doing any test driven design for FPGA work? I've gone over to doing it almost universally for C++ development, because It Just Works -- you lengthen the time to integration a bit, but vastly shorten the actual integration time. I did a web search and didn't find it mentioned -- the traditional "make a test bench" is part way there, but as presented in my textbook* doesn't impose a comprehensive suite of tests on each module. So is no one doing it, or does it have another name, or an equivalent design process with a different name, or what? * "The Verilog Hardware Description Language", Thomas & Moorby, Kluwer, 1998. -- www.wescottdesign.com
Test Driven Design?
Started by ●May 16, 2017
Reply by ●May 16, 20172017-05-16
Tim Wescott <tim@seemywebsite.really> wrote:> Anyone doing any test driven design for FPGA work? > > I've gone over to doing it almost universally for C++ development, > because It Just Works -- you lengthen the time to integration a bit, but > vastly shorten the actual integration time. > > I did a web search and didn't find it mentioned -- the traditional "make > a test bench" is part way there, but as presented in my textbook* doesn't > impose a comprehensive suite of tests on each module. > > So is no one doing it, or does it have another name, or an equivalent > design process with a different name, or what?We do it. We have an equivalence checker that fuzzes random inputs to both the system and an executable 'golden model' of the system, looking for discrepancies. If found, it'll then reduce down to a minimal example. In particular this is very handy because running the test cases is then synthesisable: so we can run the tests on FPGA rather than on a simulator. Our paper has more details and the code is open source: https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/201509-memocode2015-bluecheck.pdf Theo
Reply by ●May 17, 20172017-05-17
I do a sloppy version of it. Sometime I allow myself not to do tests for some simple and small modules which will be tested on a higher hierarchy level anyway. (Because I also make tests for modules of different hierarchy levels) Tests are randomized and they are launched with different seeds every time. If there is a problem, I also can launch the faulty test with a specific seed to repeat the problem.
Reply by ●May 17, 20172017-05-17
On 5/16/2017 4:21 PM, Tim Wescott wrote:> Anyone doing any test driven design for FPGA work? > > I've gone over to doing it almost universally for C++ development, > because It Just Works -- you lengthen the time to integration a bit, but > vastly shorten the actual integration time. > > I did a web search and didn't find it mentioned -- the traditional "make > a test bench" is part way there, but as presented in my textbook* doesn't > impose a comprehensive suite of tests on each module. > > So is no one doing it, or does it have another name, or an equivalent > design process with a different name, or what? > > * "The Verilog Hardware Description Language", Thomas & Moorby, Kluwer, > 1998.I'm not clear on all of the details of what defines "test driven design", but I believe I've been using that all along. I've thought of this as bottom up development where the lower level code is written first *and thoroughly tested* before writing the next level of code. How does "test driven design" differ from this significantly? -- Rick C
Reply by ●May 17, 20172017-05-17
On 05/16/2017 01:21 PM, Tim Wescott wrote:> Anyone doing any test driven design for FPGA work? > > I've gone over to doing it almost universally for C++ development, > because It Just Works -- you lengthen the time to integration a bit, but > vastly shorten the actual integration time. > > I did a web search and didn't find it mentioned -- the traditional "make > a test bench" is part way there, but as presented in my textbook* doesn't > impose a comprehensive suite of tests on each module. > > So is no one doing it, or does it have another name, or an equivalent > design process with a different name, or what? > > * "The Verilog Hardware Description Language", Thomas & Moorby, Kluwer, > 1998. >We don't do classical "first you write a testbench and prove it fails, then you write the code that makes it pass" TDD but we do a whole lot of unit testing before we try to integrate submodules into the larger design. I get a ton of mileage from OSVVM (http://osvvm.org/) for constrained random verification. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix.
Reply by ●May 17, 20172017-05-17
On Wed, 17 May 2017 11:47:10 -0400, rickman wrote:> On 5/16/2017 4:21 PM, Tim Wescott wrote: >> Anyone doing any test driven design for FPGA work? >> >> I've gone over to doing it almost universally for C++ development, >> because It Just Works -- you lengthen the time to integration a bit, >> but vastly shorten the actual integration time. >> >> I did a web search and didn't find it mentioned -- the traditional >> "make a test bench" is part way there, but as presented in my textbook* >> doesn't impose a comprehensive suite of tests on each module. >> >> So is no one doing it, or does it have another name, or an equivalent >> design process with a different name, or what? >> >> * "The Verilog Hardware Description Language", Thomas & Moorby, Kluwer, >> 1998. > > I'm not clear on all of the details of what defines "test driven > design", but I believe I've been using that all along. I've thought of > this as bottom up development where the lower level code is written > first *and thoroughly tested* before writing the next level of code. > > How does "test driven design" differ from this significantly?The big difference in the software world is that the tests are automated and never retired. There are generally test suites to make the mechanics of testing easier. Ideally, whenever you do a build you run the entire unit-test suite fresh. This means that when you tweak some low-level function, it still gets tested. The other big difference, that's hard for one guy to do, is that if you're going Full Agile you have one guy writing tests and another guy writing "real" code. Ideally they're equally good, and they switch off. The idea is basically that more brains on the problem is better. If you look at the full description of TDD it looks like it'd be hard, slow, and clunky, because the recommendation is to do things at a very fine-grained level. However, I've done it, and the process of adding features to a function as you add tests to the bench goes very quickly. The actual development of the bottom layer is a bit slower, but when you go to put the pieces together they just fall into place. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com
Reply by ●May 17, 20172017-05-17
On 17/05/17 16:47, rickman wrote:> On 5/16/2017 4:21 PM, Tim Wescott wrote: >> Anyone doing any test driven design for FPGA work? >> >> I've gone over to doing it almost universally for C++ development, >> because It Just Works -- you lengthen the time to integration a bit, but >> vastly shorten the actual integration time. >> >> I did a web search and didn't find it mentioned -- the traditional "make >> a test bench" is part way there, but as presented in my textbook* doesn't >> impose a comprehensive suite of tests on each module. >> >> So is no one doing it, or does it have another name, or an equivalent >> design process with a different name, or what? >> >> * "The Verilog Hardware Description Language", Thomas & Moorby, Kluwer, >> 1998. > > I'm not clear on all of the details of what defines "test driven design", but I > believe I've been using that all along. I've thought of this as bottom up > development where the lower level code is written first *and thoroughly tested* > before writing the next level of code. > > How does "test driven design" differ from this significantly?In many software environments TDD - as it is taught - more naturally fits top-down design. That's not necessary, but that's the typical mentality. TDD can and should be used for "bottom-up" "integration tests". The key point, all to often missed, is to /think/ about the benefits and disadvantages of each tool in your armoury, and use only the most appropriate combination for your problem at hand.
Reply by ●May 17, 20172017-05-17
On 5/17/2017 1:17 PM, Tim Wescott wrote:> On Wed, 17 May 2017 11:47:10 -0400, rickman wrote: > >> On 5/16/2017 4:21 PM, Tim Wescott wrote: >>> Anyone doing any test driven design for FPGA work? >>> >>> I've gone over to doing it almost universally for C++ development, >>> because It Just Works -- you lengthen the time to integration a bit, >>> but vastly shorten the actual integration time. >>> >>> I did a web search and didn't find it mentioned -- the traditional >>> "make a test bench" is part way there, but as presented in my textbook* >>> doesn't impose a comprehensive suite of tests on each module. >>> >>> So is no one doing it, or does it have another name, or an equivalent >>> design process with a different name, or what? >>> >>> * "The Verilog Hardware Description Language", Thomas & Moorby, Kluwer, >>> 1998. >> >> I'm not clear on all of the details of what defines "test driven >> design", but I believe I've been using that all along. I've thought of >> this as bottom up development where the lower level code is written >> first *and thoroughly tested* before writing the next level of code. >> >> How does "test driven design" differ from this significantly? > > The big difference in the software world is that the tests are automated > and never retired. There are generally test suites to make the mechanics > of testing easier. Ideally, whenever you do a build you run the entire > unit-test suite fresh. This means that when you tweak some low-level > function, it still gets tested. > > The other big difference, that's hard for one guy to do, is that if > you're going Full Agile you have one guy writing tests and another guy > writing "real" code. Ideally they're equally good, and they switch off. > The idea is basically that more brains on the problem is better. > > If you look at the full description of TDD it looks like it'd be hard, > slow, and clunky, because the recommendation is to do things at a very > fine-grained level. However, I've done it, and the process of adding > features to a function as you add tests to the bench goes very quickly. > The actual development of the bottom layer is a bit slower, but when you > go to put the pieces together they just fall into place.I guess I'm still not picturing it. I think the part I don't get is "adding features to a function". To me the features would *be* functions that are written, tested and then added to next higher level code. So I assume what you wrote applies to that next higher level. I program in two languages, Forth and VHDL. In Forth functions (called "words") are written at *very* low levels, often a word is a single line of code and nearly all the time no more than five. Being very small a word is much easier to write although the organization can be tough to settle on. In VHDL I typically don't decompose the code into such fine grains. It is easy to write the code for the pieces, registers and logic. The hard part is how they interconnect/interrelate. Fine decomposition tends to obscure that rather than enhancing it. So I write large blocks of code to be tested. I guess in those cases features would be "added" rather than new modules being written for the new functionality. I still write test benches for each module in VHDL. Because there is a lot more work in writing a using a VHDL test bench than a Forth test word this also encourages larger (and fewer) modules. Needless to say, I don't find much synergy between the two languages. -- Rick C
Reply by ●May 17, 20172017-05-17
On Wed, 17 May 2017 13:39:55 -0400, rickman wrote:> On 5/17/2017 1:17 PM, Tim Wescott wrote: >> On Wed, 17 May 2017 11:47:10 -0400, rickman wrote: >> >>> On 5/16/2017 4:21 PM, Tim Wescott wrote: >>>> Anyone doing any test driven design for FPGA work? >>>> >>>> I've gone over to doing it almost universally for C++ development, >>>> because It Just Works -- you lengthen the time to integration a bit, >>>> but vastly shorten the actual integration time. >>>> >>>> I did a web search and didn't find it mentioned -- the traditional >>>> "make a test bench" is part way there, but as presented in my >>>> textbook* >>>> doesn't impose a comprehensive suite of tests on each module. >>>> >>>> So is no one doing it, or does it have another name, or an equivalent >>>> design process with a different name, or what? >>>> >>>> * "The Verilog Hardware Description Language", Thomas & Moorby, >>>> Kluwer, >>>> 1998. >>> >>> I'm not clear on all of the details of what defines "test driven >>> design", but I believe I've been using that all along. I've thought >>> of this as bottom up development where the lower level code is written >>> first *and thoroughly tested* before writing the next level of code. >>> >>> How does "test driven design" differ from this significantly? >> >> The big difference in the software world is that the tests are >> automated and never retired. There are generally test suites to make >> the mechanics of testing easier. Ideally, whenever you do a build you >> run the entire unit-test suite fresh. This means that when you tweak >> some low-level function, it still gets tested. >> >> The other big difference, that's hard for one guy to do, is that if >> you're going Full Agile you have one guy writing tests and another guy >> writing "real" code. Ideally they're equally good, and they switch >> off. The idea is basically that more brains on the problem is better. >> >> If you look at the full description of TDD it looks like it'd be hard, >> slow, and clunky, because the recommendation is to do things at a very >> fine-grained level. However, I've done it, and the process of adding >> features to a function as you add tests to the bench goes very quickly. >> The actual development of the bottom layer is a bit slower, but when >> you go to put the pieces together they just fall into place. > > I guess I'm still not picturing it. I think the part I don't get is > "adding features to a function". To me the features would *be* > functions that are written, tested and then added to next higher level > code. So I assume what you wrote applies to that next higher level. > > I program in two languages, Forth and VHDL. In Forth functions (called > "words") are written at *very* low levels, often a word is a single line > of code and nearly all the time no more than five. Being very small a > word is much easier to write although the organization can be tough to > settle on. > > In VHDL I typically don't decompose the code into such fine grains. It > is easy to write the code for the pieces, registers and logic. The hard > part is how they interconnect/interrelate. Fine decomposition tends to > obscure that rather than enhancing it. So I write large blocks of code > to be tested. I guess in those cases features would be "added" rather > than new modules being written for the new functionality. > > I still write test benches for each module in VHDL. Because there is a > lot more work in writing a using a VHDL test bench than a Forth test > word this also encourages larger (and fewer) modules. > > Needless to say, I don't find much synergy between the two languages.Part of what I'm looking for is a reading on whether it makes sense in the context of an HDL, and if so, how it makes sense in the context of an HDL (I'm using Verilog, because I'm slightly more familiar with it, but that's incidental). In Really Pure TDD for Java, C, or C++, you start by writing a test in the absence of a function, just to see the compiler error out. Then you write a function that does nothing. Then (for instance), you write a test who's expected return value is "42", and an accompanying function that just returns 42. Then you elaborate from there. It sounds really dippy (I was about as skeptical as can be when it was presented to me), but in a world where compilation is fast, there's very little speed penalty. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com
Reply by ●May 17, 20172017-05-17
On 05/16/2017 01:21 PM, Tim Wescott wrote:> Anyone doing any test driven design for FPGA work? > > I've gone over to doing it almost universally for C++ development, > because It Just Works -- you lengthen the time to integration a bit, but > vastly shorten the actual integration time. > > I did a web search and didn't find it mentioned -- the traditional "make > a test bench" is part way there, but as presented in my textbook* doesn't > impose a comprehensive suite of tests on each module. > > So is no one doing it, or does it have another name, or an equivalent > design process with a different name, or what? > > * "The Verilog Hardware Description Language", Thomas & Moorby, Kluwer, > 1998. >Can you elaborate on "Test Driven Design" please? Is this some specialized design methodology, or a standard design methodology with extensive module testing, or something else completely? thanks, BobH
