Reply by Lars Asplund November 13, 20172017-11-13
After reading this thread I also feel that I need to "defend" the simplicity of VUnit :-) If you look at the VUnit setup I made for the IEEE standard libraries I can understand if it looks complicated but that is not your average project and not the place to get started. At better starting point is https://www.youtube.com/channel/UCCPVCaeWkz6C95aRUTbIwdg. There you will find a clip showing how to install VUnit in a minute, how you can create a compile script for non-trivial project in a minute, and how you can have your project fully automated by adding  5 lines of code to every testbench. http://vunit.github.io/user_guide.html is another good source of information. So what makes the IEEE libraries different? Well...
- The simulator already knows about these libraries and gets confused when you try to compile them yourself. 
- The code is not fully self-contained VHDL but relies on the simulator "magically" implementing some functionality
- Some subprograms are expected to assert warning/error/failure on certain inputs and that is not something that you can verify automatically within VHDL since the simulator stops. That can be handled with Python scripting or you can use VUnit's assertion library which supports mocking of asserts.
- I setup a continuous integration server to run all tests whenever people want to summit code to he Git repository. You don't have to do that in order to run VUnit or any other unit testing framework.
- IEEE already had 30 or so testbenches but these testbenches also contain many test cases. VUnit allow you to isolate these test cases such that you can run and have their status reported individually. Rather than having 30 passing testbenches we now have 1700 passing test cases. You don't have to do this though, 5 lines of code is sufficient.


Reply by Lars Asplund November 13, 20172017-11-13
If we are talking about the typical software developer interpretation of TDD we're dealing with code/test cycles that can be as short as one minute. This is only possible with a fully automated approach to testing and that's why we have unit testing frameworks designed with this requirement in mind. I only know of two such frameworks for HDL, VUnit and SVUnit, so the popularity of these tools may serve as an indication of how widely TDD is used. I can't speak for SVUnit but from what I understand it's mostly used by ASIC verification engineers. However, I'm "the guy" who setup the test environment for VHDL-2017 and also one of the authors of VUnit so maybe I can shed some light in that area.

VUnit is today used by both FPGA and ASIC teams, for VHDL and Verilog, from US to Japan, when developing everything from high-volume products like automotive vision to niche military system, by simulator vendors to verify their tools, and for education. If you're like me you prefer facts over claims from a promoter so I would recommend that you google for VUnit rather than TDD. You should be able to find job ads where people look for VUnit skills, that the latest version of Sigasi Studio added VUnit support, independent training providers , university education, VHDL text books, etc. You can also compare VUnit with other popular open source projects. Search for VHDL on Github and you'll find most of them. Look at the number of stars, where the supporters work, the number of forks, the number of opened issues, the number of closed. Look at the project homepages and the activities on their forums.

However, just because people use VUnit doesn't mean that they are doing TDD. What I see is mainly two types of users. Some users have their own SW experiences with TDD/unit testing or have seen SW people doing it. They know what the want and end up doing just that. The other type of users doesn't have that experience or finds the concept somewhat absurd. They are more looking at the convenience of having everything fully automated but stick with their longer code/test cycle and they tend to do less refactoring. It doesn't mean that they don't get there eventually but it usually takes longer time. Unit testing and TDD is an acquired taste for anyone and requires practice. The good thing is that you can takes small steps and still get rewarded.
Reply by Ilya Kalistru August 23, 20172017-08-23
On Saturday, June 3, 2017 at 1:54:05 AM UTC+3, Ilya Kalistru wrote:
> It's a draft of an article. > https://docs.google.com/document/d/17LgQjxYdh8Dxy4NdFWWNYQ7up8MFNG4GQdPfv3s5LzI/edit?usp=sharing > It would be great if you left you comments right in the document or here, so that I could improve it. > > Thanks.
It's finally published on edn.com ! http://tinyurl.com/y9ekp7lf
Reply by Ilya Kalistru June 2, 20172017-06-02
It's a draft of an article. 
https://docs.google.com/document/d/17LgQjxYdh8Dxy4NdFWWNYQ7up8MFNG4GQdPfv3s5LzI/edit?usp=sharing
It would be great if you left you comments right in the document or here, so that I could improve it.

Thanks.
Reply by Ilya Kalistru May 23, 20172017-05-23
On Tuesday, May 23, 2017 at 9:26:26 PM UTC+3, Kevin Neilson wrote:
> > Another important advantage of non-project mode is that it is fully com=
patible with source control systems. When you don't have projects, you don'= t have piles of junk files of unknown purpose that changes every time you o= pen a project or run a simulation. In non-project mode you have only hdl so= urces and tcl scripts. Therefore all information is stored in source contro= l system but when you commit changes you commit only changes you have done,= not random changes of unknown project files.
> >=20 > > In this situation work with IP cores a bit trickier, but not much. Cons=
idering that you don't change ip's very often, it's not a problem at all.
> >=20 > > I see that very small number of hdl designers know and use this mode. M=
aybe I should write an article about it. Where it would be appropriate to p= ublish it?
>=20 > I would like to know more about this. When I used ISE I only used script=
s (shell scripts) and when I transitioned to Vivado I promised I would use = TCL scripts but I've never done that and I'm still just using the GUI. I n= eed to use the GUI to look at schematics of critical paths or to look at pl= acement, but I'd like to use scripts to do all the PAR and timing and every= thing else. I am writing an article about that. I'll post it here. I examine timing reports in logs of Vivado, but if I have bad timings somew= here, I often use GUI as well. It's just easier to understand what part of = code creates bad timing if you investigate it visually. I just open vivado, do open_checkpoint post_place.cpt Then I examine schematics of the paths and and their placement. Non-project= mode doesn't prevent using GUI when you need it. They work fine together.
Reply by Kevin Neilson May 23, 20172017-05-23
> Another important advantage of non-project mode is that it is fully compa=
tible with source control systems. When you don't have projects, you don't = have piles of junk files of unknown purpose that changes every time you ope= n a project or run a simulation. In non-project mode you have only hdl sour= ces and tcl scripts. Therefore all information is stored in source control = system but when you commit changes you commit only changes you have done, n= ot random changes of unknown project files.
>=20 > In this situation work with IP cores a bit trickier, but not much. Consid=
ering that you don't change ip's very often, it's not a problem at all.
>=20 > I see that very small number of hdl designers know and use this mode. May=
be I should write an article about it. Where it would be appropriate to pub= lish it? I would like to know more about this. When I used ISE I only used scripts = (shell scripts) and when I transitioned to Vivado I promised I would use TC= L scripts but I've never done that and I'm still just using the GUI. I nee= d to use the GUI to look at schematics of critical paths or to look at plac= ement, but I'd like to use scripts to do all the PAR and timing and everyth= ing else.
Reply by Ilya Kalistru May 20, 20172017-05-20
> All our builds run in continuous integration, which extracts logs and > timing/area numbers. The bitfiles then get downloaded and booted on FPGA, > then the test suite and benchmarks are run automatically to monitor > performance. Numbers then come back to continuous integration for graphing. > > Theo
Nice!
Reply by Ilya Kalistru May 20, 20172017-05-20
On Saturday, May 20, 2017 at 11:17:19 AM UTC+3, rickman wrote:
> On 5/20/2017 3:11 AM, Ilya Kalistru wrote: > > Yes. It is xilinx vivado. > > > > Another important advantage of non-project mode is that it is fully compatible with source control systems. When you don't have projects, you don't have piles of junk files of unknown purpose that changes every time you open a project or run a simulation. In non-project mode you have only hdl sources and tcl scripts. Therefore all information is stored in source control system but when you commit changes you commit only changes you have done, not random changes of unknown project files. > > > > In this situation work with IP cores a bit trickier, but not much. Considering that you don't change ip's very often, it's not a problem at all. > > > > I see that very small number of hdl designers know and use this mode. Maybe I should write an article about it. Where it would be appropriate to publish it? > > Doesn't the tool still generate all the intermediate files? The Lattice > tool (which uses Synplify for synthesis) creates a huge number of files > that only the tools look at. They aren't really project files, they are > various intermediate files. Living in the project main directory they > really get in the way. > > -- > > Rick C
It does. You can tell the tool where to generate these files and I do it in a special directory. It is easy to delete them, and you don't have to add them to your source control system. As all your important stuff is in src dir and all your junk is in sim_* dirs it is easy to manage them. That's what I have in my repository Project_name \sim_Test1NameDir *sim_test2NameDir *sim_test3NameDir |\ | *sim_report.log | *other_junk *synth_Module1Dir *synth_Module2Dir |\ | *Results | | \ | | *Reports | | *bitfiles | *Some_junk *src |\ | *DesignChunk1SrcDir | *DesignChunk2SrcDir *sim_test1.tcl *sim_test2.tcl *sim_test3.tcl *synth_Module1.tcl *synth_Module2.tcl
Reply by Theo Markettos May 20, 20172017-05-20
Ilya Kalistru <stebanoid@gmail.com> wrote:
> I've solved the problem with setting up a new project for each testbench > by not using any projects. Vivado has a non project mode when you write a > simple tcl script which tells vivado what sources to use and what to do > with them.
Something similar is possible with Intel FPGA (Altera) Quartus. You need one tcl file for settings, and building is a few commands which we run from a Makefile. All our builds run in continuous integration, which extracts logs and timing/area numbers. The bitfiles then get downloaded and booted on FPGA, then the test suite and benchmarks are run automatically to monitor performance. Numbers then come back to continuous integration for graphing. Theo
Reply by Tom Gardner May 20, 20172017-05-20
On 20/05/17 08:11, Ilya Kalistru wrote:
> Yes. It is xilinx vivado. > > Another important advantage of non-project mode is that it is fully compatible with source control systems. When you don't have projects, you don't have piles of junk files of unknown purpose that changes every time you open a project or run a simulation. In non-project mode you have only hdl sources and tcl scripts. Therefore all information is stored in source control system but when you commit changes you commit only changes you have done, not random changes of unknown project files. > > In this situation work with IP cores a bit trickier, but not much. Considering that you don't change ip's very often, it's not a problem at all. > > I see that very small number of hdl designers know and use this mode. Maybe I should write an article about it. Where it would be appropriate to publish it?
That would be useful; the project mode is initially appealing, but the splattered files and SCCS give me the jitters. Publish it everywhere! Any blog and bulletin board you can find, not limited to those dedicated to Xilinx.