C++ Unit testing framework up and running
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Be nice to others! Respect the FreeCAD code of conduct!
Re: C++ Unit testing framework up and running
It seems to be assumed that for Windows packages must be in Libpack.
What about vcpkg? gtest (and libfmt) are supported.
What about vcpkg? gtest (and libfmt) are supported.
Re: C++ Unit testing framework up and running
Why should this be restricted to developers only? From my expierence with other packages I'm maintaining, a majority of testing failure reports comes from package maintainers. There are several distributions which seem to have it in their building and distribution workflow. Reasons? QA, verifying the build is working like it should. AFAIK there are even requirements for having and running unit tests before packages are distributed, for example for Python packages on PyPI. On Gentoo we practically mirror this requirement. Being a source based distribution thus requires the user to have the ability to run those tests if they like.wmayer wrote: ↑Mon Feb 06, 2023 7:02 pmWhy should package maintainers be affected? The unit testing framework IMO is a pure development thing that is only executed on a developer's machine and shouldn't be part of the application package. From a technical point of view the unit tests create stand-alone applications where all the Google test code is linked statically.We just need to teach the package maintainers to add a dependency on gtest into their build recipes, if testing is required and developers to pull sub-modules as well.
Re: C++ Unit testing framework up and running
OK, but keep in mind that these tests are not as easily accessible as the current integration tests with the Testing workbench. These are separate executables and a user must run them from the command line in order to see any results. Another thing is that the packages will get quite bloated if each class has its own test executable.Why should this be restricted to developers only?...
Re: C++ Unit testing framework up and running
With absolutely the greatest respect (and you know I mean that), I don't understand your post.wmayer wrote: ↑Tue Feb 07, 2023 9:58 am OK, but keep in mind that these tests are not as easily accessible as the current integration tests with the Testing workbench. These are separate executables and a user must run them from the command line in order to see any results. Another thing is that the packages will get quite bloated if each class has its own test executable.
The whole point of unit tests is they are separate from the code under test. You don't have to run FreeCAD to test a FreeCAD class. Its a whole different ball-game. To run unit tests (once set up) is trivial, esp from within an IDE.
Are unit tests more, or less, accessible? One requires a somewhat functional FreeCAD, the other does not. Does one require a framework? Yes.
Unit testing frameworks like gtest are not limited to running unit tests. They can equally run integration tests etc.
Test sources are tiny. I'm currently experimentally refactoring Base::Unit and have 118 tests in one file, 562 lines.
There's no way I could be doing this refactor without unit tests. Far too risky. But this way is actually fun! These unit tests run very fast: still shows 0ms. I havn't buit the project for many days (no need).
In-built FreeCAD integration tests also take up space (from what I've seen, probably more), and from experience are slow, and contribute to app bloat, even if only in a WB. Keeping tests out of the app is a Good Thing.
If. For every class to have its own tests is a worthwhile, but extroadinarily difficult target!if each class has its own test
Re: C++ Unit testing framework up and running
I think you misunderstood my points.
That's exactly what I tried to explain that the (real) unit tests are separate applications and not bundled with the FreeCAD application. Usually you run them from the IDE directly but it's also possible to run them from the command line. Because of this the tests are not easily accessible by users in case they will be added to the package.The whole point of unit tests is they are separate from the code under test. You don't have to run FreeCAD to test a FreeCAD class. Its a whole different ball-game. To run unit tests (once set up) is trivial, esp from within an IDE.
Question: a correctly written unit test that succeeds on the developer's system is it allowed to fail on any other system?
EDIT:
My concern was that if for every class an own test application is created and they all statically link the Google test code that requires a lot of extra space. In the meantime I have seen that with your PR this is not the case any more and that there is currently a single test application for all Google-based tests.In-built FreeCAD integration tests also take up space (from what I've seen, probably more), and from experience are slow, and contribute to app bloat, even if only in a WB. Keeping tests out of the app is a Good Thing.
Re: C++ Unit testing framework up and running
I was just thinking about this problem this morning, because sometimes it might be useful to be able to run these more easily as a user (in case you compiled with a different compiler, and there is some kind of compiler-specific bug). We could easily write a simple unit test runner that loads those executables and runs them from the Test WB.
- adrianinsaval
- Veteran
- Posts: 5553
- Joined: Thu Apr 05, 2018 5:15 pm
Re: C++ Unit testing framework up and running
building these during packaging doesn't mean that you also need to ship this to the user, the case of gento is special because the user is also the one building the package, but even in other packaging environments it seems like a good idea to build and run the test before deploying a package, even if the package will not include the test binaries, we're likely going to use different dependencies versions and compilers than the devs so we need to know if something breaks due to this.wmayer wrote: ↑Tue Feb 07, 2023 2:01 pm That's exactly what I tried to explain that the (real) unit tests are separate applications and not bundled with the FreeCAD application. Usually you run them from the IDE directly but it's also possible to run them from the command line. Because of this the tests are not easily accessible by users in case they will be added to the package.
Re: C++ Unit testing framework up and running
Truly well-mocked unit tests will be independent of any library dependencies. It's going to take some refactoring to get our code under test in this way, but if we're really, really doing it right ("by the book" as it were), the only thing that is variable is the compiler.
Re: C++ Unit testing framework up and running
Maybe after you've actually tried using unit tests you'll stop looking for things to worry about?
Re: C++ Unit testing framework up and running
An example of aggressive refactoring and tests might give more of an idea
https://github.com/berniev/FreeCAD/tree/Inventor3D
https://github.com/berniev/FreeCAD/tree/Inventor3D