C++ Unit testing framework up and running

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
berniev
Posts: 247
Joined: Wed Apr 13, 2022 10:45 pm
Location: Oz

Re: C++ Unit testing framework up and running

Post by berniev »

It seems to be assumed that for Windows packages must be in Libpack.

What about vcpkg? gtest (and libfmt) are supported.
User avatar
waebbl
Posts: 200
Joined: Thu Aug 16, 2018 3:12 pm

Re: C++ Unit testing framework up and running

Post by waebbl »

wmayer wrote: Mon Feb 06, 2023 7:02 pm
We just need to teach the package maintainers to add a dependency on gtest into their build recipes, if testing is required and developers to pull sub-modules as well.
Why should package maintainers be affected? The unit testing framework IMO is a pure development thing that is only executed on a developer's machine and shouldn't be part of the application package. From a technical point of view the unit tests create stand-alone applications where all the Google test code is linked statically.
Why should this be restricted to developers only? From my expierence with other packages I'm maintaining, a majority of testing failure reports comes from package maintainers. There are several distributions which seem to have it in their building and distribution workflow. Reasons? QA, verifying the build is working like it should. AFAIK there are even requirements for having and running unit tests before packages are distributed, for example for Python packages on PyPI. On Gentoo we practically mirror this requirement. Being a source based distribution thus requires the user to have the ability to run those tests if they like.
wmayer
Founder
Posts: 20243
Joined: Thu Feb 19, 2009 10:32 am
Contact:

Re: C++ Unit testing framework up and running

Post by wmayer »

Why should this be restricted to developers only?...
OK, but keep in mind that these tests are not as easily accessible as the current integration tests with the Testing workbench. These are separate executables and a user must run them from the command line in order to see any results. Another thing is that the packages will get quite bloated if each class has its own test executable.
berniev
Posts: 247
Joined: Wed Apr 13, 2022 10:45 pm
Location: Oz

Re: C++ Unit testing framework up and running

Post by berniev »

wmayer wrote: Tue Feb 07, 2023 9:58 am OK, but keep in mind that these tests are not as easily accessible as the current integration tests with the Testing workbench. These are separate executables and a user must run them from the command line in order to see any results. Another thing is that the packages will get quite bloated if each class has its own test executable.
With absolutely the greatest respect (and you know I mean that), I don't understand your post.

The whole point of unit tests is they are separate from the code under test. You don't have to run FreeCAD to test a FreeCAD class. Its a whole different ball-game. To run unit tests (once set up) is trivial, esp from within an IDE.

Are unit tests more, or less, accessible? One requires a somewhat functional FreeCAD, the other does not. Does one require a framework? Yes.

Unit testing frameworks like gtest are not limited to running unit tests. They can equally run integration tests etc.

Test sources are tiny. I'm currently experimentally refactoring Base::Unit and have 118 tests in one file, 562 lines.
There's no way I could be doing this refactor without unit tests. Far too risky. But this way is actually fun! These unit tests run very fast: still shows 0ms. I havn't buit the project for many days (no need).

In-built FreeCAD integration tests also take up space (from what I've seen, probably more), and from experience are slow, and contribute to app bloat, even if only in a WB. Keeping tests out of the app is a Good Thing.
if each class has its own test
If. For every class to have its own tests is a worthwhile, but extroadinarily difficult target!
wmayer
Founder
Posts: 20243
Joined: Thu Feb 19, 2009 10:32 am
Contact:

Re: C++ Unit testing framework up and running

Post by wmayer »

berniev wrote: Tue Feb 07, 2023 11:28 am With absolutely the greatest respect (and you know I mean that), I don't understand your post.
I think you misunderstood my points.
The whole point of unit tests is they are separate from the code under test. You don't have to run FreeCAD to test a FreeCAD class. Its a whole different ball-game. To run unit tests (once set up) is trivial, esp from within an IDE.
That's exactly what I tried to explain that the (real) unit tests are separate applications and not bundled with the FreeCAD application. Usually you run them from the IDE directly but it's also possible to run them from the command line. Because of this the tests are not easily accessible by users in case they will be added to the package.

Question: a correctly written unit test that succeeds on the developer's system is it allowed to fail on any other system?

EDIT:
In-built FreeCAD integration tests also take up space (from what I've seen, probably more), and from experience are slow, and contribute to app bloat, even if only in a WB. Keeping tests out of the app is a Good Thing.
My concern was that if for every class an own test application is created and they all statically link the Google test code that requires a lot of extra space. In the meantime I have seen that with your PR this is not the case any more and that there is currently a single test application for all Google-based tests.
User avatar
chennes
Veteran
Posts: 3881
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: C++ Unit testing framework up and running

Post by chennes »

wmayer wrote: Tue Feb 07, 2023 2:01 pm Usually you run them from the IDE directly but it's also possible to run them from the command line. Because of this the tests are not easily accessible by users in case they will be added to the package.
I was just thinking about this problem this morning, because sometimes it might be useful to be able to run these more easily as a user (in case you compiled with a different compiler, and there is some kind of compiler-specific bug). We could easily write a simple unit test runner that loads those executables and runs them from the Test WB.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
User avatar
adrianinsaval
Veteran
Posts: 5541
Joined: Thu Apr 05, 2018 5:15 pm

Re: C++ Unit testing framework up and running

Post by adrianinsaval »

wmayer wrote: Tue Feb 07, 2023 2:01 pm That's exactly what I tried to explain that the (real) unit tests are separate applications and not bundled with the FreeCAD application. Usually you run them from the IDE directly but it's also possible to run them from the command line. Because of this the tests are not easily accessible by users in case they will be added to the package.
building these during packaging doesn't mean that you also need to ship this to the user, the case of gento is special because the user is also the one building the package, but even in other packaging environments it seems like a good idea to build and run the test before deploying a package, even if the package will not include the test binaries, we're likely going to use different dependencies versions and compilers than the devs so we need to know if something breaks due to this.
User avatar
chennes
Veteran
Posts: 3881
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: C++ Unit testing framework up and running

Post by chennes »

adrianinsaval wrote: Tue Feb 07, 2023 2:38 pm use different dependencies versions
Truly well-mocked unit tests will be independent of any library dependencies. It's going to take some refactoring to get our code under test in this way, but if we're really, really doing it right ("by the book" as it were), the only thing that is variable is the compiler.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
berniev
Posts: 247
Joined: Wed Apr 13, 2022 10:45 pm
Location: Oz

Re: C++ Unit testing framework up and running

Post by berniev »

Maybe after you've actually tried using unit tests you'll stop looking for things to worry about?
berniev
Posts: 247
Joined: Wed Apr 13, 2022 10:45 pm
Location: Oz

Re: C++ Unit testing framework up and running

Post by berniev »

An example of aggressive refactoring and tests might give more of an idea
https://github.com/berniev/FreeCAD/tree/Inventor3D
Post Reply