C++ Unit testing framework up and running

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
berniev
Posts: 247
Joined: Wed Apr 13, 2022 10:45 pm
Location: Oz

C++ Unit testing framework up and running

Post by berniev »

Early December last I introduced GoogleTest framework to FreeCAD and proved with some simple tests. The venerable Mr Mayer got QtTests working as well. Today the structure has been revised and is hopefully ready for you to play. Hopefully this small step will prove useful.

IDE's can be setup to handle managing tests or you can use the command line. I'm no expert here and use CLion - works great.

One benefit of a unit test is that so long as cmake completes ok, the test will compile only what is needed for that test and be ready to run. The project does not have to compile! So adding a test can be done quickly.

The tests directory structure shadows that of the project. Logically then tests for a class go in the corresponding directory and be named like the class. A few test files and directories have been created and can be used as a guide.

The QtTest stuff I have shuffled into its own directory. Eventually these will become important. Anecdotally they are more complex and not nearly as fast.

In the bigger picture imagine every class is tested. FC has ? 4000 files so ? 4000 classes? If each class has 10 tests that would ultimately be 40,000 tests. To be useful, tests have to be run often. Very often. So tests have to run fast. Blindingly fast. So they have to very small and test just one unit -- without its associated dependencies. This is the main difference between unit tests and so-called integration tests and regression tests. Unit tests are about highlighting problems very early, and precisely. They test many, many small details.

Dependencies are the biggest enemy of unit testing (as are macros, statics, singletons, huge classes, huge functions). Isolating a unit from its dependencies can be difficult, especially for existing code, and brings sharply into focus that old code was likely not designed to be unit-tested and may need refactoring.

I recommend three excellent books:

Refactoring - Improving the design of Existing Code, Martin Fowler, 2019
Test-Driven Development -- By Example, Kent Beck, 2003
Working Effectively With Legacy Code, Michael C. Feathers, 2005

Perhaps follow the example tests in the tests dir, add a new test on a simple class and see how it works.

Now imagine that you are writing a new piece of code. You know what you want it to do but the execution is tricky to get right.
Or alternatively you want to alter some existing algorithm.
In both cases if there are tests in place you can hack the thing to bits knowing if you've broken it.
User avatar
waebbl
Posts: 200
Joined: Thu Aug 16, 2018 3:12 pm

Re: C++ Unit testing framework up and running

Post by waebbl »

Thanks a lot for your work on the testing framework! I think it's a good move to add these and continually improve the code quality by providing and using unit testing cases.

Yet I wonder one thing: Why have you included the googletest sources? IMO this adds some unneeded code to the project and increases the code size as well as the complexity for building the unit tests. How about an algorithm like

Code: Select all

find_package(GTest)
if(NOT GTest_FOUND)
  # use the external_project and / or fetch_content mechanics of cmake to pull, build and install (into the sandbox) the gtest code
endif()
I think it's preferrable to use a system provided (test) package to keep the code lean and portable. Is it meant to get moved into this direction later on and just be a proof of concept or do you want to keep that package included?
berniev
Posts: 247
Joined: Wed Apr 13, 2022 10:45 pm
Location: Oz

Re: C++ Unit testing framework up and running

Post by berniev »

Thanks for your kind words!

Priority #1 is get framework in. Tick.
Priority#2 is get people using unit testing. Big challenge here. Experience last year is that it is not understood at FC. Turned out to be unexpectedly controversial.

I am not a devops guru. If there is a better way I am all ears! Your example tests if package is installed then goes looking for it externally. I don't think gtest is in FC lib pack so that immediately causes problems for Windoze users. I had this issue recently trying to get fmtlib into FC and was sternly (and justly) rebuked by The Master.

I don't get your comment re "complexity for building unit tests". It's rediculously simple.

For now the biggest issue is to get FC devs across the concept of real unit tests in C++ and any help on that front is appreciated in advance.
User avatar
adrianinsaval
Veteran
Posts: 5541
Joined: Thu Apr 05, 2018 5:15 pm

Re: C++ Unit testing framework up and running

Post by adrianinsaval »

berniev wrote: Thu Feb 02, 2023 8:13 am I don't think gtest is in FC lib pack so that immediately causes problems for Windoze users. I had this issue recently trying to get fmtlib into FC and was sternly (and justly) rebuked by The Master.
a bit off topic here, but rather than adding more third party code into our repo we should try to add them to the libpack instead, the libpack should be determined by our needs, not drive how we develop.
User avatar
wandererfan
Veteran
Posts: 6265
Joined: Tue Nov 06, 2012 5:42 pm
Contact:

Re: C++ Unit testing framework up and running

Post by wandererfan »

berniev wrote: Thu Feb 02, 2023 8:13 am I don't get your comment re "complexity for building unit tests". It's rediculously simple.

For now the biggest issue is to get FC devs across the concept of real unit tests in C++ and any help on that front is appreciated in advance.
This is really good stuff, but I'm going to complain just a little bit. This implementation has a "build it and they will come" flavour to it.

Unless I missed something, I'm expected to read 3 books, search the web to find tutorials or introductions to GoogleTest framework, study some samples, then try to figure out how to apply it to my own work. I'm willing to do those things, but I will need the infamous
"round tuit" first.

Any chance you can put together or point us to a "welcome to unit testing using GoogleTest in FreeCAD" tutorial/presentation/lesson?
openBrain
Veteran
Posts: 9034
Joined: Fri Nov 09, 2018 5:38 pm
Contact:

Re: C++ Unit testing framework up and running

Post by openBrain »

wandererfan wrote: Thu Feb 02, 2023 2:25 pm Unless I missed something, I'm expected to read 3 books, search the web to find tutorials or introductions to GoogleTest framework, study some samples, then try to figure out how to apply it to my own work. I'm willing to do those things, but I will need the infamous
"round tuit" first.
You also have to congratulate berniev and tell him how greater he is compared to us all. :D
berniev
Posts: 247
Joined: Wed Apr 13, 2022 10:45 pm
Location: Oz

Re: C++ Unit testing framework up and running

Post by berniev »

WHY:

Last July I noticed an anomaly in the code todo with column numbers. https://github.com/FreeCAD/FreeCAD/pull/6916.
I couldn't figure out by looking at it if it was correct, but some function names didn't make sense.

How to test this code? What I did was write a version of the code in Compiler Explorer and test it there. Here's the result:

Code: Select all

Before refactor:

    In  Silent      Expect      Actual   Comment
             0   exception          26  * FAIL *
     A       0           0           0          
     Z       0          25          25          
    AA       0          26          26          
    AB       0          27          27          
    AZ       0          51          51          
    BA       0          52          52          
    CB       0          79          79          
    ZA       0         676         676          
    ZZ       0         701         701          
   AAA       0         702          26  * FAIL *
   AAZ       0         727          51  * FAIL *
   CBA       0        2080        1404  * FAIL *
   AZA       0        1352         676  * FAIL *
   ZZA       0       18252       17576  * FAIL *
   ZZZ       0       18277       17601  * FAIL *
   ALL       0         999         323  * FAIL *
    Ab       0   exception   exception          
  ABCD       0   exception         757  * FAIL *
             1          -1          26  * FAIL *
    Ab       1          -1          -1          
  ABCD       1          -1         757  * FAIL *

After refactor: All tests pass.
The code was modified until all tests pass and then carefully crafted back into FC.

The above was quite a bit of work and whilst it worked, it was itself at risk of introducing problems due to manual transcriptions. Also the tests were not available as a permanent monitor of that code.

HOW:

New file FreeCAD/tests/src/App/Range.cpp:

Code: Select all

#include "gtest/gtest.h"

#include "App/Range.h"

TEST(Range, A){
    EXPECT_EQ(App::decodeColumn("A", false), 0);
}
TEST(Range, Z){
    EXPECT_EQ(App::decodeColumn("Z", false), 25);
}
TEST(Range, AAA){
    EXPECT_EQ(App::decodeColumn("AAA", false), 702);
}
TEST(Range, TestFail){
    EXPECT_EQ(App::decodeColumn("AAA", false), 703);
}
The last is designed to fail. Assures us the testing is working! They say always start with a failing test and refactor until it passes, but I've included it for brevity.

Edit: Add an entry in FreeCAD/tests/src/App/CMakeLists.txt

Build the tests. Takes literally just a few seconds. Run is instant.

Code: Select all

/Users/bernie/CLionProjects/FreeCAD/tests/src/App/Range.cpp:15: Failure
Expected equality of these values:
  App::decodeColumn("AAA", false)
    Which is: 702
  703

Add sufficient tests to highlight possible failure modes.

MORE:

Unit testing is mainstream, not some new esoteric or theoretical concept.
Google:
"C++ unit testing" => "About 19,700,000 results"
"youtube C++ unit testing" => "About 11,600,000 results"
Last edited by berniev on Sat Feb 04, 2023 9:31 pm, edited 1 time in total.
User avatar
waebbl
Posts: 200
Joined: Thu Aug 16, 2018 3:12 pm

Re: C++ Unit testing framework up and running

Post by waebbl »

berniev wrote: Thu Feb 02, 2023 8:13 am I am not a devops guru. If there is a better way I am all ears! Your example tests if package is installed then goes looking for it externally. I don't think gtest is in FC lib pack so that immediately causes problems for Windoze users. I had this issue recently trying to get fmtlib into FC and was sternly (and justly) rebuked by The Master.

I don't get your comment re "complexity for building unit tests". It's rediculously simple.

For now the biggest issue is to get FC devs across the concept of real unit tests in C++ and any help on that front is appreciated in advance.
I'm a package maintainer for a Linux distribution and not a core developer, so my questions are from this POV and related to the build process, which includes building and running unit tests. With added complexity I meant complexity of the build process. It's not a huge jump in complexity though, but you still have to consider whether there are particular cases on how to handle building the bundled googletest framework on different platforms.

So I basically only wanted to know on the plans of including this external code, so I can decide how I can handle this in the package for my distribution. We usually try to use external packages whenever they are available and it's possible to use them. My questions didn't call for priority at all.

The possible user base which are likely to use the test framework is rather small. IMO these includes:
  • CI / CD
  • developers
  • package maintainers
Based on this, I can see several ways on how to handle the issue for Windows:
  • pre-install the googletest framework, there's a windows version available, isn't it? I think this can be done in a CI environment, developers could be tasked that it's a required dependecy if they want to contribute to developing, and package maintainers could easily add it as dependency to their build recipes
  • for Windows, it could be added to the libpack
  • the example I showed could be used, i.e. we use mechanisms of cmake to download, build and install gtest only if it's not available on the platform, which I wouldn't recommend in the first place and only use it as a last exit strategy
I agree with @adrianinsaval that we possibly shouldn't include the gtest source code with the FreeCAD sources. GTest is a tool we use for a particular purpose when building FC. We don't include other tools which are used by the project, i.e. cmake, boost, Qt, etc. While we have some third party packages this imposes the need to also update the dependency and keep it current all the time.

A little off topic is a talk from FOSDEM 2018, which shouldn't be taken too seriously and which has some good points about good and less good software practices.
berniev
Posts: 247
Joined: Wed Apr 13, 2022 10:45 pm
Location: Oz

Re: C++ Unit testing framework up and running

Post by berniev »

I think I understand the concerns.

The install guides I found for gtest all included source in one form or another either included directly or automatically downloaded by cmake.

Using a pkg is certainly simpler.

Are you in a position to add gtest (and libfmt !) to libpack? I definitely am not.
User avatar
waebbl
Posts: 200
Joined: Thu Aug 16, 2018 3:12 pm

Re: C++ Unit testing framework up and running

Post by waebbl »

berniev wrote: Fri Feb 03, 2023 2:25 pm Are you in a position to add gtest (and libfmt !) to libpack? I definitely am not.
I'm neither. I don't have access to a Windows developer machine.
Post Reply