C++ Unit testing framework up and running
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
Be nice to others! Respect the FreeCAD code of conduct!
C++ Unit testing framework up and running
Early December last I introduced GoogleTest framework to FreeCAD and proved with some simple tests. The venerable Mr Mayer got QtTests working as well. Today the structure has been revised and is hopefully ready for you to play. Hopefully this small step will prove useful.
IDE's can be setup to handle managing tests or you can use the command line. I'm no expert here and use CLion - works great.
One benefit of a unit test is that so long as cmake completes ok, the test will compile only what is needed for that test and be ready to run. The project does not have to compile! So adding a test can be done quickly.
The tests directory structure shadows that of the project. Logically then tests for a class go in the corresponding directory and be named like the class. A few test files and directories have been created and can be used as a guide.
The QtTest stuff I have shuffled into its own directory. Eventually these will become important. Anecdotally they are more complex and not nearly as fast.
In the bigger picture imagine every class is tested. FC has ? 4000 files so ? 4000 classes? If each class has 10 tests that would ultimately be 40,000 tests. To be useful, tests have to be run often. Very often. So tests have to run fast. Blindingly fast. So they have to very small and test just one unit -- without its associated dependencies. This is the main difference between unit tests and so-called integration tests and regression tests. Unit tests are about highlighting problems very early, and precisely. They test many, many small details.
Dependencies are the biggest enemy of unit testing (as are macros, statics, singletons, huge classes, huge functions). Isolating a unit from its dependencies can be difficult, especially for existing code, and brings sharply into focus that old code was likely not designed to be unit-tested and may need refactoring.
I recommend three excellent books:
Refactoring - Improving the design of Existing Code, Martin Fowler, 2019
Test-Driven Development -- By Example, Kent Beck, 2003
Working Effectively With Legacy Code, Michael C. Feathers, 2005
Perhaps follow the example tests in the tests dir, add a new test on a simple class and see how it works.
Now imagine that you are writing a new piece of code. You know what you want it to do but the execution is tricky to get right.
Or alternatively you want to alter some existing algorithm.
In both cases if there are tests in place you can hack the thing to bits knowing if you've broken it.
IDE's can be setup to handle managing tests or you can use the command line. I'm no expert here and use CLion - works great.
One benefit of a unit test is that so long as cmake completes ok, the test will compile only what is needed for that test and be ready to run. The project does not have to compile! So adding a test can be done quickly.
The tests directory structure shadows that of the project. Logically then tests for a class go in the corresponding directory and be named like the class. A few test files and directories have been created and can be used as a guide.
The QtTest stuff I have shuffled into its own directory. Eventually these will become important. Anecdotally they are more complex and not nearly as fast.
In the bigger picture imagine every class is tested. FC has ? 4000 files so ? 4000 classes? If each class has 10 tests that would ultimately be 40,000 tests. To be useful, tests have to be run often. Very often. So tests have to run fast. Blindingly fast. So they have to very small and test just one unit -- without its associated dependencies. This is the main difference between unit tests and so-called integration tests and regression tests. Unit tests are about highlighting problems very early, and precisely. They test many, many small details.
Dependencies are the biggest enemy of unit testing (as are macros, statics, singletons, huge classes, huge functions). Isolating a unit from its dependencies can be difficult, especially for existing code, and brings sharply into focus that old code was likely not designed to be unit-tested and may need refactoring.
I recommend three excellent books:
Refactoring - Improving the design of Existing Code, Martin Fowler, 2019
Test-Driven Development -- By Example, Kent Beck, 2003
Working Effectively With Legacy Code, Michael C. Feathers, 2005
Perhaps follow the example tests in the tests dir, add a new test on a simple class and see how it works.
Now imagine that you are writing a new piece of code. You know what you want it to do but the execution is tricky to get right.
Or alternatively you want to alter some existing algorithm.
In both cases if there are tests in place you can hack the thing to bits knowing if you've broken it.
Re: C++ Unit testing framework up and running
Thanks a lot for your work on the testing framework! I think it's a good move to add these and continually improve the code quality by providing and using unit testing cases.
Yet I wonder one thing: Why have you included the googletest sources? IMO this adds some unneeded code to the project and increases the code size as well as the complexity for building the unit tests. How about an algorithm like
I think it's preferrable to use a system provided (test) package to keep the code lean and portable. Is it meant to get moved into this direction later on and just be a proof of concept or do you want to keep that package included?
Yet I wonder one thing: Why have you included the googletest sources? IMO this adds some unneeded code to the project and increases the code size as well as the complexity for building the unit tests. How about an algorithm like
Code: Select all
find_package(GTest)
if(NOT GTest_FOUND)
# use the external_project and / or fetch_content mechanics of cmake to pull, build and install (into the sandbox) the gtest code
endif()
Re: C++ Unit testing framework up and running
Thanks for your kind words!
Priority #1 is get framework in. Tick.
Priority#2 is get people using unit testing. Big challenge here. Experience last year is that it is not understood at FC. Turned out to be unexpectedly controversial.
I am not a devops guru. If there is a better way I am all ears! Your example tests if package is installed then goes looking for it externally. I don't think gtest is in FC lib pack so that immediately causes problems for Windoze users. I had this issue recently trying to get fmtlib into FC and was sternly (and justly) rebuked by The Master.
I don't get your comment re "complexity for building unit tests". It's rediculously simple.
For now the biggest issue is to get FC devs across the concept of real unit tests in C++ and any help on that front is appreciated in advance.
Priority #1 is get framework in. Tick.
Priority#2 is get people using unit testing. Big challenge here. Experience last year is that it is not understood at FC. Turned out to be unexpectedly controversial.
I am not a devops guru. If there is a better way I am all ears! Your example tests if package is installed then goes looking for it externally. I don't think gtest is in FC lib pack so that immediately causes problems for Windoze users. I had this issue recently trying to get fmtlib into FC and was sternly (and justly) rebuked by The Master.
I don't get your comment re "complexity for building unit tests". It's rediculously simple.
For now the biggest issue is to get FC devs across the concept of real unit tests in C++ and any help on that front is appreciated in advance.
- adrianinsaval
- Veteran
- Posts: 5548
- Joined: Thu Apr 05, 2018 5:15 pm
Re: C++ Unit testing framework up and running
a bit off topic here, but rather than adding more third party code into our repo we should try to add them to the libpack instead, the libpack should be determined by our needs, not drive how we develop.
- wandererfan
- Veteran
- Posts: 6309
- Joined: Tue Nov 06, 2012 5:42 pm
- Contact:
Re: C++ Unit testing framework up and running
This is really good stuff, but I'm going to complain just a little bit. This implementation has a "build it and they will come" flavour to it.
Unless I missed something, I'm expected to read 3 books, search the web to find tutorials or introductions to GoogleTest framework, study some samples, then try to figure out how to apply it to my own work. I'm willing to do those things, but I will need the infamous
"round tuit" first.
Any chance you can put together or point us to a "welcome to unit testing using GoogleTest in FreeCAD" tutorial/presentation/lesson?
Re: C++ Unit testing framework up and running
You also have to congratulate berniev and tell him how greater he is compared to us all.wandererfan wrote: ↑Thu Feb 02, 2023 2:25 pm Unless I missed something, I'm expected to read 3 books, search the web to find tutorials or introductions to GoogleTest framework, study some samples, then try to figure out how to apply it to my own work. I'm willing to do those things, but I will need the infamous
"round tuit" first.
Re: C++ Unit testing framework up and running
WHY:
Last July I noticed an anomaly in the code todo with column numbers. https://github.com/FreeCAD/FreeCAD/pull/6916.
I couldn't figure out by looking at it if it was correct, but some function names didn't make sense.
How to test this code? What I did was write a version of the code in Compiler Explorer and test it there. Here's the result:
The code was modified until all tests pass and then carefully crafted back into FC.
The above was quite a bit of work and whilst it worked, it was itself at risk of introducing problems due to manual transcriptions. Also the tests were not available as a permanent monitor of that code.
HOW:
New file FreeCAD/tests/src/App/Range.cpp:
The last is designed to fail. Assures us the testing is working! They say always start with a failing test and refactor until it passes, but I've included it for brevity.
Edit: Add an entry in FreeCAD/tests/src/App/CMakeLists.txt
Build the tests. Takes literally just a few seconds. Run is instant.
Add sufficient tests to highlight possible failure modes.
MORE:
Unit testing is mainstream, not some new esoteric or theoretical concept.
Google:
"C++ unit testing" => "About 19,700,000 results"
"youtube C++ unit testing" => "About 11,600,000 results"
Last July I noticed an anomaly in the code todo with column numbers. https://github.com/FreeCAD/FreeCAD/pull/6916.
I couldn't figure out by looking at it if it was correct, but some function names didn't make sense.
How to test this code? What I did was write a version of the code in Compiler Explorer and test it there. Here's the result:
Code: Select all
Before refactor:
In Silent Expect Actual Comment
0 exception 26 * FAIL *
A 0 0 0
Z 0 25 25
AA 0 26 26
AB 0 27 27
AZ 0 51 51
BA 0 52 52
CB 0 79 79
ZA 0 676 676
ZZ 0 701 701
AAA 0 702 26 * FAIL *
AAZ 0 727 51 * FAIL *
CBA 0 2080 1404 * FAIL *
AZA 0 1352 676 * FAIL *
ZZA 0 18252 17576 * FAIL *
ZZZ 0 18277 17601 * FAIL *
ALL 0 999 323 * FAIL *
Ab 0 exception exception
ABCD 0 exception 757 * FAIL *
1 -1 26 * FAIL *
Ab 1 -1 -1
ABCD 1 -1 757 * FAIL *
After refactor: All tests pass.
The above was quite a bit of work and whilst it worked, it was itself at risk of introducing problems due to manual transcriptions. Also the tests were not available as a permanent monitor of that code.
HOW:
New file FreeCAD/tests/src/App/Range.cpp:
Code: Select all
#include "gtest/gtest.h"
#include "App/Range.h"
TEST(Range, A){
EXPECT_EQ(App::decodeColumn("A", false), 0);
}
TEST(Range, Z){
EXPECT_EQ(App::decodeColumn("Z", false), 25);
}
TEST(Range, AAA){
EXPECT_EQ(App::decodeColumn("AAA", false), 702);
}
TEST(Range, TestFail){
EXPECT_EQ(App::decodeColumn("AAA", false), 703);
}
Edit: Add an entry in FreeCAD/tests/src/App/CMakeLists.txt
Build the tests. Takes literally just a few seconds. Run is instant.
Code: Select all
/Users/bernie/CLionProjects/FreeCAD/tests/src/App/Range.cpp:15: Failure
Expected equality of these values:
App::decodeColumn("AAA", false)
Which is: 702
703
MORE:
Unit testing is mainstream, not some new esoteric or theoretical concept.
Google:
"C++ unit testing" => "About 19,700,000 results"
"youtube C++ unit testing" => "About 11,600,000 results"
Last edited by berniev on Sat Feb 04, 2023 9:31 pm, edited 1 time in total.
Re: C++ Unit testing framework up and running
I'm a package maintainer for a Linux distribution and not a core developer, so my questions are from this POV and related to the build process, which includes building and running unit tests. With added complexity I meant complexity of the build process. It's not a huge jump in complexity though, but you still have to consider whether there are particular cases on how to handle building the bundled googletest framework on different platforms.berniev wrote: ↑Thu Feb 02, 2023 8:13 am I am not a devops guru. If there is a better way I am all ears! Your example tests if package is installed then goes looking for it externally. I don't think gtest is in FC lib pack so that immediately causes problems for Windoze users. I had this issue recently trying to get fmtlib into FC and was sternly (and justly) rebuked by The Master.
I don't get your comment re "complexity for building unit tests". It's rediculously simple.
For now the biggest issue is to get FC devs across the concept of real unit tests in C++ and any help on that front is appreciated in advance.
So I basically only wanted to know on the plans of including this external code, so I can decide how I can handle this in the package for my distribution. We usually try to use external packages whenever they are available and it's possible to use them. My questions didn't call for priority at all.
The possible user base which are likely to use the test framework is rather small. IMO these includes:
- CI / CD
- developers
- package maintainers
- pre-install the googletest framework, there's a windows version available, isn't it? I think this can be done in a CI environment, developers could be tasked that it's a required dependecy if they want to contribute to developing, and package maintainers could easily add it as dependency to their build recipes
- for Windows, it could be added to the libpack
- the example I showed could be used, i.e. we use mechanisms of cmake to download, build and install gtest only if it's not available on the platform, which I wouldn't recommend in the first place and only use it as a last exit strategy
A little off topic is a talk from FOSDEM 2018, which shouldn't be taken too seriously and which has some good points about good and less good software practices.
Re: C++ Unit testing framework up and running
I think I understand the concerns.
The install guides I found for gtest all included source in one form or another either included directly or automatically downloaded by cmake.
Using a pkg is certainly simpler.
Are you in a position to add gtest (and libfmt !) to libpack? I definitely am not.
The install guides I found for gtest all included source in one form or another either included directly or automatically downloaded by cmake.
Using a pkg is certainly simpler.
Are you in a position to add gtest (and libfmt !) to libpack? I definitely am not.