External full Python WB Unit Tests

[gflorent]

I have some questions, for the more experienced folks with FreeCAD, regarding the preferred unit testing strategy for external workbenches.

1. Is any code organisation and testing framework ok (as long as it works) or should an organisation and framework (unittest I guess) be preferred for potential integration with the Test WB?

2. Is it possible to add a test suite for an external WB to the Test WB (without editing the Python code in the Test WB of the installed FreeCAD) or should the usual test runners from a testing framework be used?

3. Should the TemplatePyMod way of writing tests be considered as the example to follow to write tests (since the FreeCAD/Workbench-Starterkit repo does not seem to mention tests)?

Maybe this is all explained somewhere but I could not find the info.

Thanks in advance to anyone who may help with that.

[DeepSOIC]

I hope to be proven wrong... but it looks like, you can't. The test modules are explicitly hard-coded here:
https://github.com/FreeCAD/FreeCAD/blob ... App.py#L56

Code: Select all

def All():
    # Base system tests
    tests = [ "UnicodeTests",
              "Document",
              "UnitTests",
              "BaseTests" ]

    # Base system gui test
    if (FreeCAD.GuiUp == 1):
        tests += [ "Workbench",
                   "Menu" ]

    # add the module tests
    tests += [ "TestFem",
               "MeshTestsApp",
               "TestSketcherApp",
               "TestPartApp",
               "TestPartDesignApp",
               "TestSpreadsheet",
               "TestTechDrawApp",
               "TestPathApp" ]

    # gui tests of modules
    if (FreeCAD.GuiUp == 1):
        tests += [ "TestSketcherGui",
                   "TestPartGui",
                   "TestPartDesignGui",
                   "TestDraft",
                   "TestArch" ]

    suite = unittest.TestSuite()

    for test in tests:
        suite.addTest(tryLoadingTest(test))

    return suite

I guess, you can hack in by overwriting the function All() to inject your own tests, but that doesn't sound great.


EDIT: even worse, it's hard-coded yet again here:
https://github.com/FreeCAD/FreeCAD/blob ... Gui.py#L43

[gflorent]

There is a nice solution in the InitGui.py of https://github.com/microelly2/freecad-nurbs near line 140. The same dialog as the one used in the 'official' Test WB appears (when clicking a 'Test' command) with a drop-down list of unit tests scripts.

Here is the snippet extracted from InitGui.py

Code: Select all

class MyTestCmd2:
    """Opens a Qt dialog with all inserted unit tests"""

    def Activated(self):
        import QtUnitGui
        QtUnitGui.addTest("nurbswb.TestNurbsGui")
        QtUnitGui.addTest("nurbswb.TestNurbs")
        QtUnitGui.addTest("nurbswb.TestMeinAll.Col1")
        QtUnitGui.addTest("nurbswb.TestMeinAll.Col2")
        QtUnitGui.addTest("TestMeinAll.Col2")

    def GetResources(self):
        return {'MenuText': 'Test-test...',
                'ToolTip': 'Runs the self-test for the workbench'}


FreeCADGui.addCommand('My_Test2', MyTestCmd2())

But, unfortunately, I could not find the actual tests scripts in microelly2's repo because of the following line in the .gitignore

Code: Select all

Test*py

@microelly2: I like your testing solution in the Nurbs WB, and I am sure many others will, would you mind sharing the test files?

[DeepSOIC]

I have a different plan. The plan is to make a specialized tester document object, which is to remember and verify the result of a sequence of features. Then I am to make a test project, where I use each feature, visually verify the correctness, and attach a tester object to remember it. And the test is then to load the project, and force-recompute it.

This will probably work OK for Lattice2, which is very feature-centric. But it won't work on part-o-magic, as it is more of a gui/interactivity workbench.

[gflorent]

Maybe we can combine both. The tests ran from the test runner could either check automatically that placement and other properties are as expected or display a dialog where you can click OK or NOT OK based on what you see and controlling the test result. What do you think?

[DeepSOIC]

display a dialog where you can click OK or NOT OK based on what you see and controlling the test result.

If I see it's NOT OK, there's no point clicking the button. I can go straight ahead and debug/fix it.

[sgrogan]

Maybe we can combine both. The tests ran from the test runner could either check automatically that placement and other properties are as expected or display a dialog where you can click OK or NOT OK based on what you see and controlling the test result. What do you think?

FreeCAD must run without the gui. So these type of tests should only run when the gui is up.

[wmayer]

1. Is any code organisation and testing framework ok (as long as it works) or should an organisation and framework (unittest I guess) be preferred for potential integration with the Test WB?

If everything works with your own testing framework then I think it's fine, too.

3. Should the TemplatePyMod way of writing tests be considered as the example to follow to write tests (since the FreeCAD/Workbench-Starterkit repo does not seem to mention tests)?

The file Tests.py shows how to basically write a unit test, i.e. inherit from unittest.TestCase, offer a setUp and tearDown method and offer methods starting with "test"

2. Is it possible to add a test suite for an external WB to the Test WB (without editing the Python code in the Test WB of the installed FreeCAD) or should the usual test runners from a testing framework be used?

Yes, you can add your tests to the Test WB without changing its source. However, a few things must be polished a bit to improve the integration. Assuming you have a module called MyTest which resides in the Mod directory then you can do the following things:

• Create a file __init__.py
• Create a directory Tests with another __init__.py and e.g. TestCase.py
• The content of Tests/__init__.py can e.g. be:

  Code: Select all

  import QtUnitGui
  QtUnitGui.addTest("MyTest.Tests.TestCase")
  

• and TestCase.py

  Code: Select all

  import unittest
  
  class MyTestCases(unittest.TestCase):
    def setUp(self):
      pass
  
    def testNothing(self):
      self.assertTrue(True)
  
    def tearDown(self):
      pass
  

If you now import MyTest.Tests then the tests will be registered and the dialog comes up to run the unit test.

At the moment I haven't found a way to register a test (preferably at startup) without invoking the test dialog. I think this can be done by providing a simple list in the FreeCAD and FreeCADGui modules which stores the module names which will be considered in TestApp.py and TestGui.py. This way you could add tests directly in your Init.py or InitGui.py files.

[wmayer]

With git commit 67b5ee0931 now there are no lists in TestApp.py and TestGui.py but instead each module registers its unit tests via Init.py or InitGui.py. So, with external module it has become much easier to register tests. All what you have to do is adding this line to Init.py or InitGui.py:

Code: Select all

FreeCAD.__unit_test__ += [ "Name_of_my_test_module" ]

[DeepSOIC]

With git commit 67b5ee0931 now there are no lists in TestApp.py and TestGui.py but instead each module registers its unit tests via Init.py or InitGui.py. So, with external module it has become much easier to register tests. All what you have to do is adding this line to Init.py or InitGui.py:

Code: Select all

FreeCAD.__unit_test__ += [ "Name_of_my_test_module" ]

Oh, wow, thanks!