Path Workbench Code restructuring - heads up

Here's the place for discussion related to CAM/CNC and the development of the Path module.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
chrisb
Veteran
Posts: 53945
Joined: Tue Mar 17, 2015 9:14 am

Re: Path Workbench Code restructuring - heads up

Post by chrisb »

Russ4262 wrote: Sat Mar 11, 2023 1:28 pm I ran the JobFixerCopy.py script to the target file,
I like the possibility of this script to treat al files in a directory, but more frequently I would prefer to run it on a single file. Now I create a directory, move the file into it ...

I would like to see a script which can do both: if called with a file as parameter, it converts only that file, if it is called without, it converts all files in the current directory. I could even imagine to distinguish between a file parameter and a directory.

My poor python capabilities could suffice for my personal use, but for a migration script for the whole FreeCAD community I would prefer to see a more experienced Pythonist do it. Russ, would you mind doing this? I would then unstick this topic and you (or I can do it) create a new sticky one, which contains directly in its first post the new script.

Edit: The icing on the cake would be a macro from AddonManager, which would open the file selection dialog for selecting the file, convert it and open the converted file in FreeCAD.
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
Russ4262
Posts: 941
Joined: Sat Jun 30, 2018 3:22 pm
Location: Oklahoma
Contact:

Re: Path Workbench Code restructuring - heads up

Post by Russ4262 »

chrisb wrote: Sat Mar 11, 2023 8:44 pm ... I would prefer to see a more experienced Pythonist do it. ...
I am not such a person.

chrisb wrote: Sat Mar 11, 2023 8:44 pm ... Russ, would you mind doing this? ...
Thought I would give 'er a go.

Updates include:
  • adding textual script detail strings like author and script title
  • additional mappings provided by Gauthier
  • mapping for PathComment objects
  • upgrades later recommended by chisb
  • in_out, target_directory, and current_directory usage modes
  • custom suffix option for directory usage
  • Windows BAT file now allows for a single command-line argument used as a target directory when supplied
in_out mode:
User provides "-i IN_FILE -o OUT_FILE" filenames with paths included as command-line arguments, as originally permitted in sliptonic's version.

target_directory
User provides "-d DIRECTORY" as command-line arguments to apply changes to all FCStd files in specified directory.

current_directory
User provides no command-line arguments to apply changes to all FCStd files in current working directory.

No warranties or guarantees implied, otherwise suggested, supported or otherwise offered. Use at your own risk. Please backup your originals before enjoying with a smile the attached script.

chrisb wrote: Sat Mar 11, 2023 8:44 pm ... The icing on the cake ...
... ain't gonna happen anytime soon by this hombre.

EDIT: 2023-03-15 version 4 posted. It has minor improvements for error handling and a new argument, "--f" to force overwrite of output file if it already exists. Version 3 downloaded about 4 times.

EDIT 2023-03-16a: Version 5 posted. Suggested changes made by chrisb incorporated, with additional refactoring and some usage examples included in header portion of file. V4 downloaded twice.

EDIT 2023-03-16b: Version 6 posted. Fixes for issues reported by GeneFC later in thread. V5 downloaded 3 times.

EDIT 2023-03-19: Version 7 posted in dedicated thread, Code Restructuring - JobFixer Script. V7 adds support for lower case "fcstd" files. Any additional support or discussion is intended to take place there. V6 downloaded twice.

Russell
Last edited by Russ4262 on Sun Mar 19, 2023 3:05 pm, edited 7 times in total.
chrisb
Veteran
Posts: 53945
Joined: Tue Mar 17, 2015 9:14 am

Re: Path Workbench Code restructuring - heads up

Post by chrisb »

Thank you very much! that's eexactly what I had in mind. :D :D
Russ4262 wrote: Sun Mar 12, 2023 3:00 am
chrisb wrote: Sat Mar 11, 2023 8:44 pm ... The icing on the cake ...
... ain't gonna happen anytime soon by this hombre.
To be honest: my favourite cake comes without icing anyway :) .
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
chrisb
Veteran
Posts: 53945
Joined: Tue Mar 17, 2015 9:14 am

Re: Path Workbench Code restructuring - heads up

Post by chrisb »

Russ4262 wrote: Sun Mar 12, 2023 3:00 am ...
I have tried the last version with the single mode in the file attached to this post. I used on macOS the commandline
python3 JobFixerCopy_V3.py -i test.FCStd -o test21.fcstd

This resulted in this message

Code: Select all

Traceback (most recent call last):
  File "/private/tmp/JobFixerCopy_V3.py", line 294, in <module>
    execute()
  File "/private/tmp/JobFixerCopy_V3.py", line 286, in execute
    fixFile(args.filename.name, args.outputfilename)
  File "/private/tmp/JobFixerCopy_V3.py", line 158, in fixFile
    with zipfile.ZipFile(newfilename, "x") as zout:
  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/zipfile.py", line 1248, in __init__
    self.fp = io.open(file, filemode)
FileExistsError: [Errno 17] File exists: 'test21.fcstd'
The target file has only 38 bytes.
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
chrisb
Veteran
Posts: 53945
Joined: Tue Mar 17, 2015 9:14 am

Re: Path Workbench Code restructuring - heads up

Post by chrisb »

I get the same result in directory mode.
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
Russ4262
Posts: 941
Joined: Sat Jun 30, 2018 3:22 pm
Location: Oklahoma
Contact:

Re: Path Workbench Code restructuring - heads up

Post by Russ4262 »

chrisb wrote: Wed Mar 15, 2023 10:08 pm I get the same result in directory mode.
Evening sir.
I am able to reproduce the same errors as both your posts, but only after two consecutive executions of the script with the same arguments. The first run executes correctly without error, but the second produces the 'file exists' error seen in your error posting because the first run succeeded in producing the output file.

I updated the script to version 4, adding some error handling and an option flag, '--f', to force an overwrite of the output file if it already exists. This change seems to squelch the error produced on my end.

Unfortunately, I do not have a MacOS system with which to test. I am on Windows 10. I have run the tests (examples) provided in the windows batch file included with the script. Each example(test) is commented out with a 'rem' statement. These tests run successfully on my machine.

Thanks for the valuable feedback.

Russell

Code: Select all

OS: Windows 10 Version 2009
Word size of FreeCAD: 64-bit
Version: 0.21.0.32198 (Git)
Build type: Release
Branch: master
Hash: f51b2156f35399cab38eef1e957a59ad5a11de66
Python 3.8.16, Qt 5.15.6, Coin 4.0.0, Vtk 9.1.0, OCC 7.6.3
Locale: English/United States (en_US)
Installed mods: 
  * FC_SU
  * freecad.gears 1.0.0
  * PathExp
  * Z_MacroStartup
chrisb
Veteran
Posts: 53945
Joined: Tue Mar 17, 2015 9:14 am

Re: Path Workbench Code restructuring - heads up

Post by chrisb »

Sorry for the false positive, showing a buglet which I wasn't hunting for.
V4 works excellent now.

May I suggest to change line 302 from

Code: Select all

    print(f"Using suffix:  {suffix}")
to

Code: Select all

    if not args.outputfilename:
        print(f"Using suffix:  {suffix}")
because in the case where an explicit outputfile is given, the suffix isn't used at all.

I would also add to the usage section above:
in_out mode:
User provides "-i IN_FILE -o OUT_FILE" filenames with paths included as command-line arguments, as originally permitted in sliptonic's version. The OUT_FILE parameter is mandatory in that case.
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
chrisb
Veteran
Posts: 53945
Joined: Tue Mar 17, 2015 9:14 am

Re: Path Workbench Code restructuring - heads up

Post by chrisb »

Instead of just documenting I would rather propose another change: if an input file is given and no outputfile, then the outputfilename is constructed like in directory mode. This is the corresponding change of the elif in line 313:

Code: Select all

    elif not args.directory and args.filename:
        if args.outputfilename:
            outfilename = args.outputfilename
        else:
            outfilename = getNewFileName(args.filename.name)
        # print(f"Converting {args.filename}  to  {outfilename}")
        fixFile(args.filename.name, outfilename)
I would also like to make another change: in case the parameter --f is given, there appears still the message
Output file exists!
While this is correct, it suggests that the file may not be replaced. So I would in that case
either add another message Overwrite outputfile!
or replace the message with Output file exists and will be overwritten!
or completely suppress this message.

I can do it, just tell me what you prefer.
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
chrisb
Veteran
Posts: 53945
Joined: Tue Mar 17, 2015 9:14 am

Re: Path Workbench Code restructuring - heads up

Post by chrisb »

Here is my complete modified script including the modified message in case the file exists:

Code: Select all

# -*- coding: utf-8 -*-

import argparse
import os.path
import zipfile
import xml.etree.ElementTree as ET
import base64
import re

__title__ = "Job Fixer"
__author__ = "sliptonic, Russ4262"
__doc__ = "Helper script to migrate pre-0.21 Path Job to 0.21 code structure."
__usage__ = "This script has three modes: in_out, target_directory, and working_directory. Migrated files will created with '_current' suffix added to filename."
__created__ = "2022"
__updated__ = "2023-03-15"
__contributors__ = "Gauthier"

suffix = "_current"
force = False

objectmaps = {
    "PathScripts.PathAdaptive": "Path.Op.Adaptive",
    "PathScripts.PathCustom": "Path.Op.Custom",
    "PathScripts.PathDeburr": "Path.Op.Deburr",
    "PathScripts.PathDressupDogbone": "Path.Dressup.DogboneII",
    "PathScripts.PathDressupHoldingTags": "Path.Dressup.Tags",
    "PathScripts.PathDrilling": "Path.Op.Drilling",
    "PathScripts.PathEngrave": "Path.Op.Engrave",
    "PathScripts.PathHelix": "Path.Op.Helix",
    "PathScripts.PathIconViewProvider": "Path.Base.Gui.IconViewProvider",
    "PathScripts.PathJob": "Path.Main.Job",
    "PathScripts.PathMillFace": "Path.Op.MillFace",
    "PathScripts.PathPocket": "Path.Op.Pocket",
    "PathScripts.PathPocketShape": "Path.Op.Pocket",
    "PathScripts.PathProbe": "Path.Op.Probe",
    "PathScripts.PathProfile": "Path.Op.Profile",
    "PathScripts.PathProfileContour": "Path.Op.Profile",
    "PathScripts.PathSetupSheet": "Path.Base.SetupSheet",
    "PathScripts.PathSlot": "Path.Op.Slot",
    "PathScripts.PathStock": "Path.Main.Stock",
    "PathScripts.PathSurface": "Path.Op.Surface",
    "PathScripts.PathThreadMilling": "Path.Op.ThreadMilling",
    "PathScripts.PathToolBit": "Path.Tool.Bit",
    "PathScripts.PathToolController": "Path.Tool.Controller",
    "PathScripts.PathVcarve": "Path.Op.Vcarve",
    "PathScripts.PathWaterline": "Path.Op.Waterline",
    "PathScripts.PathComment": "Path.Op.Gui.Comment",
}

viewprovidermaps = {
    "PathScripts.PathAdaptiveGui": "Path.Op.Gui.Adaptive",
    "PathScripts.PathCustomGui": "Path.Op.Gui.Custom",
    "PathScripts.PathDeburrGui": "Path.Op.Gui.Deburr",
    "PathScripts.PathDressupTagGui": "Path.Dressup.Gui.Tags",
    "PathScripts.PathDrillingGui": "Path.Op.Gui.Drilling",
    "PathScripts.PathEngraveGui": "Path.Op.Gui.Engrave",
    "PathScripts.PathHelixGui": "Path.Op.Gui.Helix",
    "PathScripts.PathIconViewProvider": "Path.Base.Gui.IconViewProvider",
    "PathScripts.PathJobGui": "Path.Main.Gui.Job",
    "PathScripts.PathMillFaceGui": "Path.Op.Gui.MillFace",
    "PathScripts.PathOpGui": "Path.Op.Gui.Base",
    "PathScripts.PathPocketGui": "Path.Op.Gui.Pocket",
    "PathScripts.PathPocketShapeGui": "Path.Op.Gui.Pocket",
    "PathScripts.PathProbeGui": "Path.Op.Gui.Probe",
    "PathScripts.PathProfileContourGui": "Path.Op.Gui.Profile",
    "PathScripts.PathProfileGui": "Path.Op.Gui.Profile",
    "PathScripts.PathSetupSheetGui": "Path.Base.Gui.SetupSheet",
    "PathScripts.PathSlotGui": "Path.Op.Gui.Slot",
    "PathScripts.PathSurfaceGui": "Path.Op.Gui.Surface",
    "PathScripts.PathToolBitGui": "Path.Tool.Gui.Bit",
    "PathScripts.PathToolControllerGui": "Path.Tool.Gui.Controller",
    "PathScripts.PathVcarveGui": "Path.Op.Gui.Vcarve",
    "PathScripts.PathWaterlineGui": "Path.Op.Gui.Waterline",
    "PathScripts.PathComment": "Path.Op.Gui.Comment",
}


def cleanupBase64String(base64_string, node):
    """cleanupBase64String(base64_string, node) returns modified Base64 string as necessary and able
    Base64 de/encoding code used from https://www.geeksforgeeks.org/encoding-and-decoding-base64-strings-in-python/
    """
    # print(f"raw value: {base64_string}")
    base64_bytes = base64_string.encode("ascii")
    string_bytes = base64.b64decode(base64_bytes)
    string = string_bytes.decode("ascii")
    # print(f"Decoded string: {string}")
    if not string.startswith("{") or not string.endswith("}"):
        # print(f"original string: {string}")
        return base64_string

    dictStr = eval(string)
    # print(f"dictStr: {dictStr}")
    if "editModule" in dictStr.keys():
        key = "editModule"
    elif "OpPageModule" in dictStr.keys():
        key = "OpPageModule"
    elif "module" in dictStr.keys():
        key = "module"
    else:
        # print(f"original string: {string}")
        return base64_string

    pathmaps = objectmaps if node == "ObjectData" else viewprovidermaps
    module = dictStr[key]
    if module not in pathmaps:
        print(f"module {module} not substituted")
        return base64_string

    mapout = pathmaps[module]
    dictStr[key] = mapout
    # print(f"NEW pair: editModule: {mapout}")
    asciiStr = "{" + ", ".join([f'"{k}": "{v}"' for k, v in dictStr.items()]) + "}"
    # print(asciiStr)
    new_string_bytes = asciiStr.encode("ascii")
    new_base64_bytes = base64.b64encode(new_string_bytes)
    return new_base64_bytes.decode("ascii")


def cleanupDoc(docxml, node):
    print(f"cleaning {node}")
    tree = ET.parse(docxml)
    root = tree.getroot()

    pathmaps = objectmaps if node == "ObjectData" else viewprovidermaps

    objects = root.find(node)
    for child in objects:
        res = child.findall("./Properties/Property[@name='Proxy']/Python")
        for r in res:
            try:
                modstring = r.attrib["module"]
            except:
                pass
                # print(ET.tostring(r))
            if modstring not in pathmaps:
                print(f"module {modstring} not substituted")
            else:
                # if node =="ViewProviderData" and modstring == "PathScripts.PathOpGui":
                #    opname = child.attrib['name']
                #    print(f"Operation node: {opname}")
                #    if opname == "Slot":
                #        #mapout = pathmaps[modstring]
                #        #r.attrib["module"] = f"{mapout}.Slot"
                #        r.attrib["encoded"] = "no"
                #        r.attrib["value"] = ""

                mapout = pathmaps[modstring]
                # print(f"mapping: {modstring}  to  {mapout}")
                r.attrib["module"] = mapout
                r.attrib["value"] = cleanupBase64String(r.attrib["value"], node)

                # print(f"substituted {mapout} for {modstring}")
    newXML = ET.tostring(root)
    return newXML


def fixFile(filename, newfilename):
    print(f"Input file: {filename}")
    print(f"      Output: {newfilename}")
    mode = "w" if force else "x"
    if os.path.exists(newfilename):
        if mode == "x":
            print("      Output file exists!")
            print("     Consider '--f' command-line argument to force overwriting.")
            return
        else:
            print("      Output file exists and will be overwritten!")
    try:
        with zipfile.ZipFile(filename, "r") as zin:
            with zipfile.ZipFile(newfilename, mode) as zout:
                zout.comment = zin.comment
                for item in zin.infolist():
                    if item.filename == "Document.xml":
                        with zin.open(item.filename) as docxml:
                            newdata = cleanupDoc(docxml, "ObjectData")
                            zout.writestr(item.filename, newdata)
                    elif item.filename == "GuiDocument.xml":
                        with zin.open(item.filename) as docxml:
                            newdata = cleanupDoc(docxml, "ViewProviderData")
                            zout.writestr(item.filename, newdata)
                    else:
                        zout.writestr(item, zin.read(item.filename))
        # Ewith
    except FileExistsError as ee:
        print(f"      ERROR: {ee}")
    except Exception as ee:
        print(f"      ERROR: {ee}")


def getNewFileName(c):
    # Add suffix to filename
    cParts = c.split(".")
    cParts[-2] = cParts[-2] + suffix
    new_file = ".".join(cParts)
    return new_file


def processDirectory(wrkDir, directory=""):
    # print("Processing contents of base directory.")
    if directory != "":
        os.chdir(directory)
        cwd = directory
    else:
        cwd = wrkDir
    contents = os.listdir(cwd)
    # print(f"Contents: {contents}")
    candidates = [
        cnt
        for cnt in contents
        if str(cnt).endswith(".FCStd")
        and not (
            str(cnt).endswith(suffix + ".FCStd") or str(cnt).endswith("_current.FCStd")
        )
    ]
    for c in candidates:
        # print(f"\nCandidate: {c}")
        new_file = getNewFileName(c)
        if os.path.isfile(cwd + "\\" + new_file):
            print(f"Skipping {c}. Already fixed.")
            continue
        fixFile(c, new_file)
    if directory != "":
        os.chdir(wrkDir)


def is_valid_file(parser, arg):
    if not os.path.exists(arg):
        parser.error("The file %s does not exist!" % arg)
    else:
        return open(arg, "r")  # return an open file handle


def is_valid_directory(parser, arg):
    if arg == "":
        return arg
    elif not os.path.exists(arg):
        parser.error(f"The directory '{arg}' does not exist!")
    else:
        return arg


def is_valid_suffix(parser, arg):
    if not isinstance(arg, str):
        parser.error(f"The suffix provided is not a string.")
    if len(arg) > 25:
        parser.error(f"The suffix length is greater than 50 characters.")
    if len(arg) < 1:
        parser.error(f"The suffix is empty.")
    if re.search("\W", arg):  # search for non-alphanumeric characters
        parser.error(f"The suffix contains illegal characters.")
    else:
        return arg


def execute():
    global suffix
    global force

    print(f"{__title__} {__updated__}")

    parser = argparse.ArgumentParser(
        exit_on_error=False, description="Fixes FreeCAD Path Jobs"
    )
    parser.add_argument(
        "-i",
        dest="filename",
        # required=True,
        help="input FreeCAD project file, requires -o",
        metavar="FILE",
        type=lambda x: is_valid_file(parser, x),
    )
    parser.add_argument(
        "-o",
        dest="outputfilename",
        # required=True,
        help="output FreeCAD project file, requires -i",
        metavar="FILE",
    )
    parser.add_argument(
        "-d",
        dest="directory",
        help="directory containing one or more FCStd files, use without -i and -o",
        metavar="DIRECTORY",
        type=lambda x: is_valid_directory(parser, x),
    )
    parser.add_argument(
        "--s",
        dest="suffix",
        help="optional custom suffix for directory files, default='_current', [a-zA-Z0-9_]",
        metavar="SUFFIX",
        default="_current",
        type=lambda x: is_valid_suffix(parser, x),
    )
    parser.add_argument(
        "--f",
        dest="force",
        help="optional force flag to overwrite existing files",
        action="store_true",
    )
    args = parser.parse_args()

    if args.suffix:
        suffix = args.suffix
    if not args.outputfilename:
        print(f"Using suffix:  {suffix}")
    if args.force:
        force = True
        print("Forcing overwrite of output file if it exists.")

    cwd = os.getcwd()
    # print(f"CWD:  {cwd}")
    if args.directory and not args.filename and not args.outputfilename:
        # print(f"Processing directory provided: {args.directory}")
        processDirectory(cwd, directory=args.directory)
    elif not args.directory and args.filename:
        if args.outputfilename:
            outfilename = args.outputfilename
        else:
            outfilename = getNewFileName(args.filename.name)
        # print(f"Converting {args.filename}  to  {outfilename}")
        fixFile(args.filename.name, outfilename)
    elif not args.directory and not args.filename and not args.outputfilename:
        # print("Processing current working directory.")
        processDirectory(cwd)
    else:
        print(f"Arguments error: Check the arguments and syntax.")

execute()
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
GeneFC
Veteran
Posts: 5373
Joined: Sat Mar 19, 2016 3:36 pm
Location: Punta Gorda, FL

Re: Path Workbench Code restructuring - heads up

Post by GeneFC »

chrisb wrote: Thu Mar 16, 2023 9:36 am
@mlampert

@Russ4262

I think it would be a good idea to modify the very first post in this topic to include the latest version of the JobFixer.

That will make it more easily discoverable.

I have seen this approach in a number of other topics where an important file was updated on a regular basis.

Gene
Post Reply