Feature #7551 - Support to create a compound from a dxf file (was: Importing a .dxf file takes forever)

Post here for help on using FreeCAD's graphical user interface (GUI).
Forum rules
and Helpful information
IMPORTANT: Please click here and read this first, before asking for help

Also, be nice to others! Read the FreeCAD code of conduct!
Frotz
Posts: 105
Joined: Thu Jun 21, 2018 1:52 am

Feature #7551 - Support to create a compound from a dxf file (was: Importing a .dxf file takes forever)

Post by Frotz »

Why would attempting to import a .dxf file take hours with no end in sight? I imported https://www.classicarcadecabinets.com/u ... gaetan.dxf Donkey Kong Cab - Jakobud.dxf, which took maybe a minute. Another, https://www.classicarcadecabinets.com/u ... sic-v4.dxf, I let import for a couple hours and it still hadn't finished importing. The window just stops updating during the import, so all I know about what was going on is what's printed to the terminal:

Code: Select all

CDxfRead::ReadLayer() - no layer name
CDxfRead::DoRead() Failed to read layer
CDxfRead::ReadLayer() - no layer name
CDxfRead::DoRead() Failed to read layer
By contrast, importing the same file into Vcarve Pro running under Wine took about ten seconds.

Is there a better way to go about this other than start the import and go away for several hours?

Code: Select all

OS: Debian GNU/Linux 11 (bullseye) (MATE/mate)
Word size of FreeCAD: 64-bit
Version: 0.21.30417 (Git)
Build type: Unknown
Branch: master
Hash: 891637f00eaf32211927072943573f054942841c
Python 3.9.2, Qt 5.15.2, Coin 4.0.0, Vtk 7.1.1, OCC 7.5.1
Locale: English/United States (en_US)
Installed mods: 
  * Assembly3 0.11.3
  * sheetmetal 0.2.57
  * A2plus 0.4.59
  * fasteners 0.4.6
  * Manipulator 1.4.9
(edit) I got the first file wrong. Unfortunately, I can't remember where I got it and it's too big to post here.
Last edited by Frotz on Sun Sep 25, 2022 11:58 pm, edited 1 time in total.
GeneFC
Veteran
Posts: 5373
Joined: Sat Mar 19, 2016 3:36 pm
Location: Punta Gorda, FL

Re: Importing a .dxf file takes forever

Post by GeneFC »

The Nintendo dxf took about a minute on my modest computer. I did get a warning, however.

Warning.JPG
Warning.JPG (20.73 KiB) Viewed 1505 times

I do not understand the details, but this may be a clue.

Gene
Frotz
Posts: 105
Joined: Thu Jun 21, 2018 1:52 am

Re: Importing a .dxf file takes forever

Post by Frotz »

I couldn't even get that far.
User avatar
thomas-neemann
Veteran
Posts: 11801
Joined: Wed Jan 22, 2020 6:03 pm
Location: Osnabrück DE 🇩🇪
Contact:

Re: Importing a .dxf file takes forever

Post by thomas-neemann »

Frotz wrote: Sun Sep 25, 2022 11:27 pm ...
the large file also takes a long time for me. wmayer's recommendation also works here. first import into varicad viewer, export, then import into freecad. (about 20 seconds for me)
Bildschirmfoto_2022-09-26_08-20-52.png
Bildschirmfoto_2022-09-26_08-20-52.png (137.59 KiB) Viewed 1423 times

Code: Select all

OS: Ubuntu 20.04.1 LTS (XFCE/xubuntu)
Word size of FreeCAD: 64-bit
Version: 0.20.1.29410 (Git) AppImage
Build type: Release
Branch: (HEAD detached at 0.20.1)
Hash: f5d13554ecc7a456fb6e970568ae5c74ba727563
Python 3.10.5, Qt 5.15.4, Coin 4.0.0, Vtk 9.1.0, OCC 7.6.2
Locale: German/Germany (de_DE)
Gruß Dipl.-Ing. (FH) Thomas Neemann

https://www.youtube.com/@thomasneemann5 ... ry=freecad
User avatar
ThanklessLiving
Posts: 99
Joined: Sat May 30, 2020 1:49 pm

Re: Importing a .dxf file takes forever

Post by ThanklessLiving »

I opened the file in LibreCAD and by the looks of it it's a huge mess of text in the form of lines. I would recommend making separate files of each "part", then it is more likely to import successfully and you can also find which sections are problematic, if there are any. Another alternative is to omit the text all together and import without it, and add the text in FreeCAD.
Frotz
Posts: 105
Joined: Thu Jun 21, 2018 1:52 am

Re: Importing a .dxf file takes forever

Post by Frotz »

ThanklessLiving wrote: Tue Sep 27, 2022 9:09 pm I opened the file in LibreCAD and by the looks of it it's a huge mess of text in the form of lines. I would recommend making separate files of each "part", then it is more likely to import successfully and you can also find which sections are problematic, if there are any. Another alternative is to omit the text all together and import without it, and add the text in FreeCAD.
I tried editing the file in LibreCAD, but couldn't figure out how to isolate portions of the drawing. What I ultimately did was get a friend of mine to export only a single side panel from the VCarve Pro file to .dxf. This took about ten minutes to load, but it was enough to clue to me in to just what FreeCAD was choking on -- that huge mess of text that you also noticed. This is probably partially the fault of VCarve Pro for making such a mess and partially FreeCAD for not being able to efficiently deal with a large number of short segments. I managed to get what I wanted by zooming in and selecting small blocks of segments forming letters to delete them. The time between each of these operations was about two to five seconds. Saving my progress started at about ten minutes down to two seconds once I got rid of all the text.
User avatar
ThanklessLiving
Posts: 99
Joined: Sat May 30, 2020 1:49 pm

Re: Importing a .dxf file takes forever

Post by ThanklessLiving »

Frotz wrote: Tue Sep 27, 2022 10:31 pm Saving my progress started at about ten minutes down to two seconds once I got rid of all the text.
Exactly, all those arcs take a lot of computing to get imported. One option is to delete and recreate the curves after importing, another is to "refine" or "simplify" them in the dxf format into simpler arcs. Usually the difference will be a fraction of a millimeter which is hardly important in the case of contours that don't have to fit with anything else.
heda
Veteran
Posts: 1348
Joined: Sat Dec 12, 2015 5:49 pm

Re: Importing a .dxf file takes forever

Post by heda »

hm, selecting something (as in clicking on..) in librecad and hitting the del key works for me... (for the "could not figure out...")

yepp, sadly enough, fc does choke on larger dxf files.
well, one should rather say the dxf importers chokes fc, the signs are such that it is rather the importer than core that is the main culprit here, regardless of python or c importer...

actually in this particular case, I am convinced that would be possible to write an importer in python that is faster and less memory hungry than the current c importer...

it seems like the c importer consumes in the range of 1 Gb RAM for every Mb dxf - for people that know c - that has to be possible to polish a bit.

the trouble fc sometimes has with importing dxf files intrigues me, for whatever reason - think it is an interesting problem - and also think that it is important for fc's well being to have decent capabilities of importing "open" formats.

anyhow, to deal with large dxf in fc, one needs a strategy, currently that means one needs to preprocess the files outside of fc.

so, just by saving the file with lc the filesize is halved, apparently because the original file has layer specified for every entity, i.e. a lot of repeats (this does nothing for import-ability into fc though... - for obvious reasons)

btw, you never mentioned which sw has written these dxf's, one has to wonder if there are some settings there to tweak somewhere...

the issue at hand (with the larger file of the two) simply is the amounts, both amount of entities (large numbers of fc-doc objects makes fc slow) and amount of vertices.

the file is exclusively polylines, so what can one do to reduce those a bit?

one could of course manually delete selected vertices, both lc and inkscape works for that,
however for a file like this - it gets boring quickly...
so, is it possible to script?
turns out that it is - using ezdxf, one can with some simple logic reduce the number of vertices.
the script below will do that, running it twice will demolish 120k -ish points without doing much to visuals.

Code: Select all

"""
command line tool to reduce dxf files
consisting of large amounts of small linesegments

author / copyright: heda@fc-forum 2022
license: MIT
"""

import sys
import ezdxf

DRYRUN = True
MINVERTS = 20
REMOVETHRESHOLD = 0.1

args = sys.argv[1:]
if len(args) == 0:
    print('need at least one input file')
elif len(args) == 1:
    print('making a dryrun')
    filein, = args
elif len(args) == 2:
    filein, fileout = args
    DRYRUN = False

print('opening file:', filein)
doc = ezdxf.readfile(filein)
print('reading file completed...')

rcount = vcount = ecount = rdist = 0
rdistmin = REMOVETHRESHOLD

for entity in doc.entities:
    if entity.dxftype() != 'POLYLINE':
        continue

    verts = entity.vertices
    lverts = len(verts)
    
    if lverts < MINVERTS:
        continue
    
    print('POLYLINE #{}'.format(entity.dxf.get('handle')))
    ecount += 1
    vcount += lverts
    
    idxr = [-1]
    for i, tri in enumerate(zip(verts[0:], verts[1:], verts[2:])):
        if idxr[-1] == i:
            continue
        v0, v1, v2 = (v.dxf.location for v in tri)
        d01 = v0.distance(v1)
        d12 = v1.distance(v2)
        if d01 < REMOVETHRESHOLD and d12 < REMOVETHRESHOLD:
            idxr.append(i+1)
            rdist = max(rdist, d01, d12)
            rdistmin = min(rdistmin, d01, d12)
            if DRYRUN:
                print('  tagged', i+1, round(d01, 4), round(d12, 4))

    
    idxr = idxr[1:][::-1]
    rcount += len(idxr)
    print('  removing tagged: {} of {}'.format(len(idxr), lverts))
    if not DRYRUN:
        print('    dist max: {:.4f}  min: {:.4f}'.format(rdist, rdistmin))
    for i in idxr:
        del entity.vertices[i]

msg = ('finished reduction, removed {} vertices of {} ({:.1f} %)\n'
       'from {} entities. max / min length removed: {:.4f} / {:.4f}')
print(msg.format(rcount, vcount, rcount/vcount*100, ecount, rdist, rdistmin))

if not DRYRUN:
    print('saving file...')
    doc.saveas(fileout)
print('done...')
on my underpowered and useless laptop, this will then import as is (file is down to 4mb, and a save in lc takes it down to 2mb).

a zipped version of that file is attached, i.e. going from 16mb to 0.6 mb...

although this file will import, it still takes forever (as in hours) on my useless laptop (suppose one reason is because it goes to hdd-swap)

so, is there something else that one can do?

turns out there is...
converting the dxf to a svg makes wonders for import time to fc
let's use inkscape for that, for whatever reason inkscape will not import polylines, so one has to use the dxf saved by lc, since lc for whatever reason is converting the polylines to lwpolylines...

importing this svg into fc is quick, also on my useless laptop. the svg is attached as well.

given these circumstances in memory/speed, it clearly points towards that it is possible to write a faster py importer than the current c importer, if the svg importer can largely do the import without breaking a sweat, why would it not be possible to roll a new py importer that is reasonably quick...
afterall, the only thing needed to make an importer for this file is just to add Part.makeWire in the posted script, so with a handfull of additional lines, one has rolled ones own brand new fc-dxf importer...
using ezdxf, one could probably also start supporting nurbs and solids as well from dxf-files without too much effort (if that is something that someone thinks is desirable...)

this is just showing a way, hopefully someone brave will improve the dxf handling in fc (and using ezdxf is for sure giving an ahead start in such endevour..., incorporating on the fly cleaning options in such endevour would be the icing on the cake :-))

for the other dxf, just briefly looked into that one, it is of a different anatomy, so the path outlined here does not work, one would have to find different ways. one of those ways can be libreoffice (draw), which also has a capable and swift dxf-reader and writes svg, so that can be used instead of inkscape, but even if doing that the svg import breaks in fc, it stumbles on creating a zero-length line - that is seems like a fc bug that could be fixed if someone took that on...

at last, massive kudos to the author of ezdxf - time to start using it in fc?

ps: running ezdxf gave an import error to begin with for me, turned out that the reason was a too low version of type_extension, so upgrading that allowed for a clean import of ezdxf
Attachments
ncr2lc.dxf.svg.zip
(171.07 KiB) Downloaded 21 times
ncr2lc.dxf.zip
(586.58 KiB) Downloaded 21 times
Last edited by heda on Tue Oct 04, 2022 8:19 pm, edited 1 time in total.
wmayer
Founder
Posts: 20243
Joined: Thu Feb 19, 2009 10:32 am
Contact:

Re: Importing a .dxf file takes forever

Post by wmayer »

The bottleneck is this part of the dxf import.

When loading the file donkey_kong_cab_-_gaetan.dxf then it will add around 2200 part objects to the document which is already not good but still handable. However, loading the file nintendo-classic-v4.dxf would add almost 170 000 objects to the document which for most systems is by far too much.

Interaction with the 3d view would become extremely slow because the application is busy with traversing the scene graph most of the time. So, just trying to select an object in the view or tree would take a long time.

In order to reduce the loading and further interaction to a sensible time there shouldn't be an own object for every entity of the dxf but a single compound object should be used for all entities.

issue #7551
heda
Veteran
Posts: 1348
Joined: Sat Dec 12, 2015 5:49 pm

Re: Importing a .dxf file takes forever

Post by heda »

as suspected...
with a simple Part.Wire and using ezdxf, the speed is at same level as the svg-importer....
so a handful lines of code eliminates the need to go via librecad/inkscape and svg...

this small snippet tests that, fast enough and with small memory footprint (compared to the current dxf-importer to part shapes)

Code: Select all

import ezdxf

import Part

fdoc = App.newDocument('ncrx')
fdoc.FileName = fdoc.Label + '.FCStd'

filein = '/home/../ncr2.dxf'
ddoc = ezdxf.readfile(filein)

for entity in ddoc.entities:
    if entity.dxftype() != 'POLYLINE':
        continue

    verts = entity.vertices

    verts = [tuple(v.dxf.location) for v in verts]
    if entity.is_closed:
        verts += [verts[0]]
    poly = Part.makePolygon(verts)
    Part.show(poly, 'Polyline')

fdoc.recompute()
fdoc.save()
would say that proof of concept of using ezdxf as basis for a new fc dxf importer is now there...
acc to pypi its mit-license, so should be possible to even include it in the fc-distribution as well

think the only question left is who will have a large enough itch to scratch to get it done.
fwiw, my itch is scratched now, just wanted to see how hard it would be and if I ever have the need to crunch large dxf in the future i would know what to do...
Post Reply