[solved] "All"-File management / database with text files in the directory tree for linux

[thomas-neemann]

Big problems arose among my customers due to databases that have disappeared from the market, were bought up or when software was changed.

i want to program a text-based system. I asked in the forum whether freecad can lock files. if not, it is not a problem to use chmod to lock by loading and to release it again after saving (linux systems)

[user1234]

postgresql is open source and one of the biggest databasesystems in the world. And if the end their support, you can take your sql database and feed it in (with adaption) in a new sql system. This is one of the main function of sql databases.

And when you making this on textifles only, how will you maintain this? Who have writing and reading access to the textfiles. What if someone sabotage the textfiles. How you will get logfiles? And so on .....

Have not someone wrote a macro or addon for sql? furti was his name. I do not know how capable this is/was.

That commercial CAD databasesystems (which often based on sql) ad ERP databasesystems are sometimes a pain, especially with their "service" and licence can understand.

Greetings
user1234

[onekk]

I agree with thomas-neeman about risks of using a database, but take in consideration that it is part of Python standard library.

Take the following as a brainstorming discussion.

PostgresSQL is great, but this is an additional program.

As FreeCAD use Python there is no other prerequisite other than having a Python interpreter on the machine, or in case of AppImage, squlite3 is already on board as it is part of the Python interpreter.

try to use the examples in the link I sent you with an AppImage, and You could test the thing.

Another solution could be to read files, and creating the database on the fly, in memory, to speed up things, with a minimal overhead.

Another things to take in consideration is that sqlite3 is used also as example in Firefox to store many things (even web pages) as it could manage "binary blob", in theory you could even put an entire FCStd file as a content of a field.

Another pratical thing would be to use a json "database", as json is include in Python Standard library too and already used by FreeCAD.

maybe another solution, would be to have a simple python dictionary that could be stored as a json file to be "persistent".

There are many solution other than relying on an external library, even to use a sort of ini file, as I do often when dealing with settings for my programs, not more hassle and quite "human readable too".



Regards

Carlo D.

[thomas-neemann]

... What if someone sabotage the textfiles....

I have been able to secure some companies to the intrnet, but I think it is impossible to secure a company internally. that can only be based on trust. in my opinion

[thomas-neemann]

I agree with thomas-neeman about risks of using a database,...

i have been observing databases in the cad environment since the 80s. my concerns from back then have all come true. some of my customers got wise out of harm. they don't want any databases more now.

[heda]

sounds like you are looking for a lightweight pdm-system for fc.
that by itself is a very good idea and addition to fc capability.

as a complete amateur here is what i would try to make in such case.

- find a spot in the fcstd file format where one could add nodes in the xml (nodes that reflect the table in an sqlite database)
- if nothing else one can add xml formatted text in the "text document" that already is there as part of the file format

- make a gui inside of fc, search functionality, results presentation, "make scan" etc in the location of the 3d main view, so like a text document or macro file.

- the database would be a sqlite, filled from data from a scan of folders, taking all meta-data that is in the fcstd file and putting it into the db, if one wants to hook up extra data it is just to add some gui function for adding extra data (saved as a blob in the db, like for example the thumbnails in the start-menu or similar)

- if one then wants to have a stand alone version, one can easily create a qt-gui as stand-alone, but more portable would be to have a local server and an html page as gui, picking data from the sqlite. (that html could be part of the sqlite, so with the db-file and one python file as server the pdm could be run on any computer in "offline mode")

just some basic thoughts - one could add a lot more detail for describing intended functionality.
something like this I think would be very extendable, for example one can easily set up which fields one would like to have for oneself, and that would be possible to make work

it would not really be any extra files, the "core pdm data" is in the fcstd file,
and one can modify this through fc gui or through a stand-alone gui.

it is not so much code to make a proof of concept, but probably a lot of code to make something like this error proof with checking for rights to modify data or consistency checks and boring things like that, making it "user proof".

creating some database which is governing the fcstd files sounds way too corporate for me

[onekk]

Yes, but even if Python or Qt or Coin3D were not developed anymore you have same risk, or even OCCT.

You have to find an alternative and create a sort of migrating script.

No risk at all is impossible, risk management is usual best thing you could do.

Following another set of consideration about sqlite3.

From this considerations the suggestion of using sqlite3 that is in the "python Standard Library" so not prone to be cancelled as project as if you have a FreeCAD AppImage you have it "on board".

But it depends also on numbers, for few files, I suppose under 50 there is no problem, for many files, i think that a sort of indexing engine is needed, and as said you could even implement it internally to speed up you code.

You could simply use dump to create a human readable file that will be recreated on the fly on the next run.

https://www.sqlitetutorial.net/sqlite-dump/

So internally you could simply use the power of a relational database and have "at the end" a mean to have some human readable data to recover from a crash or to create a "migration script"

Other consideration about "long term stability" are simply "reduced to zero" as your customer already use it if they use Firefox or derivative to browse the web, or simply use an Android phone or an iPhone, Windows 10 and Mac OS, see: https://sqlite.org/mostdeployed.html.

Simply they have sqlite in their pockets, without knowing.

It could be considered a "well established" and "mature technology", and in the tragic eventuality it will fail, you will be in "good company" as even the hardware on which you application will run, are using the database engine you are using.

I doubt recoll could have a better "long term stability" than Python.

Last post about this subject, let's talk about some code!

Regards

Carlo D.

[thomas-neemann]

...
I doubt recoll could have a better "long term stability" than Python.
..

if there is no longer recoll you can continue with find and grep, or with any search program.

if someone has been using this method via find and grep since the 80s, they could still use their data today. 0 euro license costs and I suspect he could still work with it in decades. I don't know anything more durable.

[onekk]

sounds like you are looking for a lightweight pdm-system for fc.

creating some database which is governing the fcstd files sounds way too corporate for me

But FreeCAD has this problem, think of having many assemblies and you want to merge them in a complex project, how do you do this sort of things?

Solutions are implemented using Links to external "documents" and this is the way @realthunder is moving (not seen in deep but I had read some discussions about this argument)

So the need for a more complex document format has already been asked.

OP will make a sort of indexing for complex project, some similar things are already done in STEP files, as the format was created having in mind interoperability between different industries in big projects (think of some big things, like Space Shuttle or other things, like ISS and so on, but even a simple car will suffice).

The point is that FCStd file is not actually easily extensible, I have had some talk with yorik some time ago (around one year ago), but the fear is that the format will became too complex to manage and too prone to malware as it will involve the need to some automation when opening documents, that could lead to some "malware injection" (or this is what i remember about).


As sqlite is part of "Python Standard Library" and FreeCAD already have it "on board" it will be not difficult to implement a sort of "index of complex document" like having a "pivot document" that refer to other intermediate documents.

The thing is already feasible as FCStd is simply a "zipped directory" so you could put many FCStd files into a sligtly different "directory tree", only problems I could see will be "file dimensions" and possible corruption problem that will arise if all the gears are not working perfectly, you have many "man hours" of work wasted in some milliseconds.

I could see the fear to have such a "leviatan file", but same thing happens when you make a backup of your HD and in business this is not seen as a "problem".

You are using "established ways" to do things, you are using one of the "most old and tested" compression algorithm and similar instruments.

However some decision has to be taken, and "reliability" is one of the factor to take into the formula, other could be:

- complexity
- external library could not be run on some machines (at least the thing should be run on Windows, MacOs and Linux)
- human readability in sense of "ability to recover from a crash" so having many files wil minimize the chances to destroy an "entire project" with a wrong "keypress" or a "short circuit".
- long term mantainability
- ease of use ? (complex things aren't easy "by definition")

Regards

Carlo D.

-

[onekk]

...
I doubt recoll could have a better "long term stability" than Python.
..

if there is no longer recoll you can continue with find and grep, or with any search program.

if someone has been using this method via find and grep since the 80s, they could still use their data today. 0 euro license costs and I suspect he could still work with it in decades. I don't know anything more durable.

Good point!

Some questions:

- you have some metadata to recover from the FCStd files?

- You have however the need of a "pivot document" to "arrange" all the relations between different parts?

So a sort of XML file, or other "stucture format" like json or other things (I'm ever referring to existing tools in "Python Standard Library") will may do the job, as you could load simply the project.py file in the "project directory" and let the script load things and open GUI windows to ask things, and then load files on FreeCAD.

(I don't like at all the FCMarco extension to fake the python content)

I have done similiar things for cabinet making an "crenelated box making", but also to automate "guitar frets creation" using FreeCAD.

A simple python file will load some data from an ini file and act accordingly, it create the model, many files to be transformed in DXF file to drive CNC machine or even many GCode files to produce real things.

But even other complex things, like creating STL files to be "3d printed", all of them driven by a py file loaded and launched using the "internal macro editor".

ex2.jpg (40.35 KiB) Viewed 1664 times

sliding.png (8.78 KiB) Viewed 1664 times

GUI-new.png (50.88 KiB) Viewed 1664 times

All made with scripting!

Regards

Carlo D.

Regards

Carlo D.[