Version Control Support in FreeCAD

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
chrisb
Veteran
Posts: 53930
Joined: Tue Mar 17, 2015 9:14 am

Re: Version Control Support in FreeCAD

Post by chrisb »

realthunder wrote: Tue Aug 06, 2019 7:18 am That's exactly what I did.
Sorry for being ignorant, I had followed only the forum.
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
User avatar
vejmarie
Posts: 713
Joined: Mon Jan 04, 2016 4:52 pm
Location: Somewhere between France, USA and Taiwan
Contact:

Re: Version Control Support in FreeCAD

Post by vejmarie »

Hi,

Can you have a look to https://forum.freecadweb.org/viewtopic.php?f=8&t=36844. Part of this work is also version control. I must admit that I didn't started to code anything regarding it, but I share the same point of view. I started by implementing a "cloud saving/restore" feature on S3 infrastructure. My current branch is able to export the flat files (non zip) into an Amazon S3 object storage infrastructure (for the one willing to self host, you can use min.io it is also working with it). This is done at bucket level (the project name is becoming the bucket name) I got also the reader.

My next step was to extend the API, to support version control at file level without using git infrastructure, but writing something which is taking into consideration the FreeCAD specific file format. When I'll be able to manage versioning at Cloud level infrastructure, I do have plan to use node-occ and OpenWebCAD that I started a couple of months ago to allow web rendering of FreeCAD files directly through a web servers for comment and informations. I know the rendering works, I need to implement the server side of the message feedback.

Displaying differences between 2 shapes is part of the work I liked to do either on the web server, or directly inside FreeCAD. It could be done by requesting the end user to identify a "common" face (to determine shape orientation), or use perhaps a special freeCAD implementation which keeps track of model changes between two file saves.

Perhaps we could work in sync on this work ? I used a client / server approach as I believe that model sharing is a must have function of the tool, and I am getting bored at managing my backups, and the fact that I live in two different places around the world ( I am not good at managing my data, other than through a cloud platform).

I am by the way really interested to see you progress on the shape comparaison thing.

vejmarie
User avatar
vejmarie
Posts: 713
Joined: Mon Jan 04, 2016 4:52 pm
Location: Somewhere between France, USA and Taiwan
Contact:

Re: Version Control Support in FreeCAD

Post by vejmarie »

I looked at your code and modifications, that is a lot of changes ;). As I did modify the same kind of files, I'll be needing to serialize my PR with yours, which is fine. When do you expect to push these modifications upstream ? (mine are in https://github.com/vejmarie/FreeCAD/tree/cloud, it is missing the Reader, but this is done, I just need to sync the branch with my machine). I can wait without many issues !
User avatar
vejmarie
Posts: 713
Joined: Mon Jan 04, 2016 4:52 pm
Location: Somewhere between France, USA and Taiwan
Contact:

Re: Version Control Support in FreeCAD

Post by vejmarie »

Hello,

Just a few questions (which might be totally irrelevant). Why did you changed things inside the whole FreeCAD code and didn't re-implemented a Writer class like FileWriter which will have isolated the specific required changes. I might be wrong but I think most of the required changes could have been done at that level (except probably the split of the Document.xml but I didn't spent too much time thinking about it), easing the code and allowing extension like:

- Default Writer / Reader -> through zipio
- FileWriter Writer / Reader -> Flat file
- CloudWriter -> Push to object storage

For the CloudWriter, I replaced the ofstream by a stringstream (ok it requires more memory), and manipulate the data inside the stringstream.

Your proposed PR is extending in some way FileWriter class by converting binary data into base64 encoding, and allowing split of Document.xml ?
I might be totally wrong, do not hesitate to point me out on what couldn't be done that way ;). I have seen some modification in FemMesh, which I didn't had to make at all for the CloudWriter and I might have missed stuff there which require some of my attention to be sure I am saving everything ;).

I am just curious, do not rewrite anything, this is just to enhance my knowledge ;)
realthunder
Veteran
Posts: 2190
Joined: Tue Jan 03, 2017 10:55 am

Re: Version Control Support in FreeCAD

Post by realthunder »

vejmarie wrote: Tue Aug 06, 2019 8:17 am I looked at your code and modifications, that is a lot of changes ;). As I did modify the same kind of files, I'll be needing to serialize my PR with yours, which is fine. When do you expect to push these modifications upstream ? (mine are in https://github.com/vejmarie/FreeCAD/tree/cloud, it is missing the Reader, but this is done, I just need to sync the branch with my machine). I can wait without many issues !
If you'd like to rebase on my patch, please wait for a while, as the current patch is developed on my personal branch which contains lots of other stuff. I'll submit the PR once my other PR got merged, because they are somewhat dependent.

That being said, I took a quick look at your code, and I advise you to reconsider adding the cloud functionality as a FC module. You can use Python if you want, or C++, which is a lot harder. The reason is that I think the cloud protocol will be too specific to be included in the core. This is also the same reason why I didn't add any VC specific code in this patch. Besides, network operation has high latency, which does not work well with FC's mostly single threaded core. My VC patch offers a new signal to inform modules when and where the document is saved locally, and what are the files involved. Python module can register to the signal by defining a class with method slotDocumentFilesSaved(), and call App.addDocumentObserver(). I can add other functions if required.

vejmarie wrote: Tue Aug 06, 2019 8:45 am Your proposed PR is extending in some way FileWriter class by converting binary data into base64 encoding, and allowing split of Document.xml ?
I might be totally wrong, do not hesitate to point me out on what couldn't be done that way . I have seen some modification in FemMesh, which I didn't had to make at all for the CloudWriter and I might have missed stuff there which require some of my attention to be sure I am saving everything
The proposed patch can be considered a (minor) refactor of FC document read and write, to make it more efficient and flexible, and most importantly easy to work with external VC software. I extended the Writer class to easily embed non XML contents. It can transform binary to base64 on the fly, and also saving arbitrary plain text as CDATA section. The modification in various properties (e.g. FemMesh you mentioned) is to support embedding property content into XML file. Every property (except PropertyFile which may include binary file) can now save text based content that can be embedded directly into the XML file. But that is optional. The user has control ranging from saving everything into (Gui)Document.xml, to splitting among objects, or even individual property files, binary or text based. The property modifications are not strictly required. I took this opportunity to tidy things a little bit. The upstream implementation generates too many small files.
Try Assembly3 with my custom build of FreeCAD at here.
And if you'd like to show your support, you can donate through patreon, liberapay, or paypal
reox
Posts: 929
Joined: Sat Aug 13, 2016 10:06 am
Contact:

Re: Version Control Support in FreeCAD

Post by reox »

Kunda1 wrote: Mon Aug 05, 2019 7:10 pm @reox would love to hear this news
reox wrote:
yes :shock: that sounds very good 8-)
User avatar
vejmarie
Posts: 713
Joined: Mon Jan 04, 2016 4:52 pm
Location: Somewhere between France, USA and Taiwan
Contact:

Re: Version Control Support in FreeCAD

Post by vejmarie »

realthunder wrote: Tue Aug 06, 2019 9:31 am That being said, I took a quick look at your code, and I advise you to reconsider adding the cloud functionality as a FC module. You can use Python if you want, or C++, which is a lot harder. The reason is that I think the cloud protocol will be too specific to be included in the core. This is also the same reason why I didn't add any VC specific code in this patch. Besides, network operation has high latency, which does not work well with FC's mostly single threaded core. My VC patch offers a new signal to inform modules when and where the document is saved locally, and what are the files involved. Python module can register to the signal by defining a class with method slotDocumentFilesSaved(), and call App.addDocumentObserver(). I can add other functions if required.
thanks for your answer. I spent a lot of time trying to figure out if I shall write a module or make it part of core. After a while, I believe that Version Control and Object Storage export shall be part of core package (aka FreeCAD application like Part module is) but not core code, and shall be implemented in a dual way (push the specific code to a module while generic infrastructure stands into the core code).

They are both extremely useful features which are currently lacking to FreeCAD and there use shall be as much transparent to the end user as we could (roughly when we kick the save button if the end user has preconfigured VC and Cloud it shall go directly to it). I know that there are many different Cloud option, and that the latency might be a challenge especially in a single threaded implementation etc .. But that will be the same with VC. There are also use case where the latency will be totally hidden (in my case I got a 10gbps mac mini which is writing at the same speed than locally to an SSD, and I know this is not most of the use case, but a small engineering team could benefit from using a shared network).

The current code is a very bad implementation, it is used to test the functionality and check if it works on the 3 O/S that we support and dig a little within the file structure of FC which was new to me. My hope is to see it introduced within the current dev version and get it fully functional for our next release. Regarding language choices, C++ is easier to me (I am aging). Might be great if an S3 class could exist somewhere instead of using curl directly, but the API is properly documented, and Swift support also a subset of S3 API. My target is to use only that subset, which makes the solution compatible with S3, Swift and min.io.

This has tend to make me implement a first PoC within the core code, as the API I used is widely available. I am always fighting with myself on what is the right size of a PR and feature add on. I am more in a process to commit the code in a small step by step approach instead of a big PR (I made big PR in the past) as to get feedback from other developers and enhance the implementation.

Regarding the implementation, I used Werner guidance who pointed me out to write a new Writer class, which worked like a charm in the end. That is also why I pointed you out to it. I am more than fine to see VC and your changes being part of core as this is key for team work.

I remember a discussion opened by Yorik a few "years" ago about the fact that using git (or text related VC) as VC was not as efficient as he was hoping (I think he made a module to prototype all of this). Following that discussion I thought that version control shall be more "Undos" oriented and keep track of changes between file saving operation. File saving could be managed by using incremental backup style stuff and rebuilt during open operation, while VC differences could be performed using a recording of operations, which could be magnify using the display of 2 versions within the same window.

A lot of work ahead, but some very interesting one ! Hope this help to understand my thoughts. I will have a closer look to your work and reuse as much as I can it.

In fact, putting the cloud push code to a module is a good thing to do. I am just writing currently PoC and will move in a step by step approach but pushing to a module code which is implementation specific. I still need to add ways to shortcut the save button within the UI, and finish the first implementation of the reader etc etc before doing so. So roughly when the work will be finished, we will have core adaptation and a new module which will connect to the core adaptation (roughly the new writer and reader class). It will take me time, and it is easier for me to implement initial PoC and feature into the core as to determine what is specific vs not, and then move into a module what must be moved there. Hope this help to understand the way I think regarding this ;) .

I will then interface my module to your code for VC. Did you envisioned to have a look to store the modifications performed at the end user level ?
User avatar
saso
Veteran
Posts: 1920
Joined: Fri May 16, 2014 1:14 pm
Contact:

Re: Version Control Support in FreeCAD

Post by saso »

vejmarie wrote: Tue Aug 06, 2019 12:01 pm My target is to use only that subset, which makes the solution compatible with S3, Swift and min.io.
Will this work also with something like PostgreSQL? My brainstorming a long time ago about this was sort of like this, user would have a DB (for example PostgreSQL) on the same machine and the option to save to files (as existing) or to the DB, this would be just for the user. In case of a multi user setup, the local DB would still be used but more like a cache, for example if the connection goes down, the user is still able to access and save to the local DB and in the background the changes get synchronized from it to the "central" DB automatically when the connection is back up. Central DB could be a simple PostgreSQL setup on a local network or something remote, hosted, HPC, S3, Swift, min.io,...
vejmarie wrote: Tue Aug 06, 2019 12:01 pm Following that discussion I thought that version control shall be more "Undos" oriented and keep track of changes between file saving operation.
If I remember correctly (and if I have understood it correctly in the first place :) ), I think I have read it somewhere that Onshape is doing it in this way, sort of storing and replaying the undo history... Whatever they are doing IMO their implementation for the VC of the model is working very nicely.
Last edited by saso on Tue Aug 06, 2019 1:56 pm, edited 6 times in total.
realthunder
Veteran
Posts: 2190
Joined: Tue Jan 03, 2017 10:55 am

Re: Version Control Support in FreeCAD

Post by realthunder »

vejmarie wrote: Tue Aug 06, 2019 12:01 pm Regarding the implementation, I used Werner guidance who pointed me out to write a new Writer class, which worked like a charm in the end.
Yes, that can surely work. But I am not convinced that you need to directly tap into FC document saving and restoring from the core in order to achieve this. Here is what I imagine how a cloud saving and restoring module can work. As I mentioned before, FC offers the document observer mechanism to inform modules of various events. A cloud module would handle the BeforeSaveDocument event to check if there is any on going cloud syncing, and warn user about it. It would also handle AfterDocumentSaved event to check the resulting files, find out the changes, and start syncing in the background. Pulling changes from the remote can be done in the background as well. I don't see why it has to be tightly coupled with the core.

Did you envisioned to have a look to store the modifications performed at the end user level ?
At the very coarse level, since the xml files can now be splitted among files, which are named using the object's internal name. So finding out which objects are changed is trivial. Finding out which properties in the objects are changed requires a little parsing, but not difficult either, so we can easily build GUI showing property diffs. But ultimately, we'll need the shape diff tools, which will require significantly more work. Other workbench may provide their own diff tools, e.g. spreadsheet.
Try Assembly3 with my custom build of FreeCAD at here.
And if you'd like to show your support, you can donate through patreon, liberapay, or paypal
User avatar
vejmarie
Posts: 713
Joined: Mon Jan 04, 2016 4:52 pm
Location: Somewhere between France, USA and Taiwan
Contact:

Re: Version Control Support in FreeCAD

Post by vejmarie »

saso wrote: Tue Aug 06, 2019 1:26 pm
Will this work also with something like PostgreSQL? My brainstorming a long time ago about this was sort of like this, user would have a DB (for example PostgreSQL) on the same machine and the option to save to files (as existing) or to the DB, this would be just for the user. In case of a multi user setup, the local DB would still be used but more like a cache, for example if the connection goes down, the user is still able to access and save to the local DB and in the background the changes get synchronized from it to the "central" DB automatically when the connection is back up. Central DB could be a simple PostgreSQL setup on a local network or something remote, hosted, HPC, S3, Swift, min.io,...

....

If I remember correctly (and if I have understood it correctly in the first place :) ), I think I have read it somewhere that Onshape is doing it in this way, sort of storing and replaying the undo history... Whatever they are doing IMO their implementation for the VC of the model is working very nicely.
It could work with a DB, or at least I could implement it with any kind of output in the end. I think the undo might be a really good approach. I used onShape at the early stage and didn't checked how they managed versioning, but keep that stack seems to me a good approach as we could replay stuff. I was thinking about using it for tutorial at the early stage of the idea.
Post Reply