Ticket #5559 - Splitting FreeCAD up in to smaller packages

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Ticket #5559 - Splitting FreeCAD up in to smaller packages

Post by Kunda1 »

Summary:
Due to the explosive growth of FreeCAD development there is a growing necessity to designate what Workbenches should ship by default and what Workbenches could be downloaded separately. Beside that, there is a growing addon repository with a newly implemented Addon Manager (bundled by default in to v0.17.9944) which is encountering a lack of pre-requisite dependency checking (i.e. 3rd party python libs and applications).

Background:
issue #857 titled "Split package into smaller ones." opened by @wmayer references a thread https://forum.freecadweb.org/viewtopic. ... 132#p24310
wmayer wrote:Maybe we should take into account to offer FreeCAD split in several packages instead of a monolithic one. So, we have e.g. freecad-core with the core system, freecad-part, ..., freecad-ship. Then it's no problem to define a hard dependency for a certain module. This way the user has more control over which modules he actually wants to use and thus reduces the amount of uselessly installed packages to a minimum.
@yorik also sees a need for this to be done; recently in a thread discussing pre-requisite checking for dependencies in the new Addon Mamager
yorik wrote:Yes, we could think about something like that... It could be something simple, such as adding some "PREREQUISITE.md" file to the repo, that would contain a list of other workbenches this one depends on, that the addons manager could read, and install the needed ones if needed...
Questions:
  • Do we split every Workbench up in to its own package or designate what WBs constitute a FreeCAD-core package that will ship as the default and just split the non-core WBs up?
  • Should we map what dependency each WB has on other WBs ?
  • Some WBs have 3rd party deps like OpenFoam or GDAL etc... how should FC proceed ?
Discussion:
Re: 3rd party dependencies
The idea is to initially just report to the user what 3rd party dep they need to install before a specific WB can be installed. Perhaps eventually we could use something like paver to install 3rd party python libs?

Current 0.17 WBs
Image
Last edited by Kunda1 on Mon May 09, 2022 8:18 pm, edited 2 times in total.
Reason: Added GH ticket number to thread title
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Re: Splitting FreeCAD up in to smaller packages

Post by Kunda1 »

Mods, not sure I posted to correct subforum. Please move if necessary. Thanks!
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
User avatar
kkremitzki
Veteran
Posts: 2511
Joined: Thu Mar 03, 2016 9:52 pm
Location: Illinois

Re: Splitting FreeCAD up in to smaller packages

Post by kkremitzki »

Kunda1 wrote: Sun May 28, 2017 12:08 pm
Questions:
  • Do we split every Workbench up in to its own package or designate what WBs constitute a FreeCAD-core package that will ship as the default and just split the non-core WBs up?
If something like this were to be done, here's a few groupings I'd think would make sense at a package level:
  • μFreeCAD - Start WB + only a very lightweight core including Sketcher, maybe Part, Draft. Targeted towards things like low-powered PCs, Raspberry Pi.
  • FreeCAD-Core - everything in μFreeCAD plus compound workbenches like Arch, PartDesign, etc.
  • FreeCAD-Engineering - requires FreeCAD-Core, includes dependency-heavy things like FEM, CFD
  • FreeCAD-Extras - requires FreeCAD-Core, includes other workbenches that are not core like Robot, Ship, Raytracing,
Like my FreeCAD work? I'd appreciate any level of support via Patreon, Liberapay, or PayPal! Read more about what I do at my blog.
chrisb
Veteran
Posts: 53919
Joined: Tue Mar 17, 2015 9:14 am

Re: Splitting FreeCAD up in to smaller packages

Post by chrisb »

For me the most basic FreeCAD would contain PartDesign and Part. Is this already too heavy?

I think it is important to have a package, which can be easily identified and used by newbies. It is probably what you call "Core" or at least it is similar, I would call it "Standard".
A Sketcher Lecture with in-depth information is available in English, auf Deutsch, en français, en español.
Jee-Bee
Veteran
Posts: 2566
Joined: Tue Jun 16, 2015 10:32 am
Location: Netherlands

Re: Splitting FreeCAD up in to smaller packages

Post by Jee-Bee »

what core is depends a lot on type of work done with FreeCAD.
For me as mechanical Engineer is idd Part; Part Design; Techdraw and future Assembly.

But for some other is Arch core...

Point what stuck me and that stuck me either with self compiling is that i in this case miss some (relatively) important features like Open SCAD (parametric) redefine features. which mean that still i need some other workbenches just for one or two features.
I'm sure that i'm not the only with this.
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: Splitting FreeCAD up in to smaller packages

Post by looo »

I think for development it would make sense to have a small core-system. Maybe without workbenches... But splitting up everything into parts, mostlikely will have the need of a good package-manager to solve all the version troubles. As long as freecad doesn't have a consistent tool for delivering packages, I would not start with splitting the sources.

But I am fine with outsourcing some pure python workbenches like ship, arch?, image?, plot.
triplus
Veteran
Posts: 9471
Joined: Mon Dec 12, 2011 4:45 pm

Re: Splitting FreeCAD up in to smaller packages

Post by triplus »

Workbenches are organizational structure. FreeCAD commands are what users use to get the job done. Not workbenches. Removing workbench from upstream code in my opinion isn't needed if there aren't any real technical/practical reasons to do that.

As for workbench list. It can be set at compile time or by using:

Tools -> Customize... -> tab Workbenches

When it comes to quality and useful FreeCAD upstream command contributions. Hopefully we will see more of that in the future.
abdullah
Veteran
Posts: 4935
Joined: Sun May 04, 2014 3:16 pm
Contact:

Re: Splitting FreeCAD up in to smaller packages

Post by abdullah »

I think there are two separate "issues" to tackle Development packages or units and distribution packages. I note that those problems are separated and do not necessarily require a same solution.

An "explosive growth of FreeCAD development" is not a problem, not even a feature is just a wonder, pure addiction, I want more of an explosive growth of FreeCAD. Please, give me more. I just can't take enough of it.

I think we have to identify which are the perceived drawbacks of such a growth, for example:

1. We are flooding core developers (e.g. those who review and merge) with pull request and the situation is becoming unhandable.

There are a bunch of solutions that do not require splitting. Splitting, per se, is not the solution (although may force decisions that are the solution, but not a requirement). There, the solution can come from enlarging the core team or delegating the work on some workbenches.

Again, I want to remark, this does not require splitting the distribution packages.

2. We want to decrease the size of the minimum distribution package.

If we want to solve this, the splitting is necessary to decrease the footprint.

But then I think we have to still provide a standardized solution to avoid having problems answering the question "What is FreeCAD?" (apart from a lively awesome community). There is a very high risk that then: "Yep, you know, I did it with FreeCAD. Ok, let's install and open, wait, I do not have that Workbench, addonManager, nice, let's install, oh, oh, what did you use Windows? This workbench hangs in Ubuntu... We can not open it here." or the similar situation "FreeCAD is signed by developers and I can use it in my work environment complying with the policy, but I can not use that workbench you used because it is not.". "Man, there is no Internet connection over here, we are stuck".

I am not sure what is the solution for this. But I think it is very important that a new user/new install has everything (that is considered to be FreeCAD) by default for reasons of standardisation of the tool.

Other arguments, other approaches

Maybe one approach is to provide a "superminimum almost useless" FC that when opened invites the user either to "download it all" or customize at that point. A new user most probably will "download it all" (one could argue that "download it all" is the standardized tool that has all). Then there may be some use cases for a customizing at that point (I do not have HD space, I am using a system with tiny resources, I do not have the time now to download everything because I need to finish my actual job).

I would like to point out that the tool standardisation problem, we have it today with some powerful WB that are developed separately, tested separatedly, distributed separatedly. Do a survey to see how many people know, Assembly 2 WB, or Lattice WB. From those who know it by name, ask if they have it installed. Now, am I proposing to merge everything together in an indivisible bundle for distribution? That is as bad a solution as an unconditional splitting (although it seems it is widely used commercially, see discussion about the 6 GB download of VS 2013 in the Extension thread). What I am trying to say is that we need to be aware of the problems that we are facing, we will face, and provide appropriate solutions so as not to create new problems.

Now I can also play the counterargument. In FreeCAD there is a WB called Ship WB that I have to download everytime, every update, I have never used and probably I never will. One pro probably low merit argument is that at least I know that it exists (if I ever come accross having to do something having to do with Ships). In any case:

Could we provide this in an alternative form (for example, be part of the download all, and if I go to customize, inform me about what it is, and allow me to select it or not)? If we bring the current standardisation issue at hand, it may even have three options, "Download full FC", "Download full FC and all 3rd party WBs", "Let me customize".

I do not intend to provide a solution. I hope I created some "problems to be solved" in your heads.

Some potential risks of splitting:
1. Risk derived from non-uniform licensing, late detection of unacceptable licenses, problems of distribution.
2. Security risks derived from installing from different sources, different repositories, signed content, trust, company policy, linux distribution's policy.
3. Non-standard exposure of users to FC (if everybody has a very different stock tool, then one could argue there are different tools).
4. Isolation of developers in WBs forming closed groups, risks associated to departing from well established programming practices or taking diverging approaches.
5. (Lack of/insufficient) exposure of developers to FC's codebase.
6. Higher tendency towards repetition of code/functionlaity in separate WBs.
6. Increase in complexity (for the user, for the developer).

Bottom line, I love the "growth". Hopefully we will come with a solution that provides the ingredients for more exposure to FC, higher codebase, more growth.
User avatar
microelly2
Veteran
Posts: 4688
Joined: Tue Nov 12, 2013 4:06 pm
Contact:

Re: Splitting FreeCAD up in to smaller packages

Post by microelly2 »

There are large solutions with a lot of extensions and they work.
My example is typo3 - a content management sytem. I work with it for many years and there is a endless list of extensions.
Sometimes there is a notification that there was found a security leak in an extension, so I can deactivate it and wait for the fix.

There should be a way to check the quality of common packages .
Still no idea how this can work. :roll:

Projects should have a state (alpha, beta, stable). So users have a first indicator of the quality of a ackage.

There can be done some improvements for the 3rd party extensions
- dependency checks
- platform checks

I will do this for my workbenches next (gdal, cv2, ...). I think having a method in the Init.py already can help.


A last remark: Do not split the core FreeCAD. The workbench list is historical and (often) well documented.
galou_breizh
Posts: 436
Joined: Wed Sep 15, 2010 9:38 am

Re: Splitting FreeCAD up in to smaller packages

Post by galou_breizh »

I was thinking about this lately and though I would try one of the existing solutions for managing modular projects, namely catkin.

catkin is the build system used by ROS (Robot Operating System) but it is supposed to be ROS independent. It is also used as test system and distribution system. Build and runtime (i.e. OS packages) dependencies are managed, both ROS dependencies and system dependencies. It was designed primarily for Linux which offers package managers with dependency/version management. For it to work on other operating systems, these have to support package management, but other solutions are possible if this is not the case. It's very easy in catkin to create a new module, you use the provided template, set the dependencies, modify the module's CMakeLists.txt and you can compile. There is also the possibility to test FreeCAD and the modules without installing.

From my point of view, the advantage of splitting is that you don't need to play around cmake options if, as a developer, you want to deactivate a broken "3rd party" module. In your development workspace, you would have only the modules you're interested in or you cannot easily deactivate the compiling of a specific module (it's just a matter of adding a file with a specific name into the concerned directory).

Another advantage of splitting is that each developer could choose his/her license because different packages are distributed separately.

There is no extra difficulty for the user because some metapackage can be created, so that by installing, let's say, freecad-full, all the modules listed in this metapackage would be installed as dependent OS packages. If the user wants to install a new package, this would be easily done through the OS package manager (I think catkin allows installation relocation in order to avoid requesting administrator rights).

However, I may be a bit soon to do this splitting as the number of packages is still not that big. This is something that needs to be though thoroughly.

Gaël
Post Reply