conda / mamba based extension manager

This subforum is specifically to discuss packaging issues on different platforms (windows, mac, linux), and using different packaging systems (conda, etc...)
Post Reply
wolfv
Posts: 2
Joined: Mon Apr 26, 2021 3:19 pm

conda / mamba based extension manager

Post by wolfv »

Hi all,

I am the author of a package manager called "mamba". It's very similar to "conda" and works with the same package format (but mamba is written in C++, and using a different solver backend). I think a few of you are aware of conda, since there are FreeCAD builds on conda-forge and as far as I understand, some of the final downloadable builds are derived from those.

Mamba is also cross-platform, cross-language (incl. C/C++) package manager and runs very well on Linux, OS X and Windows. We also have a "libmamba" that is just a library interface to mamba.

The idea that came up was: could "libmamba" be used as "extension manager" for FreeCAD? So that FreeCAD plugins are in fact "conda"-packages that can be installed through libmamba. This would have some nice benefits:

- proper dependency resolving, and allowing inter-dependencies between extensions
- support for native libraries (C/C++/Rust/Go dependencies)
- could inter-op / build on top of conda-forge
- FreeCAD could have it's own "channel" (repository), either as static file server, on Anaconda.org or self-hosted "quetz" instance (https://github.com/mamba-org/quetz)

This would be somewhat new territory for mamba/libmamba, but it would be very interesting for me to see if we could work something out! So far we have used "libmamba" only in the "rhumba" extension to R that brings package manager capabilities based on conda packages to the R language.

Looking forward to a discussion here!
User avatar
looo
Veteran
Posts: 3941
Joined: Mon Nov 11, 2013 5:29 pm

Re: conda / mamba based extension manager

Post by looo »

wolfv wrote: Tue Apr 27, 2021 6:48 am The idea that came up was: could "libmamba" be used as "extension manager" for FreeCAD? So that FreeCAD plugins are in fact "conda"-packages that can be installed through libmamba.
Hi wolfv and welcome to the freecad-forum!

I would like to help with getting mamba included in the conda-based bundles (especially the appimage). The main advantage I see from doing so are smaller bundles which contain only the core of freecad + mamba as dependency solver. This way we do not have to add things like (scipy, sympy, ...)

With mamba it should be possible to install compiled extensions and guarantee compatibility with freecad (examples currently already available from conda-forge are Assembly3 or openglider).

But there are for sure some difficulties which have to be solved. So I simple start with some questions:
1. Can we tell mamba where to install additional packages (best to do this local (user) and specific for the freecad version (0.19, 0.20,...)?
2. Is there a way to avoid updating libraries that are already bundled within the appimage and use these libraries as constraints for additional libraries?
3. Can we use mamba in the github-action to create the bundles (speeding up the process to create the environments)?


I also created an appimage that only includes freecad with dependencies + mamba and libmamba. I guess this is something we can experiment with. https://github.com/FreeCAD/FreeCAD-Bund ... /tag/mamba

Code: Select all

import os
from mamba import mamba_api
mamba_api.Context().conda_prefix = os.environ["PREFIX"]
wolfv
Posts: 2
Joined: Mon Apr 26, 2021 3:19 pm

Re: conda / mamba based extension manager

Post by wolfv »

Hi looo,

thanks for the warm welcome!

1. Yes, installation location is completely customizable. One thing that is usually required is a "root prefix" which contains a "pkgs dir". On conda, that is usually something like `~/miniconda`, and the `~/miniconda/pkgs`. This is used to have a cache of packages, with the final installation being "hard-links" that point to the files in this cache. Since conda / mamba manage virtual environments, this can save disk-space since we can re-use the same package files across environments.
I am just thinking out loud: for the FreeCAD use-case we could have a FreeCAD-version-independent cache in `~/.freecad/plugin_pkgs` with environments unders `~/.freecad/freecad-0.19/plugins/...` as the installation location for the plugins.

It would also be possible to _not_ use this caching mechanism and always copy files into the installation directory. That is just one config variable that would need to be changed.

2. yeah, that is a good & interesting question. We already have a concept of virtual (system) packages like `__glibc`, or `__osx` to constrain on libraries / versions that are not managed by mamba / conda. For this use-case we'd probably want to extend that to freecad and other "fixed" libraries by the freecad installation. We don't have a straight-forward mechanism for that yet, but I think we can just drop a non-existent `freecad-constraints` package into the installation prefix and give it the constraints for other libraries as they are installed. But there might be some additional work to do! I'd be happy to tackle that.

3. Do you mean use mamba in general, or for this specific use-case? :) We could create pre-solved environments for the plugins using something like `conda-lock` (which can also use mamba & micromamba). If you're referring to the call to `conda create ...` in the current github action, then you could just substitute that with `mamba create` and I'd expect that it would work pretty much the same way as it does currently (sometimes there are slight solver differences).

I need to dive deeper into how the FreeCAD plugins work ... looking forward to this!
User avatar
sgrogan
Veteran
Posts: 6499
Joined: Wed Oct 22, 2014 5:02 pm

Re: conda / mamba based extension manager

Post by sgrogan »

wolfv wrote: Tue Apr 27, 2021 6:48 am I am the author of a package manager called "mamba".
looo wrote: Wed Apr 28, 2021 12:23 pm I would like to help with getting mamba included in the conda-based bundles (especially the appimage).
looo wrote: Wed Apr 28, 2021 12:23 pm 2. Is there a way to avoid updating libraries that are already bundled within the appimage and use these libraries as constraints for additional libraries?
wolfv wrote: Fri Apr 30, 2021 6:17 am 2. yeah, that is a good & interesting question. We already have a concept of virtual (system) packages like `__glibc`, or `__osx` to constrain on libraries / versions that are not managed by mamba / conda. For this use-case we'd probably want to extend that to freecad and other "fixed" libraries by the freecad installation. We don't have a straight-forward mechanism for that yet, but I think we can just drop a non-existent `freecad-constraints` package into the installation prefix and give it the constraints for other libraries as they are installed. But there might be some additional work to do! I'd be happy to tackle that.
wolfv wrote: Fri Apr 30, 2021 6:17 am I am just thinking out loud: for the FreeCAD use-case we could have a FreeCAD-version-independent cache in `~/.freecad/plugin_pkgs` with environments unders `~/.freecad/freecad-0.19/plugins/...` as the installation location for the plugins.
I'm very interested in this!
"fight the good fight"
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Re: conda / mamba based extension manager

Post by Kunda1 »

sgrogan wrote: Fri Apr 30, 2021 5:10 pm
wolfv wrote: Fri Apr 30, 2021 6:17 am I am just thinking out loud: for the FreeCAD use-case we could have a FreeCAD-version-independent cache in `~/.freecad/plugin_pkgs` with environments unders `~/.freecad/freecad-0.19/plugins/...` as the installation location for the plugins.
I'm very interested in this!
This thread should be x-posted with chennes's efforts to standardie the addons with metadata (https://forum.freecadweb.org/viewtopic.php?f=10&t=57448)
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
User avatar
chennes
Veteran
Posts: 3878
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: conda / mamba based extension manager

Post by chennes »

Thanks for the ping, @Kunda1 -- I've done some work on dependency resolution from within FreeCAD as part of my work developing the package metadata standard, but it's obviously a work-in-progress, and doesn't address any of the other bullet points that this does.

ETA: I should point out that when I say "dependency resolution" in this case, I am referring only to one FreeCAD package depending on another -- not the deeper sort of system-level resolution that this type of manager would enable.
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
User avatar
mnesarco
Posts: 446
Joined: Thu Mar 26, 2020 8:52 pm

Re: conda / mamba based extension manager

Post by mnesarco »

Hi Friends,

I have just discovered this thread. I am very interested in this approach. Dependency management is hard especially with native c++/c/rust/.. dependencies.
Post Reply