A blog about about Damien Merenne

Managing projects environments with Direnv and Micromamba 🔗

Feb 11, 2025, filled in development, shell, tech

Managing project development and build environments nowadays can be complicated, between different operating systems, different and of course incompatible tool versions, the different development languages and platforms. After a lot of iteration, trying to use some or a mix of Docker, Conan, Pip, Homebrew, Nix or native package managers, I think we might have found a solution that's working for us. We set up on Direnv and Micromamba to manage the basic tooling and bootstrap our environments.

Direnv allows us to configure the shell environment per directory. When entering a directory containing a .envrc file, it will load it, export some variables, setup virtual environments for Python or Ruby if needed and do custom bootstrapping tasks like downloading vendor binary tools.

We combine Direnv with Micromamba to download tools inside the project's directory, without cluttering the user's system. Everything is self-contained in the project directory. It can download and install multiple Python versions, CMake, the Java virtual machine and lot's of other tools. We configure Direnv so that it knows about the Micromamba environment. It will bootstrap the environment or update it automatically if it changes. Here is a typical example of a Micromamba environment definition file and a matching Direnv .envrc.

            
name: my-env
channels:
- conda-forge
dependencies:
- just
- python 3.12.*
- libprotobuf 3.20.*
- terraform
- awscliv2
- pip:
  - -r requirements.txt
  - -r requirements-dev.txt
      
          

            
# Bootstrap micromamba direnv support
source_url https://gitlab.com/-/snippets/4803021/raw/main/direnv-micromamba.sh sha256-v9hWwL8HsCOdN+f8HBSijmXbGTgpPAxQJcUgPaYku3A=

# Setup and activate the micromamba environment.
use micromamba my-environment my-env.txt

# Setup some variable
export AWS_PROFILE=my-aws-profile

# Use a per project bash history file
export HISTFILE=$PWD/.bash_history

# Inject some path into the Python environment
path_add PYTHONPATH "${PWD}"

# Allow user to customize the environment using a local direnv file
source_env_if_exists .envrc.local

# Bootstrap project
if [ ! -d "my-dir" ]; then
  bootstrap.sh
fi
      
          

The first lines tell Direnv to download and check the sha256 of the Micromamba integration script that's used next. The use micromamba lines calls Micromamba to setup the environment my-environment that is defined in my-env.txt in the .micromamba-my-environment directory and activates it. It will also watch the environment definition file so that the environment is rebuilt if the file changes. As you can see, Micromamba supports installing pip packages in it's own environment. While Direnv supports setting up a Python virtual environment, it doesn't support running pip out of the box.

Here is the Direnv/Micromamba integration code snippet.

            
# Define `use micromamba` direnv command that installs micromamba if needed,
# bootstrap and activates an environment given micromamba environment definition
# files.
use_micromamba() {
    ENV="$1"
    shift
    FILES="${@:-${ENV}.env.yml}"
    PREFIX="${PWD}/.micromamba-${ENV}"
    SHA="${PREFIX}/environment.sha256sum"
    BINDIR="${PWD}/.bin"
    PATH_add "${PWD}/.bin"

    # Install micromamba if needed
    if ! which micromamba >/dev/null; then
        echo "direnv: installing micromamba in ${PWD}/.bin"
        mkdir -p "${BINDIR}"
        INSTALL_MICROMAMBA="$(fetchurl https://raw.githubusercontent.com/mamba-org/micromamba-releases/main/install.sh sha256-7+/a3LEvnYPcDGoUNWrrtzenAxFuc6xMTQpRfdZiBTg=)"
        INIT_YES=no BIN_FOLDER="${BINDIR}" bash < "${INSTALL_MICROMAMBA}" >/dev/null 2>&1
        PATH="${PATH}"
        if ! which micromamba >/dev/null; then
            echo could not find micromamba, ensure $BIN_FOLDER is in your PATH
            exit 1
        fi
    fi

    # Hook micromamba into the shell
    eval "$(micromamba shell hook --shell posix)"

    # Regenerate micromamba environment when the environment definitions are changed.
    watch_file ${FILES}

    # If already installed, check if the environment yaml definition file has been updated
    if [ ! -e "${SHA}" ] || ! sha256sum --quiet -c "${SHA}" >/dev/null 2>&1; then
        echo "direnv: micromamba $ENV environment updated, recreating environment"
        rm -fr "${PREFIX}"
    fi

    # Create the environment if needed
    if [ ! -d ${PREFIX} ]; then
        echo "direnv: creating micromamba ${ENV} environment, it might take a while"
        micromamba --quiet create --prefix "${PREFIX}" $(for i in $FILES; do echo --file $i; done) --yes || return 1
        sha256sum "${FILES}" > "${SHA}"
    fi

    # Activate micromamba environment
    micromamba activate "${PREFIX}" || return 1
}
      
          

Happy new year! 🔗

Jan 1, 2025, filled in life

May 2025 bring you the best!

Someone else has your problem too 🔗

Dec 31, 2024, filled in development, tech

TLDR

I was perusing through the excellent Sacha Chua Emacs new's where she links to this article. @naiquevin@fosstodon.org talks about technical debt in Emacs configuration. I saw there a pattern I've seen a few times. A software user stumbles upon a problem, looks it up on the internet and copy/paste a random solution.

What I now know is that for most of the problems I'm hitting when using a tool, there's almost certainly someone out there with the same problem. And most of the time, it's a lot of people. So what I'm doing these days is to always look first at the project's issue tracker before searching the web for a random solution. Sometimes there's the actual solution in there, sometimes there's a workaround, but there's almost always a reference to my problem. It can also be a WONTFIX but then with some explanations of why that way is not a good way. But at least, you get the actual expected way of doing things and you can start to understand how the software is supposed to be used.

I don't know how many users of a piece of software there is but we're billions here on earth so there's always someone. I implemented a Home Assistant integration for a niche domotics system developed by a small local company that's out of business today. I thought I would be alone in needing this combination. Fast forward 3 years later, and I stumble upon a comment on the Home Assistant forum's from a user looking for the same thing. Even at my small local scale, using niche systems, there were people with the same needs as me.

Now I'm also a developer so another thing I learned is that even if it's a missing feature or bug, if it's open source, use the code Luke. At first it can be a bit difficult to understand someone else's code but you get used to it. You get used to understand how software projects are made, and guess what? There's not that many ways. Of course there are multiple languages, architectures, build systems and combinations of those, but at the end of the day, it's always the same. Even if you don't know the language, take a stab at it. Look at the code. Nowadays, the various forges makes it so easy you don't even have to check out the code locally.

The project I'm working on these days involve using a nRF53 embedded processor. I like to use a Blackmagic probe to do my debugging on embedded stuff. It's open source, open hardware and, I think, much simpler to use than the proprietary JLink stuff. Unfortunately, that processor was not supported. So I hopped on their project's development page, and lo, there's an open issue for it. I never messed with the jtag protocol and such low-level stuff, so my first reflex was to simply subscribe to the issue and propose my help but I didn't felt up to the task of implementing it myself. And then I went to the code, looked how the other similar processor are implemented. It's about 500 lines of C which at first seemed a bit esoteric. I just copy-pasted it, looked up the code for my processor and started to edit bits by bits. And I finally got it working and learned a lot in the process.

There's almost always somebody who has or had your problem. There's almost always a sanctioned solution for it. Looking up the official documentation and bug tracker is the first thing to do before looking random stuff on the web. If you're a developer, look at the code. At the very least you'll learn something, you might even discover you're able to contribute a fix.

Image based / atomic Linux distributions experiments 🔗

Oct 23, 2024, filled in linux, vanillaos, tech

I've read about image based Linux a while ago. It seems a good idea at first sight: Android has been doing that for years and its working great. On Android, all the applications are basically sand-boxed and application updates are managed independently from system updates. With Flathub being now more mature, I thought about giving atomic Linux a try.

A few month ago, I went ahead and installed Fedora Silverblue. Install went find and I could setup a reasonably complete environment easilyh I never managed to get my Yubikey working. Also, being a Debian user for more than twenty years, I felt a bit lost in that Fedora based distribution. So I abandoned the atomic distribution thing and went back to a classic Debian based distribution.

Fast-forward to today where I'm writing this on a VanillaOS installation. It's atomic and Debian based, using snapshot of Debian Sid as basis. I'm still getting used to it but it feels way more like home in here. I find The documentation is a bit sparse, the handbooks are helping but they lack a bit in depth. I needed to learn how to add packages to the main system overlay. Also, everything is became simpler once I understood the VSO shell concept. It's basically a Distrobox based container with some helper to tie them into the environment. The shell is started in that container by default. You can use host-shell in case you need to run some command in the system's shell.

I installed some applications as flatpaks (some of them are setup by the VanillaOS installer at install time): Firefox, Slack, Zoom, LibreOffice... Emacs wouldn't work so well as a flatpak as it needs to interact with lots of external tools so I installed it in the VSO shell, along with the tools I use for development. For day to day use, it feels more or less like Debian until you need to mess with the lower level system. We'll see if I manage to stay around.

Starting to write a bit again 🔗

Oct 23, 2024, filled in life

So after years without writing anything, I feel like I should be sharing a bit more about what I'm doing. Hopefully, some people will find interesting stuff here, probably around technology, development, music and Emacs, who knows?

Anyway, here I am... and if you read this, welcome!