Package Management Hell - Compute Modalities

Got a new laptop? Ready to become a data labeler? Link to Andrej's post. Ready to do data science work? Curate data for our AI overlords and become a bootloader for them? 🤖

You need Python!

⚠️ Welcome to the Package Management Hellscape!

Dependency hell is a real thing! 🪦

The Backend Wars: A Brief History

In the beginning, there was chaos. Package managers sprouted like mushrooms after rain:

pip / PyPI - The OG Python package installer
uv - The new Rust-powered speed demon
conda / mamba - From the Prefix.dev team's previous work
poetry, pipenv, pdm - The poetry of dependency management
Other languages joined the party: npm , yarn , bun , cargo, go mod
Even K8s has helm for package management!
Recent gRPC data tools has buf cli for proto file validation
And N linters like..

Hell, because which APIs to use? Which ecosystems to support? How to avoid the dreaded dependency conflicts?

📝 Note: I'll write about other language package managers someday. This is my current workflow which might help people building in the AI/ML ecosystem.

Enter Pixi: The Rust-Powered Savior

Why Pixi? Let's check the street cred:

🦀 Rust backend for blazing fast dependency resolution
🪟 Great Windows ecosystem support (and Linux too!)
Some VP at Microsoft is quoting it
Big tech and frontier startups are using it
The creator of Swift/LLVM is backing it
Mojo/Modular uses Pixi on their backend

"Remember: generating code (via vibes or punch card machines) is easy. The maintainer's job (packaging new features, bug fixes into clean easy-to-use pipelines and making the right decisions with quality discussions... without getting feelings hurt) for a long time is hard!" - Amdahl's Law of Package Management

Linus would definitely agree on this! 😂

The Installation Journey

Phase 1: The Horror Show 👀

v26i@fireworker-v26i-mac0 ~ % which python3
/usr/bin/python3
v26i@fireworker-v26i-mac0 ~ % which pip3
/usr/bin/pip3
v26i@fireworker-v26i-mac0 ~ % which pip
pip not found
v26i@fireworker-v26i-mac0 ~ % python3 --version
Python 3.9.6

System Python 3.9.6? In 2025? Time to fix this!

Installing Pixi

v26i@fireworker-v26i-mac0 ~ % curl -fsSL https://pixi.sh/install.sh | bash

This script will automatically download and install Pixi (latest) for you.
Getting it from this url: https://github.com/prefix-dev/pixi/releases/latest/download/pixi-aarch64-apple-darwin.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 18.2M  100 18.2M    0     0  12.2M      0  0:00:01  0:00:01 --:--:-- 49.0M
The 'pixi' binary is installed into '/Users/v26i/.pixi/bin'
Updating '/Users/v26i/.zshrc'
Please restart or source your shell.

v26i@fireworker-v26i-mac0 ~ % source ~/.zshrc

🚨 CRITICAL: Order of PATH is important! /Users/v26i/.pixi/bin being at the beginning is critical!

v26i@fireworker-v26i-mac0 ~ % echo $PATH
/Users/v26i/.pixi/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:...

Installing Python 3.10.17

Why 3.10.17? Released April 8, 2025 - picked for CUDA version parity/interop from Python macOS downloads.

v26i@fireworker-v26i-mac0 ~ % pixi global install python=3.10.17 pip
Global environments as specified in '/Users/v26i/.pixi/manifests/pixi-global.toml'
├── pip: 25.1.1 (installed)
│   └─ exposes: pip, pip3
└── python: 3.10.17 (installed)
    └─ exposes: 2to3, 2to3-3.10, idle3, idle3.10, pydoc, pydoc3, pydoc3.10, 
                python, python3, python3-config, python3.1, python3.10, python3.10-config

The Plot Twist 😱

v26i@fireworker-v26i-mac0 ~ % python --version
Python 3.10.17
v26i@fireworker-v26i-mac0 ~ % python3 --version
Python 3.9.6  👀 insert scary gif!

Wait, what?! Let's investigate:

v26i@fireworker-v26i-mac0 ~ % ls -larth /Users/v26i/.pixi/bin/python*
-rwxr-xr-x@ 16 v26i  staff   427K May  8 17:38 /Users/v26i/.pixi/bin/python3.10
-rwxr-xr-x@ 16 v26i  staff   427K May  8 17:38 /Users/v26i/.pixi/bin/python3
-rwxr-xr-x@ 16 v26i  staff   427K May  8 17:38 /Users/v26i/.pixi/bin/python

v26i@fireworker-v26i-mac0 ~ % /Users/v26i/.pixi/bin/python3 --version
Python 3.10.17

After another source ~/.zshrc:

v26i@fireworker-v26i-mac0 ~ % python3 --version
Python 3.10.17  ✅ Now good!

🎉 Leaving system Python and pip alone! YAY!

The Weird Part

Even though we have Python 3.10.17, pip is running on Python 3.13?!

v26i@fireworker-v26i-mac0 ~ % python3 --version
Python 3.10.17
v26i@fireworker-v26i-mac0 ~ % pip --version
pip 25.1.1 from /Users/v26i/.pixi/envs/pip/lib/python3.13/site-packages/pip (python 3.13)

This is because pip is installed as a separate environment in Pixi!

Phase 2: Project-Local Environment Bootstrapping

Working with existing projects

If your project already has requirements.txt:

pixi add --pypi -r requirements.txt

For projects with pyproject.toml dynamic dependencies - best to decouple if builds are taking too long, as we don't want package sizes to grow large.

Pixi Commands Quick Reference

pixi add python=3.13.3 - Add exact Python version
pixi install - Only needed after manual pixi.toml edits or git pull
pixi add ... - Edits TOML and installs in one step
pixi remove package-name - Remove a package

Note: pixi install is only needed when you've manually tweaked pixi.toml (or pulled in a fresh repo) and want to sync your lockfile/env to those edits.

TODO

Add ANSI color capability to shell for readability of CRITICAL, WARNING words etc like mobaexterm
Add discussion/comments section using X SSO or GitHub auth
Explain what the .zshrc function does internally
Add note about default directory paths (~home)
Write about rattler-build or packaging examples (new article?)
Unix timestamp epoch redirect - link to that ChatGPT rabbit hole about 32-bit vs 64-bit timespan
Try to include info about ternary computers (bit duality doesn't change I guess)

If any link returns 404, try using Wayback Machine archives!

Package Management Hell 🔥