Thanks to this article I discovered the tool
PDM, which of course I wanted to try as soon as possible. During the project initialization, I stumbled upon a few issues.
They made me realize that I don’t know (or know naively) many topics around binaries on the OS and Python package
management. That’s why I decided to publish my notes, or way of understanding, for Python on MacOS, PEP582, and PDM
in this blog post.
OS-depth level
Directory definition
In software development, everything has its purpose, and its destination to fulfill (at least should have). The matter does
not differ for directories in our OS. The one created for our needs like Music, Videos or Documents.
And the others, created by OS by default during installment. Of course their purposes may have changed
for the time they are around, but let’s take a look what were their initial targets.
$ man hier
[...]
/usr/ contains the majority of user utilities and applications
bin/ common utilities, programming tools, and applications
local/ executables, libraries, etc. not included by the basic operating system
[...]
So now we know, that /usr/bin/ should contain default binaries
(shipped by OS) and /usr/local should contain applications installed by the user.
System env variable
Going forward, we can use env variable to point the directory for OS
to look up the binaries. For this, we can use $PATH system-level variable that holds a list of directories.
Whenever you are entering a command in the terminal like cat, vim, or nano, the OS is looking for the corresponding
name in provided directories via $PATH. If there is a match, it runs a matching binary.
Let’s check how it looks on our OS:
$ printenv PATH
/Users/meldo8/.docker/bin:/Users/meldo8/.cargo/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin
We got a list of directories separated by :. Let’s make them easier to read:
/Users/meldo8/.docker/bin
/Users/meldo8/.cargo/bin
/usr/local/bin
/System/Cryptexes/App/usr/bin
/usr/bin
/bin
/usr/sbin
/sbin
Anytime we type any command to a terminal, the OS is looking for the matching binary in those dirs from top to bottom.
Python-depth level
Python dependencies
Python binary per se is an OS-level dependency. Mostly, during the development of software, we are using python-level
dependencies, like various libs (numpy, pandas, or boto3) or frameworks (fastapi or robyn). Let’s take a look
at how these modules are being managed in our OS. We will be using pip, which is the
package installer for Python.
Let’s see if our OS has installed the package polars.
$ python3 -m pip list | grep polars
$
The -m flag is linking our pip with the selected Python interpreter
python3. Thanks to this, we know that we are dealing with installed packages around the python3 command.
Triggering just pip list may show a list of packages for different interpreters present on our machine like python3.11.
The output of the command is empty which means, the package is not present. So let’s install it.
$ python3 -m pip install polars
Defaulting to user installation because normal site-packages is not writeable
Collecting polars
Using cached polars-0.16.4-cp37-abi3-macosx_10_7_x86_64.whl (14.6 MB)
Requirement already satisfied: typing_extensions>=4.0.1 in /Users/meldo8/Library/Python/3.9/lib/python/site-packages (from polars) (4.4.0)
Installing collected packages: polars
Successfully installed polars-0.16.4
$ python3 -m pip list | grep polars
polars 0.16.4
In the log of the install process, we can see where pip was looking for packages ->
/Users/meldo8/Library/Python/3.9/lib/python/site-packages. Let’s take a look for a lib polars there.
$ ls /Users/meldo8/Library/Python/3.9/lib/python/site-packages | grep polars
polars
polars-0.16.4.dist-info
The lib is actually there, but how does the Python binary knows where to look for it, while we are executing some scripts?
Python packages source
By default, Python is looking for dependencies in directories defined in sys.path.
$ python3
Python 3.9.6 (default, Oct 18 2022, 12:41:40)
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
[
> '',
> '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip',
> '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9',
> '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload',
> '/Users/meldo8/Library/Python/3.9/lib/python/site-packages',
> '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages'
> ]
One way to populate this array is via using the $PYTHONPATH.
It works similarly to $PATH. It adds the list of directories, in which Python is looking for packages.
$ printenv PYTHONPATH
$
$ export PYTHONPATH="test-path"
$ printenv PYTHONPATH
test-path
$ python3
Python 3.9.6 (default, Oct 18 2022, 12:41:40)
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
[
'',
'/Users/meldo8/test-path',
'/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip',
'/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9',
'/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload',
'/Users/meldo8/Library/Python/3.9/lib/python/site-packages',
'/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages'
]
In the begging $PYHOTNPATH is empty, so we set it to test-path.
And that new value can be now seen in the output of the sys.path.
The rest of the directories are provided by module site.
As we can read in the docs
This module is automatically imported during initialization.
[...]
Importing this module will append site-specific paths to the module search path and add a few built-ins.
Methods responsible for populating the sys.path with dirs are
getsitepackages()
and getuserpackages().
The one more interesting for us is the second one, which is providing the dirs for Python libs installed by the user.
$ python3
Python 3.9.6 (default, Oct 18 2022, 12:41:40)
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import site
>>> site.getsitepackages()
['/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages', '/Library/Python/3.9/site-packages', '/AppleInternal/Library/Python/3.9/site-packages', '/AppleInternal/Tests/Python/3.9/site-packages']
>>> site.getusersitepackages()
'/Users/meldo8/Library/Python/3.9/lib/python/site-packages'
And here is our dir with libs ;)
Global Python package
In the previous steps, we installed the polars package as the global one. It means it can be accessed in every project
in our current workstation via binary python3.
Let’s create a simple script and trigger it without any pre-install step.
import polars as pl
df = pl.DataFrame(
{
'Company': ['Ford','Toyota','Toyota','Honda','Toyota',
'Ford','Honda','Subaru','Ford','Subaru'],
'Model': ['F-Series','RAV4','Camry','CR-V','Tacoma',
'Explorer','Accord','CrossTrek','Escape','Outback'],
'Sales': [58283,32390,25500,18081,21837,19076,11619,15126,
13272,10928]
}
)
print(df)
$ python3 global_package/main.py
shape: (10, 3)
┌─────────┬───────────┬───────┐
│ Company ┆ Model ┆ Sales │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 │
╞═════════╪═══════════╪═══════╡
│ Ford ┆ F-Series ┆ 58283 │
│ Toyota ┆ RAV4 ┆ 32390 │
│ Toyota ┆ Camry ┆ 25500 │
│ Honda ┆ CR-V ┆ 18081 │
│ ... ┆ ... ┆ ... │
│ Honda ┆ Accord ┆ 11619 │
│ Subaru ┆ CrossTrek ┆ 15126 │
│ Ford ┆ Escape ┆ 13272 │
│ Subaru ┆ Outback ┆ 10928 │
└─────────┴───────────┴───────┘
The python3 interpreter is taking the polars package from the dirs we described above.
What about the situation when we want to have different versions in each project? or just not to pollute our working env with python packages with every new developing service?
Python tools
Virtual Env
One of the solutions is virtualenv. It creates separate directories, just
for our current project, where the packages are going to be installed. Let’s give it a try.
$ python3 -m venv ./venv/
$ ls venv/
bin/ include/ lib/ pyvenv.cfg
$ ls venv/bin
Activate.ps1 activate.csh get_objgraph* pip3* python@ python3.9@ watchmedo*
activate activate.fish pip* pip3.9* python3@ undill*
It creates the directory for virtual env with prepared binaries of pip and python3. These binaries are linked
to OS-level binaries. It is very important to know, that you need the OS-level Python interpreter, in order to be able
to create working virtual env in your project.
Let’s initiate the virtual environment and check how it deals with package dirs.
$ source ./venv/bin/activate.fish
(venv) $ python3
Python 3.9.6 (default, Oct 18 2022, 12:41:40)
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
[
'',
'/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip',
'/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9',
'/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload',
'/Users/meldo8/PycharmProjects/python-binary-pdm/virtualenv_package/venv/lib/python3.9/site-packages'
]
The very last path is new to us. It actually points to the dir created in our venv in a specific project. Thanks to this
the Python interpreter, triggered via virtual shell, will
know where to look for packages for this particular project.
(venv) $ ls ./venv/lib/python3.9/site-packages/
_distutils_hack/ pip/ pkg_resources/ setuptools-58.0.4.dist-info/
distutils-precedence.pth pip-21.2.4.dist-info/ setuptools/
Site-packages are almost empty for now, we can start to populate the dir with our needs. Let’s test the package, that is
not installed globally - robyn.
(venv) $ python3 -m pip install robyn
Collecting robyn
Using cached robyn-0.24.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (6.3 MB)
Collecting uvloop==0.17.0
Using cached uvloop-0.17.0-cp39-cp39-macosx_10_9_x86_64.whl (1.5 MB)
Collecting watchdog==2.2.1
Using cached watchdog-2.2.1-cp39-cp39-macosx_10_9_x86_64.whl (88 kB)
Collecting multiprocess==0.70.14
Using cached multiprocess-0.70.14-py39-none-any.whl (132 kB)
Collecting dill>=0.3.6
Using cached dill-0.3.6-py3-none-any.whl (110 kB)
Installing collected packages: watchdog, uvloop, dill, multiprocess, robyn
Successfully installed dill-0.3.6 multiprocess-0.70.14 robyn-0.24.1 uvloop-0.17.0 watchdog-2.2.1
(venv) $ ls ./venv/lib/python3.9/site-packages/ | grep robyn
robyn
robyn-0.24.1.dist-info
Now, we got installed the robyn package in our venv dir, which shouldn’t be accessible via global interpreter outside the virtual
shell. Let’s check it with example web app.
(venv) $ python3 main.py
INFO:robyn:Starting server at 127.0.0.1:8080
INFO:robyn:Press Ctrl + C to stop
INFO:actix_server.builder:Starting 1 workers
INFO:actix_server.server:Actix runtime found; starting in Actix runtime
While the virtual shell is activated the script is starting without any issues.
$ (venv) $ deactivate
$ python3 main.py
12:45:51
Traceback (most recent call last):
File "/Users/meldo8/PycharmProjects/python-binary-pdm/virtualenv_package/main.py", line 1, in <module>
from robyn import Robyn
ModuleNotFoundError: No module named 'robyn'
While we are utilizing the global interpreter directly, it does not know the package robyn. So the script is not able
to import the robyn module.
Now, we got our separation, but the price for that is we need to build venv every time we want to start
a particular service. We still need to have a global interpreter present on our working machine to link it to our
project-specific Python interpreter. Also triggering the code needs to be done inside the virtual shell, to use
the interpreter, which is aware of the packages’ directory in the current project.
Is there a better way to link global interpreters with project-level packages?
PEP582 and PDM
Finally, we reach the real reason behind this blog post. We will try to get through the PEP582.
PEP means Python Enhancement Proposals. This particular proposition is about changing sys.path, to include
the directory __pypackages__ and prioritize it over site-packages dirs. The __pypackages__ dir will be used to store
packages per project, but thanks to adding it to sys.path, it could be used by the global Python interpreter. The tool
already utilizing this feature is pdm. Let’s go through a quick use-case with
this tool and feel the future.
brew install pdm - to install tool on MacOS
$ mkdir pdm_package
$ cd pdm_package/
$ pdm init
Creating a pyproject.toml for PDM...
Please enter the Python interpreter to use
0. /usr/local/bin/python3.11 (3.11)
1. /usr/bin/python3 (3.9)
2. /usr/local/bin/python3.8 (3.8)
3. /usr/local/Cellar/python@3.11/3.11.1/Frameworks/Python.framework/Versions/3.11/bin/python3.11 (3.11)
Please select (0): 1
Using Python interpreter: /usr/bin/python3 (3.9)
Would you like to create a virtualenv with /Library/Developer/CommandLineTools/usr/bin/python3? [y/n] (y): n
You are using the PEP 582 mode, no virtualenv is created.
For more info, please visit https://peps.python.org/pep-0582/
Is the project a library that is installable?
A few more questions will be asked to include a project name and build backend [y/n] (n): n
License(SPDX name) (MIT):
Author name (meldo8):
Author email (...):
Python requires('*' to allow any) (>=3.9):
Changes are written to pyproject.toml.
We are creating a dir for a new project and initiating it with pdm. There are two crucial decisions, from the perspective of
python dependencies management:
- during the first step. We need to pick the global interpreter, that is going to be used in our project. It can be changed later.
- during the second step. We need to decide if we want to utilize PEP582 functionality, that’s why we decline to create a virtual environment.
$ ls
.pdm.toml pyproject.toml
pdm created for us two files:
.pdm.tomlcontains configuration specific to our system. Right now there is only a Python interpreter that is going to be used in this project. It shouldn’t be committed to a remote repo.pyproject.tomlcontains the project’s build metadata and dependencies needed for PDM. It should be committed to the repo.
To enable PEP582 we need to evaluate pdm --pep582.
$ pdm --pep582
if test -n "$PYTHONPATH"
set -x PYTHONPATH '/usr/local/Cellar/pdm/2.4.2/libexec/lib/python3.11/site-packages/pdm/pep582' $PYTHONPATH
else
set -x PYTHONPATH '/usr/local/Cellar/pdm/2.4.2/libexec/lib/python3.11/site-packages/pdm/pep582'
end
$ eval "$(pdm --pep582)"
The reason why we see .../python3.11/... in the output is because pdm was installed as the dependency of python3.11
interpreter. It doesn’t mean that pdm as a tool can be used only within this interpreter, which we will see shortly.
Firstly, what are eval and test doing?
From the docs we can read the:
$ man eval
[...]
eval - evaluate the specified commands
[...]
$ man test
[...]
Tests the expression given and sets the exit status to 0 if true, and 1 if false.
[...]
Ok, so now we know that with eval "$(pdm --pep582)"we are evaluating the script that is either appending already
existing $PYTHONPATH or creating the bare new one with value
'/usr/local/Cellar/pdm/2.4.2/libexec/lib/python3.11/site-packages/pdm/pep582'. Thanks to this and
site packages loading
we are making our python3 interpreter aware of dir __pypackages__ in our projects directories.
To utilize it, let’s create the same example web app with robyn. This time we are going to trigger the code with
global interpreter, having the package installed only on the project level, not OS (global).
$ pdm add robyn
Adding packages to default dependencies: robyn
🔒 Lock successful
Changes are written to pdm.lock.
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 5 to add, 0 to update, 0 to remove
✔ Install multiprocess 0.70.14 successful
✔ Install watchdog 2.2.1 successful
✔ Install dill 0.3.6 successful
✔ Install uvloop 0.17.0 successful
✔ Install robyn 0.24.1 successful
🎉 All complete!
$ ls
__pypackages__/ main.py pdm.lock pyproject.toml
$ ls __pypackages__/3.9/
bin/ include/ lib/
pdm created for us __pypackages__ dir and lock file.
We can see that structure of __pypackages__ is very similar to venv dir. Let’s check if our lib is there.
$ ls __pypackages__/3.9/lib/ | grep robyn
robyn
robyn-0.24.1.dist-info
Present!
The final test is ahead. Firstly, let’s make sure we don’t have the robyn global package.
$ python3 -m pip list | grep robyn
$
An empty response, so we got no robyn package available.
Now we are triggering the web server in /pdm_package/main.py with the global interpreter.
$ python3 ./pdm_package/main.py
INFO:robyn:Starting server at 127.0.0.1:8080
INFO:robyn:Press Ctrl + C to stop
INFO:actix_server.builder:Starting 1 workers
INFO:actix_server.server:Actix runtime found; starting in Actix runtime
It works! Despite not having the package on the OS level the Python interpreter was able to import the robyn module
from __pypackages__.
Now let’s take a look, if it means, that we can reuse lib from pdm_package project in virtualenv_package.
$ python3 ./virtualenv_package/main.py
Traceback (most recent call last):
File "/Users/meldo8/PycharmProjects/python-binary-pdm/./virtualenv_package/main.py", line 1, in <module>
from robyn import Robyn
ModuleNotFoundError: No module named 'robyn'
Great, it fails. Everything is according to the plan. Our package from one project is not “leaking” into another one. Thanks to it we can have, for example, separated versions of the same package across different projects.
Summary
PDM seems to be a really nice tool to manage Python packages. It allows us to have one proxy less (without virtualenv)
and still, possess the packages separation and many other features.
Refreshing the directory hierarchy and linking mechanism behind binary and Python interpreters should make it easier
to actually configure a pdm tool in your new project.