Python environments

Introduction

Python is an amazing language but the one thing it really lacks is a good package manager. While it does have a default package manager, its not a particularly good one - it can’t publish packages, doesn’t support lock files and makes it a pain to deal with virtual environments.

Its so bad in fact that we have so many similar tools trying to do the same thing - pipenv, pipx, poetry and anaconda are just some of the more popular ones.

And since everyone prefers a different way to package applications, install them, version them and even run them, the entire process is incredibly complex, as illustrated by this excellent XKCD comic

Unfortunately, there’s no one solution to all of this. Until that time comes, below are some recommendations.

But I use <insert IDE here>

Yes. Lots of modern IDEs, especially something like PyCharm, do hide some of these pain points. But not everyone likes using such a heavy(and bloated) IDE for a simple scripting language like Python.

And even if you use them, you can still run into issues from time to time and having a Black Box solution makes it difficult to debug even the simplest of issues.

Installing Python

If you’re using the python which came along with your system, stop. Ideally, you should never be touching your system python - at least not for developing large complex applications with lots of dependencies.

This is because other dependencies on your system might depend on the system python(this is especially true if you’re running Linux) and installing/removing packages from here might resultant in borked applications.

Instead, use pyenv. It is a python version manager tool for Unix like systems. For example, once you have it installed, you just need to run

pyenv install 3.9.9

in the terminal to install python 3.9.9 or any other version. You can now use this python install without worrying about breaking your system.

All these python installs are independent of each other - their packages don’t interfere and you can seamlessly switch between multiple python versions.

Unfortunately, pyenv doesn’t work on Windows. But there’s a fork called pyenv-win for those of you who use Windows.

Installing Packages

Even with isolated python versions, you shouldn’t install packages globally. This is because pip, as mentioned previously, is a terrible package manager and the moment you have more than 2 applications on the same python version, you’ll realize that there’s no good way to keep track of the dependencies of both these applications separately.

The solution to this is to use virtual environments. Unfortunately, just like everything with python, you have a lot of options here. You could handle virtual environments manually with venv or go for a more higher level tool like virtualenv or virtualenvwrapper. But even with these, the process is not as simple as something like bundle on Ruby or npm on Node.

Instead, use poetry. Poetry handles virtually everything for you - the virtual environment, the lock file, packaging and even publishing your package to PyPI.

You could use something like pipenv too, but poetry supports the pyproject.toml standard as proposed in PEP 517 and PEP 518.

Installing Applications

If you need to install python applications, like say youtube-dl, use pipx

With pipx, every application is installed in its own virtual environment. This way none of their dependencies will ever conflict with each other.

Another way would be to install the application from homebrew or your Linux distributions repositories if its available there. This obviously comes with the same caveat as before - don’t mess with your system python. If you do, its very likely that these applications will stop working.

Conclusion

And with that, you’re hopefully much better equipped to deal with the madness that is Python application development.