Poetry 101

Package Dependency Management with Poetry

Official Poetry Website

Installation of Poetry in a separate venv

Instructions from the official website here

For Linux/Ubuntu

curl -sSL https://install.python-poetry.org | python3 -

Follow the on screen instructions after the output Installing Poetry (1.2.1): Done, the version of poetry might be different.

# Add Poetry's bin directory to the PATH environment variable of the shell configuration to execute poetry
# replace my username with yours in the command below
export PATH="/home/shamik/.local/bin:$PATH"

# test that everything is set up by running the following
poetry --version
# it should output something like this Poetry (version 1.2.1)

Default poetry installation path ~/.local/share/pypoetry

Now poetry can be executed in the native shell, however, if you open a new shell it won't be available and you will have to add Poetry's bin directory to the PATH environment variable in the shell configuration file. To resolve the issue of exporting poetry path every time we open a shell, we can set it permanently in ~/.bashrc. Add the following to the end of the ~/.bashrc.

# initialising poetry
# replace my username with yours in the command below
export PATH="/home/shamik/.local/bin:$PATH"

Update Poetry

poetry self update
# can be used to specify a particular version also 
poetry self update 1.2.1

Uninstalling Poetry

curl -sSL https://install.python-poetry.org | python3 - --uninstall

Create a new Poetry Project

# poetry new <project name>
poetry new my_poetry

This creates a new directory my_poetry which has the following structure. Notice that the package name is by default my_poetry.

To create a new poetry project with a different package name.

# poetry new <project name> --name <package name>
poetry new my_poetry --name mypoetry

This creates a new directory my_poetry which has the following structure. Notice that the package name is not default but what you set it to be mypoetry.

However, if one wants to house all the code under a particular folder called, for instance src, then while making a new poetry project one can specify the same.

# poetry new <project name> --src
poetry new my_poetry --src

This creates a new directory my_poetry which has the following structure. Notice that the package name is default but it's inside the src folder.

Create a new virtual environment(venv) in the project

The venv can be created only if you are in the poetry project directory as poetry requires pyproject.toml to create it. Once you are in the project directory, check for any venv by poetry env list, if you haven't created one before it shouldn't return anything.

To create one simply type poetry env use python3 this creates a venv for the project using the system python version, which was used to install poetry.

Now you can see the name of the venv with poetry env list.

Default poetry venv location

By default poetry will store all the venvs under .cache/pypoetry/virtualenvs. One can check all the poetry environments here.

Note: By default the venv created by poetry might not be available to VSCode interpreter, but the interpreter path retrieved by typing poetry env info can be added.

Creating a venv with specific python version

# poetry env use <python version or /full/path/to/python>
poetry env use 3.8.0

Installing Project Specific Packages

To add a package which the project requires.

# this will always try and install the latest version of the library
poetry add pytest
# to specify a particular version
poetry add pytest==7.1.3
# or
poetry add pytest<=7.1.3

This will automatically add the particular package in the pyproject.toml file under project dependencies, generally under the header [tool.poetry.dependencies], and create a poetry.lock file. The poetry.lock file contains detailed installed package information(package versions) along with their dependencies. If this file is present, poetry will always rely on this file to install the dependencies.

Installing Development Specific Packages

Sometimes, while developing we might like to have additional packages installed which aren't useful for running the project, for instance a code formatter like autopep8, a documentation generator like Sphinx, and a static analysis tool like pycodestyle, Pylint, Flake8, mypy, or coverage.py. To add these kind of packages one must use the -G option along with the poetry add -G dev <package name>.

poetry add -G dev autopep8

This will automatically add the particular package in the pyproject.toml file under development dependencies, generally under the header [tool.poetry.group.dev.dependencies], and create/update the poetry.lock file.

Manually Adding Packages to pyproject.toml

Adding a specific version of the requests library under the project dependencies [tool.poetry.dependencies] in pyproject.toml.

requests = "^2.26.0"
beautifulsoup4 = "4.10.0"

After adding requests, below are two possible scenarios

Scenario without a poetry.lock file

Run poetry install. In the background poetry will look for a poetry.lock file and when it doesn't find one, it will resolve the dependencies listed in pyproject.toml.

Scenario with a poetry.lock file

Run poetry lock. In the background poetry will look for a poetry.lock file and when it finds one, it will resolve the dependencies listed in poetry.lock file considering the new additions to the project dependencies. Note: This doesn't mean that the above packages are installed yet, as we didn't install after having run poetry lock. When you run poetry show it will show these packages, but they haven't been installed as poetry show lists all packages available. On a side note, poetry show -t displays all the packages with their dependencies. To confirm the same run poetry run python as a REPL and try and import these packages. You will encounter ModuleNotFoundError.

Installing the new additions

poetry install

This reads the poetry.lock file and installs all the dependencies. Now all these packages will be ready for importing.

Updating Packages with dependencies

These are the two steps to be followed always:

1. Update the packages
2. Install the updated packages Note: If you are unsure about any of the update operations, you can always use the --dry-run option with any of the commands below. This will display all the changes that will take place without performing them.

Updating all packages

poetry update
poetry install

This updates all packages and their dependencies within their version constraints.

Updating specific packages

poetry update requests beautifulsoup4
poetry install

It will look for new versions of these packages given their version constraints and if found will install them along with updating the dependencies if required. If you want to update a dependency with a version that’s higher than the defined version in the pyproject.toml file, you need to adjust the pyproject.toml file beforehand.

Updating a package to the latest version

poetry add requests@latest
poetry install

If you don't use add and instead use update, then it will not update it.

Adding Poetry to an existing project with requirements.txt

1. Initialise the project with poetry with poetry init inside the directory of the project
2. Initialise a venv using poetry env use python3. If it's any other python installed by pyenv then pyenv local <python version> and then poetry env use <python version>, e.g. pyenv local python3.9 & poetry env use python3.9
3. Add the requirements.txt with poetry add `cat requirements.txt` with a --dry-run option first and then you can look at the output to check whether the current requirements.txt can be installed with the mentioned packages in them
4. Generate a requirements.txt file after installing all the relevant packages in poetry with this poetry export -o poetry_generated_requirements.txt --without-hashes
5. Step out of the poetry project directory otherwise it will install the packages in the active poetry venv
6. Use pip install -r requirements.txt in the venv of choice

Some Poetry commands

Click to look at some of the Poetry Commands!

Poetry Command	Explanation
$ poetry --version	Show the version of your Poetry installation.
$ poetry new	Create a new Poetry project.
$ poetry init	Add Poetry to an existing project.
$ poetry run	Execute the given command with Poetry.
$ poetry add	Add a package to pyproject.toml and install it.
$ poetry update	Update your project’s dependencies.
$ poetry install	Install the dependencies.
$ poetry show	List installed packages.
$ poetry lock	Pin the latest version of your dependencies into poetry.lock.
$ poetry lock --no-update	Refresh the poetry.lock file without updating any dependency version.
$ poetry check	Validate pyproject.toml.
$ poetry config --list	Show the Poetry configuration.
$ poetry env list	List the virtual environments of your project.
$ poetry export	Export poetry.lock to other formats.