Introduction to the Poetry tool for Python projects

Table of Contents

• 1. What is Poetry?
• 2. Why use it?
• 3. Using Poetry with Nix
  • 3.1 "Ad hoc" usage
    • 3.1.1 with `nix-shell`
    • 3.1.2 with `nix-env`
  • 3.2 Declarative installation
• 4. Life cycle of a Poetry project
• 5. Poetry's configuration and virtual environment management

1. What is Poetry?

Poetry is modestly described by its official documentation as a dependency management tool, but it is in fact emerging as a de facto standard^†) to

^{[†]: Quoting How to manage Python projects with Poetry}

• control virtual environments

• create and publish packages/libraries

• quickly set up new Python projects (or to easily convert existing ones)

2. Why use it?

Even though the Python eco-system is rich with internal (venv, pyvenv) and external tools (pip, pyenv, pipenv, virtualenv, etc.) that take on one or two of the roles listed above, this diversity can become an issue:

• differing conventions and proliferating command line options incur some cognitive overhead

• one has to carefully choose which tools to use together

• not all such projects run on every operating system (e.g., pyenv does not support Windows directly)

Poetry aims to consolidate the functionality of all these tools (TODO: paraphrase, https://www.software.com/src/poetry-can-redefine-the-future-of-python), while striving to stay easy to use and configure.

^{[†]: Paraphrasing Is Poetry the future of Python?}

3. Using Poetry with Nix

Nix is a cross-platform system package manager supported on most major distributions of Unix derivative operating systems (Linux, BSDs, MacOS, etc.) but it is not available on Windows.

^{The title of the section is deliberately not called "Installation" also because the "Installation" section of the Poetry documentation comprehensively covers this topic for operating systems not managed with Nix (the reason I elected to demonstrate using Poetry with Nix in the first place), and because Nix allows using software (and software components) in a way that doesn't conform to the traditional use of the term "installation".}

3.1 "Ad hoc" use with nix-shell

$ nix-shell -p python3 poetry

nix-shell will start a sub-shell with the latest Python 3 and Poetry executables, and one can start hacking.

NOTE: Once exiting the sub-shell, python3 and poetry are still available on the system (see which poetry and which python3) but they are not referenced in the PATH environment variable. Hence they will need to be aliased or called by their full path. (Packages used with nix-shell are also subject to garbage collection after leaving the sub-shell.)

nix-shell can also be used to set up a more elaborate development environment, and this is probably the most flexible and convenient way.

ASIDE: The blog post Developing Python with Poetry & poetry2nix: reproducible flexible Python environments nicely expands on this topic.

3.2 Imperative installation with nix-env

$ nix-env -i python3 poetry

The only difference between with this one-liner and the one in the previous section is that python3 and poetry are now available in PATH, and they won't be garbage collected (unless removed with [nix-env -e] first).

TODO: nix-env -i poetry fails with error: selector 'poetry' matches no derivations; related to channels?

3.3 Declarative installation

WARNING: This is a more advanced (and sprawling) Nix topic, and I still believe that the most convenient way is using nix-shell, but it's mentioned here for completeness sake.

The following are the best places to start with, depending

• whether on NixOS: Chapter 6. Package Management in the NixOS manual

• or using Nix on other Unix derivatives: 2.6. Declarative Package Management of the Nixpkgs manual

Nix also offers a lot of freedom on how to do things, and this can complicate things, therefore the following resources are also suggested for reading:

• Part III. Package Management in the Nix manual

• Declarative Configuration on the NixOS Wiki

• Declarative package management for normal users thread on NixOS Discourse

• Declarative approach to installing packages for user only on Reddit

• How do I do declarative package management using nix package manager on Debian? on Reddit

• Difference between nix profiles and home-manager on NixOS Discourse

• /etc/nixos/configuration.nix vs ~/.config/nixpkgs/config.nix on Reddit

4. Life cycle of a Poetry project

There are many "getting started" guides out there (e.g., 1, 2, 3, 4, 5, 6), therefore I was striving to show (a hint of) the big picture.

Starting a
 new Python project
from scratch?

Starting a...

Yes

Yes

No

No

poetry new

poetry new

poetry init

poetry init

Specify and install dependencies

  Directly: 1. Edit {project-root}/             

 

  Interactively: 

Specify and install dependencies...

pyproject.toml

pyproject.toml

poetry add

poetry add

In the root of an existing project to start managing it with Poetry.

In the root of an existing...

Set up a new Python project managed with Poetry via an interactive  questionnaire. 

See directory template as well.

Set up a new Python project...

2. poetry install

2. poetry install

See intro to specifying dependencies, and then the reference.

See intro to specifyi...

①

①

poetry update

poetry update

Update dependencies without version constraints (for others, the constraints have to be updated first).

Update dependencies without vers...

time passes

time p...

Is existing
project already
managed with
Poetry?

Is existing...

Yes

Yes

No

No

poetry install

poetry install

E.g., project clone from an online repo, moved from another machine, etc.

E.g., project clone from an...

poetry remove

poetry remove

②

②

Other dependency management commands:

poetry search

poetry show

poetry lock

poetry export

Other dependency management commands:...

After editing pyproject.toml, it is recommended to use poetry check.

After editing pyproject.toml...

③

③

poetry run

poetry run

poetry shell

poetry shell

Develop

Develop

... and run/test the project maybe with

... and run/test the proj...

Amend dependencies

Amend...

Publish package

Publish package

Re-fit project parameters

Re-fit project parameters

poetry env

poetry env

poetry config

poetry config

time passes

time p...

④

④

poetry config always creates a global configuration file if one is not present.

poetry config always creates a...

poetry build

poetry build

poetry publish

poetry publish

E.g., project clone from an online repo, moved from another machine, etc.

E.g., project clone from an...

poetry publish --build

poetry publish --buildViewer does not support full SVG 1.1

5. Poetry's configuration and virtual environment management

WARNING: Please take any info in this section with grain of salt: put this together based on the official documentation, but I'm a new to Python, and it hasn't been reviewed by more experienced users at all.

The flowchart below shows how Poetry's configuration options come into play when

• taking out your project for a spin (using commands such as poetry shell, poetry run)

• making changes in configuration (e.g., with poetry config), or most times

• managing the virtual environment (mostly when using poetry env).

Is local config file
{project-root}/poetry.toml
present?

Is local config file...

Use it

Use it

①

①

Location not configurable

Location not configurable

Poetry command

Poetry command

②

②

Use the global configuration file

(for Linux: ~/.config/pypoetry/config.toml)

Use the global configuration file...

②

②

①

①

To create/manage a project-local config file, use

poetry config --local [key] [value]

To create/manage a project-local config file, use...

Yes

Yes

No

No

Is project running in a virtual environment?

Is project running in a virtual environment?

virualenvs.create

virualenvs.create

Is pip installed?

Is pip installed?

false

false

No

No

Runtime error at one point? It should be there according to the note

Runtime error at one point? It...

  ?

  ?

③

③

③

③

Given that the default value of virtualenvs.in-project boolean configuration option is None, this is a valid concern. See the table at the bottom as well.

Given that the default value of virtualenvs.in-project...

Yes

Yes

true

true

virtualenvs.in-project
∉
{true, false}

virtualenvs.in-project...

Yes

Yes

No

No

{project-root}/.venv
present?

{project-root}/.venv...

Yes

Yes

Use it

Use it

No

No

Create .venv
 in virtualenvs.path

Create .venv...

virtualenvs.in-project

virtualenvs.in-project...

other

other

false

false

{project-root}/.venv
present?

{project-root}/.venv...

Yes

Yes

No

No

Ignore it

Ignore it

Is project running in a virtual environment?

Is project running in a virtual environment?

Use {project-root}/.venv
if present, create it otherwise.

Use {project-root}/.venv...

true

true

Use Python installed with Pip?

Use Python installed with Pip?

Is activated
Python version compatible
with the project?

Is activated...

Not running in a virtual environment

Not running i...

Yes, running in a virtual environment

Yes, running in...

What Python executable

 to use?

What Python executable...

Yes

Yes

Use it

Use it

Is there a
compatible version?

Is there a...

No

No

Yes

Yes

Switch to it

Switch to it

No

No

STOP, and ask user to configure one manually (e.g., pyenv, poetry env)

STOP, and ask user to configure...

④

④

④

④

That is,

• not set explicitly at all
• the default value None
• null
• etc.

That is,...Viewer does not support full SVG 1.1

The above flowchart may not convey clearly how Poetry chooses the virtual environment to use so this table is to provide an alternative representation:

virtualenvs.create	virtualenvs.in-project	{project-root}/.venv present	expected behavior
true (default)	true	yes	use {project_root}/.venv
		no	create {project_root}/.venv
	false	yes	ignore {project_root}/.venv and use/create global one at virtualenvs.path
		no	use/create global one at virtualenvs.path
	other (∉ {true, false})	yes	use {project_root}/.venv
		no	(?) use/create global one at virtualenvs.path
false (and pip is available)	true	yes	I presume that as virtual environments are disabled, the rest depends on the active Python version and pip.
		no
	false	yes
		no
	other (∉ {true, false})	yes
		no
other (∉ {true, false})	true	yes	Given that the default value of virtualenvs.in-project boolean configuration option is None, this I presume that virtualenvs.create may also be given a value other than true or false - although unsure what [Poetry]'s behaviour will be.
		no
	false	yes
		no
	other (∉ {true, false})	yes
		no