Python: Using Poetry with an existing project in GitHub

How to use Poetry for package and dependency management with an existing GitHub repository. Poetry will also create and manage a virtual development environment, and help you publsh your Python package. Poetry stores all of the project information it needs in the pyproject.toml file, which consolidates and replaces many other Python setup and management files.

Using Poetry with an existing GitHub repo

Once Poetry is installed and configured on your development workstation, you can use it to manage packages and dependencies in your Python projects.

Then, clone the repo you want to use with Poetry to your development workstation.

Set up Poetry in your GitHub repo directory

Now that you've cloned your new GitHub repo onto your development workstation, let's configure Poetry in your new project directory before we add any Python code or packages.

Getting started with Poetry

First, to ensure you have the latest version of Poetry, run:

poetry self update

You don't need to do this if you just installed Poetry, but it's a good habit to check for the latest version of Poetry when starting a new project.

Also, if you know you're going to need to use specific packages in your new Python project, assemble a list of the packages you want to add to your project.

You can always add more packages later, but you'll save a few steps if you install the packages you want when you first set up your project in Poetry.

Poetry can create an entire project directory structure from scratch using the poetry new command, but since you've already created the project directory by cloning your new GitHub repo, you'll use the poetry init command instead. Let's say you created a new repo directory called my_poetry_repo.

Create the pyproject.toml file

Poetry uses the standard pyproject.toml file, first described in [PEP 518 – Specifying Minimum Build System Requirements for Python Projects](https://peps.python.org/pep-0518/), to store all the information it needs about your project.

Python will automatically generate a pyproject.toml file for you, and you can configure it using the interactive prompts Poetry provides. Here's how to create your pyproject.toml file with Poetry:

cd /path/to/my_poetry_repo
poetry init

This will launch an interactive Q&A where the answers you provide are used to create your project's pyproject.toml file. You can specify project packages and development packages (like flake8) during the Q&A.

The project packages you'll want to install depend on the requirements of your specific project. There are several development packages that we like to use in almost all of our Python projects.

Here's a list of development packages we find especially useful (in alphabetical order):

• black: Python code formatter
• flake8: Python style guide enforcement
• isort: Sort import statements automatically
• mypy: optional Python variable type system
• pylint: linter for Python
• pytest: automated testing
• pytest-cov: code coverage reports for pytest

Don't worry about getting everything right, because you can always edit the pyproject.toml file by hand later on:

vim pyproject.toml

Of course, you can use any code editor or IDE to edit this file.

In addition, if you ever want to add another package to your pyproject.toml file, you can use the poetry add command, like this:

poetry add package_name

When you've completed all your edits, push your new pyproject.toml file to GitHub. In the root of your project directory, enter these git commands:

git status
git add pyproject.toml
git commit -m 'Autogenerated by poetry init'
git push

You can also use the built-in tools in your code editor or IDE to sync with your remote GitHub repo.

Installing packages with Poetry

Once you've set up your pyproject.toml file the way you like it, you'll to install all the packages it specifies. In the root of your project directory, run:

poetry install

This will create the virtual environment and then install all of the packages described in the pyproject.toml file. By default, Poetry creates your virtual environment outside of your project directory, which is exactly what you want if your project is under source code control. This way, the generic Python package files won't be incorporated in your unique Python code repo. Since some Python packages are quite large, this will save a considerable amount of space in your GitHub repo.

You can change this and several other configuration options in the pyproject.toml file -- see the Poetry Configuration Documentation for details. But, as we've seen, the default settings are exactly what you want for most Python projects.

Running poetry install for the first time will create a new poetry.lock file. The Poetry documentation recommends that you typically commit your poetry.lock file to version control:

Committing this file to VC [version control] is important because it will cause anyone who sets up the project to use the exact same versions of the dependencies that you are using.

Here's how to do this at the command line, from your project root directory:

git status
git add poetry.lock
git commit -m 'Autogenerated by poetry install'
git push

Pro tip: these separate git commits tell a story that's easy for other contributors to follow.

Using your Poetry virtual environment

At this point, you don't have any Python code in your project directory. Also, because you used poetry init to set up our Poetry environment, instead of poetry new project_name you'll still need to create other project directories, like tests.

Once you've created your project directories and added some code, activate your poetry virtual environment with this command:

poetry shell

This creates a new shell with your virtual environment fully configured. You can now use the import statement to use the installed Python packages in your own code.

When you're done with your Poetry environment, exit the poetry shell like this:

exit

This will also return you to the original shell you used to activate the poetry shell.

You can also use the poetry run command to run a single Python script or command line tool (like, say, black); or activate the poetry virutal environment in your current shell with the source command.

See the Basic usage page in the Poetry documentation for details on how to use these additional commands in your specific operating system environment.