Platform Tools

Using the dbt-platform-helper package

See the package documentation for detail on what the package is and how to use it.

If you are migrating a service to DBT PaaS, GOV.UK PaaS to DBT PaaS Migration will also be relevant for you.

Supported Python versions

3.9, 3.10, 3.11 and 3.12.

Contributing to the dbt-platform-helper package

Getting started

1. Clone the repository:

   git clone https://github.com/uktrade/platform-tools.git && cd platform-tools
   

2. Install the required dependencies:

   pip install poetry && poetry install && poetry run pre-commit install
   

Testing

Requirements

The following tools are required to run the full test suite.

• checkov
• AWS Copilot

Unit tests

Run poetry run pytest in the root directory to run all tests.

Or, run poetry run tox in the root directory to run all tests for multiple Python versions. See the tox configuration file.

Note: by default the tests are run using multiple processes for speed. When running using multiple processes pdb (python debugger) does not play nicely and will error.

To allow pdb to work correctly, disable multiple processes using the --numprocesses 0 option:

poetry run pytest --numprocesses 0

We use Codecov to monitor the comprehensiveness and performance of our unit tests.

Manual testing

You may want to test any CLI changes locally.

Run poetry build to build your package resulting in a package file (e.g. dbt_platform_tools-0.1.40.tar.gz) in a dist folder. You may need to bump up the package version before doing so.

Copy the package file(s) to the directory where you would like to test your changes, and make sure you are in a virtual environment. Run platform-helper --version to check the installed package version (e.g. 0.1.39).

Note

Copying the package file is optional, but recommended. You can keep the package file in the dist folder and install the package from your directory.

Run pip install <file> and confirm the installation has worked by running platform-helper --version which would output version 0.1.40 following our example.

Important

When testing is complete, do not forget to revert the dbt-platform-helper installation back to what it was; e.g. pip install dbt-platform-helper==0.1.39.

End to end testing

Because this codebase is only fully exercised in conjunction with several others, we have platform-end-to-end-tests, which orchestrates the testing of them working together.

Publishing

Publishing to PyPI happens automatically when a GitHub Release is published. To publish the Python package dbt-platform-helper manually, you will need an API token.

1. Acquire API token from Passman.
   • Request access from the SRE team.
   • Note: You will need access to the platform group in Passman.
2. Run poetry config pypi-token.pypi <token> to add the token to your Poetry configuration.

Update the version, as the same version cannot be published to PyPI.

poetry version patch

More options for the version command can be found in the Poetry documentation. For example, for a minor version bump: poetry version minor.

Build the Python package.

poetry build

Publish the Python package.

Note: Make sure your Pull Request (PR) is approved and contains the version upgrade in pyproject.toml before publishing the package.

poetry publish

Check the PyPI Release history to make sure the package has been updated.

For an optional manual check, install the package locally and test everything works as expected.

Releases

Release structure

• Use Conventional Commits in our pull request titles. These become the release commit messages in the main branch. The advantage of this is that it's clear from a list of commits, what kind of changes they include (fix=patch, feat=minor, !=breaking change) and most importantly whether commits include breaking changes
• Use GitHub Releases to document and manage releases
• Non-breaking changes already in the main branch should be released before merging a breaking change
• Minor and patch releases should always include zero downtime
• Documentation for releases with breaking changes should include the upgrade path
• Where possible the upgrade path should allow for zero downtime, if this is not possible, it should be flagged up big and visible

Release Automation

Merging to main

• Merging to main will trigger the pull-request-regression-tests pipeline in the platform-tools AWS account to run regression tests
• We use the release-please GitHub action to create and update a release PR when changes are merged to main
  • The release PR will automatically update the pyproject.toml version number and generate release notes based on the commits merged since the last release
  • Merging the release PR will create a draft GitHub release for the next version with release notes

Publishing GitHub release

Publishing a GitHub release should automatically:

• Run the full pull-request-regression-tests pipeline
• Trigger a CodeBuild project called platform-tools-build in the platform-tools AWS account to run. This runs the buildspec-pypi.yml file which contains the build steps to publish the new platform-helper package version to PyPI
• Trigger a rebuild of the DBT Platform Documentation, so it includes the latest release documentation (currently WIP)
• Push a notification to the development community via the #developers channel in Slack

Publishing to PyPI

Successful completion of the CodeBuild project platform-tools-build in the platform-tools AWS account executes the buildspec-pypi.yml file which publishes the new application version to PyPI. This build process involves the following steps:

• Retrieving a list of prior releases from the dbt-platform-helper PyPI project repository.
• Verifying if the dbt-platform-helper package version in the root pyproject.toml file does not exist in the list of releases obtained from PyPI.
• If the version does not exist in the PyPI release list, the application is built and published to PyPI as a new release with the version stated in the application pyproject.toml file
• Next the script again checks if the updated version number is included in the PyPI release list. If found, it indicates that the new package version exists in PyPI.
• A Slack message is sent to the #developers channel to notify others that the platform-helper tool has been updated

Release procedure

1. Work on a new branch
2. Create a PR and have it reviewed
3. Once approved:
   • If it is a breaking change, you must release any outstanding non breaking changes before merging
   • Merge to main
4. A release PR will automatically be created when changes are merged to main
   • The release PR is updated with next version number and release notes based on the commits since the last release
5. Merge the release PR to create a GitHub release
6. Ensure the release notes contain an upgrade path for any breaking changes
7. Check PyPI for the new published version