Skip to content

Run OpenCodelists in Dokku in Codespaces #2487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

StevenMaude
Copy link
Contributor

@StevenMaude StevenMaude commented May 1, 2025

See #2483.

This PR is currently a sketch of adding a configuration for running OpenCodelists in Dokku in a dev container/Codespaces. It is not complete enough to be merged, but functional enough to try out. If it's useful, it may warrant some tidying before merging.

It adds a minimal devcontainer.json configuration and a script that:

  • installs just, necessary to build the test databases and Docker image
  • installs Dokku via docker-compose (it was tricky to get Dokku working outside of Docker, in the dev container directly)
  • builds the OpenCodelists test data
  • copies the OpenCodelists tests data into the right place for Dokku
  • configures Dokku to run OpenCodelists
  • builds the OpenCodelists production Docker image
  • deploys that OpenCodelists image with Dokku

You can see the Django logs by running:

docker exec -t dokku sh -c "dokku logs opencodelists --tail"

With this commit, you should get a working OpenCodelists in Dokku in
Codespaces, with the web server viewable, and the minimal set of test
data available.

This would need some refining and polishing before merging.

It also does not yet restart the server for a stopped codespace; we'd
need a `postAttachCommand` providing in the `devcontainer.json` and an
appropriate script that might only need to do something like:

```sh
cd /workspaces/opencodelists
docker-compose up -d
docker exec -t dokku sh -c "dokku ps:rebuild opencodelists opencodelists:latest"
```
Comment on lines +7 to +10
- "3022:22"
- "8080:80"
- "8443:443"
- "7000:7000"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Are all these ports necessary? It might only be 7000 — the port for the production OpenCodelists image — with 22 useful if you want to SSH in (though you can run docker exec and docker cp which should cover most needs).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: We don't specify a Docker image. Do we want to use a different image than the default? And do we want to do some of the setup currently in dokku.sh in a Dockerfile instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: Some of the setup here could probably go in Dockerfile and be built as part of the image.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: This file could probably go somewhere other than the root directory.

Comment on lines +4 to +5
cd /tmp && wget https://github.com/casey/just/releases/download/1.40.0/just-1.40.0-x86_64-unknown-linux-musl.tar.gz && tar -xzf just-1.40.0-x86_64-unknown-linux-musl.tar.gz && chmod 555 just
sudo mv /tmp/just /usr/bin/
Copy link
Contributor Author

@StevenMaude StevenMaude May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: While downloading this way is OK, although it would be better to specify a version somewhere as a variable, this is necessary because the default dev container image is currently based on Ubuntu 22.04, which doesn't have just in its repositories. (Ubuntu 24.04 does, albeit the version is probably older than the current release.)

cd /workspaces/opencodelists
cp dotenv-sample .env
touch db.sqlite3 # workaround because build-dbs-for-local-development expects a db.sqlite3
yes Y | just build-dbs-for-local-development nuclear
Copy link
Contributor Author

@StevenMaude StevenMaude May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: yes is necessary because build-dbs-for-local-development has an interactive user confirmation; not sure if the capitalised Y is necessary.

Copy link
Contributor Author

@StevenMaude StevenMaude May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: The use of yes results in failure if you use set -o pipefail because there's a broken pipe when the command on the right-hand side completes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: We need Python to run the initial setup in build-dbs-for-local-development.

We're currently on Python 3.12 for OpenCodelists.

The default dev container image currently has Python 3.12 available, which is convenient, but more luck than anything else.

To be more confident of always having the correct Python version in future, we'd need to retrieve the required version of Python ourselves.

"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {}
},
"postCreateCommand": "/bin/bash /workspaces/opencodelists/dokku.sh"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: We could provide ${containerWorkspaceFolder} as an argument to dokku.sh here, which then avoids hardcoding the directory in dokku.sh.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: We could provide a postAttachCommand script which should restart the installed server correctly when a codespace is started after stopping.

@StevenMaude
Copy link
Contributor Author

StevenMaude commented May 1, 2025

Seeing the following in the Dokku logs:

2025-05-01T10:03:12.548362602Z app[web.1]: [error    ] Exception while exporting Span batch. [opentelemetry.sdk.trace.export]
2025-05-01T10:03:12.548414148Z app[web.1]: Traceback (most recent call last):
2025-05-01T10:03:12.548417251Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
2025-05-01T10:03:12.548419729Z app[web.1]:     sock = connection.create_connection(
2025-05-01T10:03:12.548421924Z app[web.1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548424235Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
2025-05-01T10:03:12.548426311Z app[web.1]:     raise err
2025-05-01T10:03:12.548428157Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
2025-05-01T10:03:12.548430460Z app[web.1]:     sock.connect(sa)
2025-05-01T10:03:12.548432425Z app[web.1]: ConnectionRefusedError: [Errno 111] Connection refused
2025-05-01T10:03:12.548433990Z app[web.1]: 
2025-05-01T10:03:12.548435596Z app[web.1]: The above exception was the direct cause of the following exception:
2025-05-01T10:03:12.548437506Z app[web.1]: 
2025-05-01T10:03:12.548439249Z app[web.1]: Traceback (most recent call last):
2025-05-01T10:03:12.548440830Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
2025-05-01T10:03:12.548442797Z app[web.1]:     response = self._make_request(
2025-05-01T10:03:12.548444429Z app[web.1]:                ^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548446183Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 493, in _make_request
2025-05-01T10:03:12.548448502Z app[web.1]:     conn.request(
2025-05-01T10:03:12.548450315Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/connection.py", line 445, in request
2025-05-01T10:03:12.548452270Z app[web.1]:     self.endheaders()
2025-05-01T10:03:12.548454097Z app[web.1]:   File "/usr/lib/python3.12/http/client.py", line 1333, in endheaders
2025-05-01T10:03:12.548456029Z app[web.1]:     self._send_output(message_body, encode_chunked=encode_chunked)
2025-05-01T10:03:12.548457725Z app[web.1]:   File "/usr/lib/python3.12/http/client.py", line 1093, in _send_output
2025-05-01T10:03:12.548459472Z app[web.1]:     self.send(msg)
2025-05-01T10:03:12.548461285Z app[web.1]:   File "/usr/lib/python3.12/http/client.py", line 1037, in send
2025-05-01T10:03:12.548463331Z app[web.1]:     self.connect()
2025-05-01T10:03:12.548465183Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/connection.py", line 276, in connect
2025-05-01T10:03:12.548482387Z app[web.1]:     self.sock = self._new_conn()
2025-05-01T10:03:12.548484614Z app[web.1]:                 ^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548486738Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/connection.py", line 213, in _new_conn
2025-05-01T10:03:12.548488838Z app[web.1]:     raise NewConnectionError(
2025-05-01T10:03:12.548490764Z app[web.1]: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x70ca0f503110>: Failed to establish a new connection: [Errno 111] Connection refused
2025-05-01T10:03:12.548493269Z app[web.1]: 
2025-05-01T10:03:12.548495191Z app[web.1]: The above exception was the direct cause of the following exception:
2025-05-01T10:03:12.548497242Z app[web.1]: 
2025-05-01T10:03:12.548499138Z app[web.1]: Traceback (most recent call last):
2025-05-01T10:03:12.548502510Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
2025-05-01T10:03:12.548504978Z app[web.1]:     resp = conn.urlopen(
2025-05-01T10:03:12.548506999Z app[web.1]:            ^^^^^^^^^^^^^
2025-05-01T10:03:12.548508862Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
2025-05-01T10:03:12.548511059Z app[web.1]:     retries = retries.increment(
2025-05-01T10:03:12.548513349Z app[web.1]:               ^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548515362Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
2025-05-01T10:03:12.548517722Z app[web.1]:     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
2025-05-01T10:03:12.548519928Z app[web.1]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548522070Z app[web.1]: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x70ca0f503110>: Failed to establish a new connection: [Errno 111] Connection refused'))
2025-05-01T10:03:12.548525282Z app[web.1]: 
2025-05-01T10:03:12.548527328Z app[web.1]: During handling of the above exception, another exception occurred:
2025-05-01T10:03:12.548529499Z app[web.1]: 
2025-05-01T10:03:12.548531462Z app[web.1]: Traceback (most recent call last):
2025-05-01T10:03:12.548533565Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 362, in _export_batch
2025-05-01T10:03:12.548536309Z app[web.1]:     self.span_exporter.export(self.spans_list[:idx])  # type: ignore
2025-05-01T10:03:12.548538551Z app[web.1]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548540614Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 189, in export
2025-05-01T10:03:12.548543023Z app[web.1]:     return self._export_serialized_spans(serialized_data)
2025-05-01T10:03:12.548545234Z app[web.1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548547678Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 159, in _export_serialized_spans
2025-05-01T10:03:12.548553122Z app[web.1]:     resp = self._export(serialized_data)
2025-05-01T10:03:12.548555271Z app[web.1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548557463Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 133, in _export
2025-05-01T10:03:12.548559797Z app[web.1]:     return self._session.post(
2025-05-01T10:03:12.548561968Z app[web.1]:            ^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548564030Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/requests/sessions.py", line 637, in post
2025-05-01T10:03:12.548566380Z app[web.1]:     return self.request("POST", url, data=data, json=json, **kwargs)
2025-05-01T10:03:12.548568668Z app[web.1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548571078Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2025-05-01T10:03:12.548573520Z app[web.1]:     resp = self.send(prep, **send_kwargs)
2025-05-01T10:03:12.548575650Z app[web.1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548577832Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2025-05-01T10:03:12.548580195Z app[web.1]:     r = adapter.send(request, **kwargs)
2025-05-01T10:03:12.548582257Z app[web.1]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-01T10:03:12.548584368Z app[web.1]:   File "/opt/venv/lib/python3.12/site-packages/requests/adapters.py", line 700, in send
2025-05-01T10:03:12.548586615Z app[web.1]:     raise ConnectionError(e, request=request)
2025-05-01T10:03:12.548588873Z app[web.1]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x70ca0f503110>: Failed to establish a new connection: [Errno 111] Connection refused'))

Possibly due to not configuring any OpenTelemetry settings?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: This file could probably go somewhere other than the root directory.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: This file is taken directly from Dokku's documentation with only the addition of port 7000.


cd /workspaces/opencodelists
cp dotenv-sample .env
touch db.sqlite3 # workaround because build-dbs-for-local-development expects a db.sqlite3
Copy link
Contributor Author

@StevenMaude StevenMaude May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: See #2482 for more information.

touch db.sqlite3 # workaround because build-dbs-for-local-development expects a db.sqlite3
yes Y | just build-dbs-for-local-development nuclear

docker-compose up -d
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: This should probably be the v2 usage:

docker compose up -d

The devcontainer Docker feature makes this work automatically, but we should use the more modern form.

docker exec -t dokku sh -c "dokku apps:create opencodelists"
docker exec -t dokku sh -c 'dokku config:set opencodelists BASE_URLS="http://localhost:7000,http://127.0.0.1:7000" DATABASE_DIR="/storage" DATABASE_URL="sqlite:////storage/db.sqlite3" DJANGO_SETTINGS_MODULE="opencodelists.settings" SECRET_KEY="thisisatestsecretkeyfortestingonly" TRUD_API_KEY="thisisatesttrudkeyfortestingonly"'

sudo mkdir -p /var/lib/dokku/data/storage/opencodelists
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: I got very, very confused with permissions because I overlooked the docker-compose.yaml volumes configuration.

With the Docker Compose set up, this directory is on the host running Dokku in a container, not inside the Dokku container! (Inside the OpenCodelists container, I was seeing an empty directory owned by root.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant