DOC: Architecture - Required env vars and split into containers

[ghsa-retrieval]

What type of documentation would you like?
Documentation about the architecture of DejaCode and how its components interact with each other and other systems (postgres, redis, scancode, ldap, ...) for authentication

Documentation topic
I'm currently attempting to build a helm chart for DejaCode based on the provided Docker Compose solution, but the current architecture left some questions not answered by current documentation. I have asked about this on gitter and Philippe Ombredanne instructed me to file a ticket.

1. Recently the scheduler has been introduced for updating vulnerability information. In Docker Compose it is run as a separate container. Is there any reason it is not simply a background task in the worker?
2. Currently all containers started by compose share environment variables. Most likely web, worker and scheduler do not all need access to the DB and redis password as well as other env vars. Which ones are actually needed for which container? Relevant env vars are:
   • DATABASE_PASSWORD
   • REDIS_URL
   • DEJACODE_REDIS_PASSWORD
   • SECRET_KEY
   • SECRET_KEY_FALLBACKS
   • EMAIL_HOST_PASSWORD
   • AUTH_LDAP_BIND_PASSWORD
   • SCANCODEIO_API_KEY
   • PURLDB_API_KEY
   • VULNERABLECODE_API_KEY

[pombredanne]

@ghsa-retrieval thanks for the detailed report.
@tdruez FYI

[tdruez]

Recently the scheduler has been introduced for updating vulnerability information. In Docker Compose it is run as a separate container. Is there any reason it is not simply a background task in the worker?

Yes, there could be multiple worker services started, where there is only one scheduler needed.

Most likely web, worker and scheduler do not all need access to the DB and redis password as well as other env vars.

The web, worker, and scheduler services all need access to the DB as well as a Redis connection.

DATABASE_PASSWORD
REDIS_URL
DEJACODE_REDIS_PASSWORD
SECRET_KEY
SECRET_KEY_FALLBACKS
EMAIL_HOST_PASSWORD
AUTH_LDAP_BIND_PASSWORD
SCANCODEIO_API_KEY
PURLDB_API_KEY
VULNERABLECODE_API_KEY

All those env vars (Django related) can be required by the services running the DejaCode Docker image: web, worker, and scheduler.

For example, the VULNERABLECODE_API_KEY needs to be available in the worker context for the "Fetch vulnerabilities" background task. It's also required by the web service to fetch data from VulnerableCode on various events.

Now, the docker.env is also included in the db service but only the POSTGRES_* vars are used in that service context.
We could move those into another .env file but I'm not sure about the benefits of doing so.

[ghsa-retrieval]

Thank you for the information! This means that the Helm chart will have to provide all the secrets to all three pods.