fhir-to-lakehouse

Reads FHIR bundles from Kafka, encodes them using the Pathling encoders, and stores them as Delta Lake tables.

Development

The compose.yaml contains the development fixtures required to run the program out-of-the-box:

• Apache Kafka (Exposed on <127.0.0.1:9094>)
• Kafbat - an Apache Kafka UI (Exposed on <127.0.0.1:8084>)
• MinIO (Exposed on <127.0.0.1:9000> and <127.0.0.1:9001> for the UI)
• mock-data-loader: used to pre-load Kafka with sample FHIR bundles

start all services using

docker compose up

and the program itself using

python src/main.py

Tests

Unit Tests

pytest --cov=src tests/unit/

Integration Tests

Currently, running these tests outside of the CI requires some manual effort:

kind create cluster --config=tests/integration/kind-config.yaml
helm dep up tests/integration/fixtures/
helm upgrade --install --wait fixtures tests/integration/fixtures/
helm upgrade --install --wait --set "stream-processors.enabled=true" --set "stream-processors.processors.fhir-to-delta.container.image.tag=test" fixtures tests/integration/fixtures/

To run the integration tests

pytest tests/integration

To check the table counts

duckdb -no-stdin -init tests/integration/check-counts.sql