Storing DAG Run Logs in Database Instead of File System

[PraveenKumarM21]

I’m looking for a way to store task logs for a DAG run in a database instead of the default file system or remote storage (e.g., S3, GCS). My current setup is:

Executor: KubernetesExecutor

Deployment Method: Helm chart on Kubernetes

Questions:

1. Is there an existing way to store DAG run logs in a database?

2. Can we configure a custom logging_handler to achieve this?

3. Are there any community plugins or recommendations for handling log storage in a database efficiently?

Any guidance or references would be greatly appreciated.

[potiuk]

1. no
2. yes - you can explore https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html#advanced-configuration
3. not that I am aware, but you can check https://airflow.apache.org/ecosystem/

[PraveenKumarM21]

Thanks for the response!

The reason I want to store task logs in a database is that I need to view live logs for the DAG runs I trigger. With the current setup, logs are stored in files or remote storage, which doesn’t allow real-time querying or filtering of logs easily.

Would there be any recommended approach to achieve this?

Appreciate any insights!

[potiuk]

All that I wrote above. Those are the resources you should read and follow.