dlt version

1.9.0

Describe the problem

We are running a DLT pipeline using Dagster. Our pipeline is configured this way:

pipeline = dlt.pipeline(
    pipeline_name=pipeline_name, 
    destination=dlt.destinations.snowflake(
        credentials=settings.target_connection_config.to_connection_string()), 
    dataset_name=dataset_name,
    progress=dlt.progress.log(log_period=30.0, logger=logger)
)

After upgrading DLT from version 0.4 to 1.9, the log_period seems to have no effect, and it is logging continuously, polluting our logs and running up our costs since we are sending the logs to Datadog.

In version 0.4 it was working correctly.

Expected behavior

When log_period is set to 30, dlt should log progress no more than once every 30 seconds.

Steps to reproduce

Configure a pipeline as above, observe that it logs continuously rather than every 30 seconds.

Operating system

Linux

Runtime environment

Kubernetes

Python version

3.12

dlt data source

sql_database

dlt destination

Snowflake

Other deployment details

Dagster

Additional information

No response