[Athena Destination]: Invalid Security Token with Profile Name in 1.3.0+

[tomsej]

dlt version

1.3.0+

Describe the problem

I am experiencing difficulties with the Athena destination in versions 1.3.0 and above of the library. The code I am using worked perfectly in version 1.2.0, but after upgrading, I encounter an error related to invalid security tokens when using AWS credentials specified by profile name. If I provide aws_access_key_id and aws_secret_access_key it works but problem is when I specify profile_name or use AWS_PROFILE env variable.

<class 'dlt.destinations.exceptions.DatabaseTransientException'>
An error occurred (UnrecognizedClientException) when calling the StartQueryExecution operation: The security token included in the request is inval

Expected behavior

No response

Steps to reproduce

• Set up a .dlt/secrets.toml file with the following content:

[sources.credentials]
drivername = "postgresql+psycopg2"
database = "main"
host = "host"
port = 5432
username = "username"
password = "password"

[destination.filesystem]
bucket_url = "s3://bucket-destination"

[destination.filesystem.credentials]
profile_name = "default"

[destination.athena]
query_result_bucket = "s3://bucket-results"
athena_work_group = "workgroup"

[destination.athena.credentials]
profile_name = "default"

• Use the following code to run the pipeline:

from dlt.sources.sql_database import sql_table

if __name__ == "__main__":
    import dlt

    @dlt.source(name="public", max_table_nesting=0)
    def main_public_test():
        tbl = sql_table(
            table="currencies",
            backend="pyarrow",
            defer_table_reflect=True,
        )
        tbl.apply_hints(table_format="iceberg")
        return [tbl]

    try:
        info = dlt.pipeline(
            pipeline_name="main",
            dataset_name="x_test",
            destination="athena",
            progress="log",
            dev_mode=True,
        ).run(main_public_test())
    except Exception as e:
        print(e)

Operating system

macOS

Runtime environment

Local

Python version

3.12

dlt data source

Postgres

dlt destination

AWS Athena / Glue Catalog

Other deployment details

No response

Additional information

• This issue does not occur in version 1.2.0; it works fine when I downgrade.
• I have verified that the credentials are correct and that the same AWS SDK versions (botocore and boto3) are being used.
• I do not see any relevant changes in the release notes for version 1.3.0 that could explain this behavior.