Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bigquery insert-from-staging replace strategy uses staging-optimized pattern #2422

Open
loveeklund-osttra opened this issue Mar 19, 2025 · 0 comments
Labels
bug Something isn't working community This issue came from slack community workspace

Comments

@loveeklund-osttra
Copy link

dlt version

1.6.1

Describe the problem

Hey, I'm loading data into bigquery and I want to use the
https://dlthub.com/docs/general-usage/full-loading#the-insert-from-staging-strategy
insert from staging strategy.
But when looking at the query that gets run on bigquery it runs

DROP TABLE IF EXISTS `<proj_id>`.`<dataset>`.`<destination_table>`;

CREATE TABLE `<proj_id>`.`<dataset>`.`<destination_table>`` CLONE `<proj_id>`.`<dataset>`.`<destination_table>`;

So it looks like it still runs staging-optimized ?

Expected behavior

I expect the query to look more like

    INSERT INTO `{destination_table}` ({columns})
    SELECT {values}
    FROM `{staging_table}` s;

Steps to reproduce

run a pipeline with a bigquery target and set "replace_strategy": "staging-optimized", and then look at they queries that are run in your project history.

Operating system

macOS

Runtime environment

Docker, Docker Compose

Python version

3.12

dlt data source

sql_table

dlt destination

Google BigQuery

Other deployment details

Na

Additional information

The workaround would be to first make dlt write out to a staging dataset and then run something like
this query yourself in some function afterwards
CREATE OR REPLACE TABLE{destination_table}AS SELECT * FROM{staging_table} s;
Which I guess could work, but would be nicer if it worked as expected :)

@loveeklund-osttra loveeklund-osttra changed the title bigquery insert-from-staging replace strategy doesn't work bigquery insert-from-staging replace strategy uses staging-optimized pattern Mar 19, 2025
@anuunchin anuunchin added community This issue came from slack community workspace bug Something isn't working labels Mar 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working community This issue came from slack community workspace
Projects
Status: Todo
Development

No branches or pull requests

2 participants