-
Notifications
You must be signed in to change notification settings - Fork 152
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Understanding how elementary resolves dbt models with CI jobs #1474
Comments
Hi @wusanny ! As of this time we don't have an official support for dbt's CI jobs feature. I'll tag this as an Enhancement but definitely makes sense to reconsider / adapt the behavior there. Something I'd definitely be open to though is to add a flag that forces the elementary schema to be fixed on CI runs - if you'd like to contribute this that would be great. |
Describe the bug
Uncertain if this is a bug or expected behaviour, but we are quite confused with the current behaviour, thus raising this issue to get further clarification.
We have observed that the first time we ran a CI job after installing elementary dbt package, the elementary models will get built into a temporary PR schema, instead of their own custom elementary schema as specified in
dbt_project.yml
.For context, in dbt Cloud, CI jobs will materialize the models in a temporary schema unique to the PR which will then be dropped once the PR is merged/closed (docs for reference here). It is expected that the Elementary models are still written into their own schema defined in the
dbt_project.yml
file (Elementary should still override this schema and write its models into its own schema NOT the temp PR schema).After that PR has been merged and a production job is ran, Elementary models from all the subsequent CI jobs will start writing into the expected schema.
To Reproduce
Prerequisite:
Steps to reproduce the behavior:
dbt build --select state:modified+
' > Debug Logs > Download full debug logscreate or replace table DEVELOPMENT.dbt_cloud_pr_537847_8_elementary_new.dbt_invocations
- instead ofDEVELOPMENT.dbt_sanny_elementary_new.dbt_invocations
.Expected behavior
The expected behaviour is for all elementary models to be built into the custom elementary schema, regardless if it was the initial CI run or not. Note that prior to the merge of that first PR, any metadata that was inserted into the temporary PR schema will disappear when the PR is merged and temporary PR schema is dropped (default behaviour for dbt's CI jobs). Client's data is then lost and cannot be recovered.
Environment (please complete the following information):
Additional context
Debug logs for references
1. Debug log run 261771270 (1st CI run).log
2. Debug log run 261772372 (after merge, main prod run).log
3. Debug log run 261772795 (2nd CI run).log
This is the observed behaviour from multiple testings:
database.pr_9_elementary_schema.model_name
)database.pr_9_elementary_schema.model_name
)database.pr_10_elementary_schema.model_name
)database.dbt_sanny_elementary_schema.model_name
)database.pr_10_elementary_schema.model_name
)database.dbt_sanny_elementary_schema.model_name
)The text was updated successfully, but these errors were encountered: