feat: Expand sql table resource config #2396
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
It's possible to specify sql table using
config.toml
file (see docs).The problem is this setting-based approach allows only
append
strategy and doesn't allow to specify customprimary
ormerge
keys.While
primary
keys are reflected from source table it's not possible for views.So this PR allows to specify
write_disposition
,primary_key
andmerge_key
inconfig.toml
file with some restrictions (see Additional Context):Additional Context
There is a problem with not total alignment between
def resource(...)
method's arguments and fields in config classes.Current config system doesn't support
Union
types (and probably it's not possible and there is no point) but this leads to the situation that we can't use plain string values:but we have to use more complex dict or lists values:
If you have any better ideas how to realize this opportunity I'm open to adjust my PR or implement it different way.