update lb delete dag from main (so we can test it) #120
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s_to_snowflake(). (#91)
* Update changelog. * Increment version in setup.py.
* init * fix null array * looking into typeerror * safe map cast * safe map cast * list casting * list casting * debugging * debugging * change comment * update changelog * label mapped tasks * remove straggler code * Update setup.py Increment setup.py with latest version. --------- Co-authored-by: sleblanc23 <[email protected]> Co-authored-by: Jay Kaiser <[email protected]>
There can be multiple column combinations equivalent to the 'highest match rate'. This ensures that we are consistently using the SAME combination of columns when that situation occurs.
* draft * chain taskgroup in * fix * type fix * remove unnecessary rerouting * oops * add total_counts args to readme * default total_counts to false * fix indent * fix optional args and logging message
* change default of pull_all_deletes to false no longer necessary * update changelog
* draft of move log helper to util and use for em fails * fix import * add back kwargs * reintroduce structured log * rework * . * making util work as executor or wrapper creator * return to old log format * only log errors * fixes to em call * simplify wrapper for both hope this works * log even on exit * need raise exception * . * remove log filter * print conn id * pass context * need to pass through logging args ew * access context correctly * filter to actual errors * combining logs * fix kwargs and error handling * context should be passed as a kwarg not arg * restructure the logs * . * Unnest serialize_argument helper. * Revert EarthmoverOperator and apply logging wrapper to operator in EarthbeamDAG. * Commit missing code from previous commit. --------- Co-authored-by: jayckaiser <[email protected]>
* Add mapped_index_templates to dynamic EMLB taskgroups. * Add exit code return logic to EarthmoverOperator, even if unused in EM updates. * Update Snowflake match rate logic to only retrieve records that meet the required match rate. * Add check_new_match_rates task to gate sending tasks on non-matched assessment records and refactor taskstream logic for clarity. * Update remove files operator to skip null xcoms and only trigger on non-skipped taskgroups. --------- Co-authored-by: ejoranlienea <[email protected]>
* refactor bulk sql queries to reflect full refresh behavior * clear deletes and keychanges records on full refresh * add change version records for deletes and keyChanges on full refresh * maintain return pattern * delete only enabled endpoints on full refresh * retrieve configured endpoints through helper function
When pulling prior max CVs on a resource, we need to exclude deletes and key_changes, not take the max across all records
…nal when not enabled in the DAG. (#112)
* add map index to log * need to pull it out * Update earthbeam_dag.py * move to the run earthmover one * Update earthbeam_dag.py * Update earthbeam_dag.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.