Skip to content

edu_edfi_airflow v0.3.0
Compare
Choose a tag to compare
@jayckaiser jayckaiser released this 11 Jun 19:44
· 16 commits to main since this release
v0.3.0
e3a39dc
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

New features

  • Add /keyChanges ingestion for resource endpoints
  • Add new method for EdFiResourceDAG endpoint instantiation using resource_configs and descriptor_configs arguments in init
    • The prior methods EdFiResourceDAG.{add_resource, add_descriptor, add_resource_deletes} are deprecated in favor of this more performant approach.
  • Refactor EdFiToS3Operator taskgroup into three options (determined by run_type argument):
    • "default": One EdFiToS3Operator task per resource/deletes/keyChanges endpoint
    • "bulk": One BulkEdFiToS3Operator task in which all endpoints are looped over in one callable
    • "dynamic": One dynamically-mapped EdFiToS3Operator task per resource with deltas to ingest

Under the hood

  • Copies from S3 to Snowflake in EdFiResourceDAG are now completed in a single bulk task (instead of one per endpoint)
  • EdFiResourceDAG and EarthbeamDAG now inherit from ea_airflow_util DAG factory EACustomDAG
  • Streamline XCom passing between tasks in EdFiResourceDAG
  • Change-version window delta counts are made when checking change versions in Snowflake.
    • Only resources with rows-to-ingest are passed to the Ed-Fi operator.

Full Changelog: v0.2.5...v0.3.0

Assets 2
Loading