v0.3.5

New features

• Add optional most_recent_file flag to SharefileToDiskOperator to extract the most recent version of a singleton file from a path.

Under the hood

• Log a warning message when SharefileHook.folder_id_from_path() finds no files for a given path.
• Change logic in callable sql.s3_dir_to_postgres() to raise an AirflowException if any copy of S3 key fails, instead of only when all fail.

Full Changelog: v0.3.4...v0.3.5

v0.3.4

What's Changed

• Feature/update sharefile by @ejoranlienea in #58

Full Changelog: v0.3.3...v0.3.4

ea_airflow_util v0.3.3

Under the hood

• Add dest_filename argument to s3_to_sharefile callable to optionally override filename
• Add LoadSharefileCustomUsersDag to top-level package import path

Fixes

• Run DAG setup method during LoadSharefileCustomUsersDag initialization

Full Changelog: v0.3.2...v0.3.3

ea_airflow_util v0.3.2

New features

• AWSParamStoreToAirflowDAG allows more flexibility when passing Parameter Store paths. Use {tenant_code} when the tenant is in the middle of the path, instead of the end.
• Add s3_to_sharefile and disk_to_sharefile callables
• Add methods to the SharefileHook
• Add ShareFile callable check_for_new_files() to assert expectations in ShareFile directory
• Add S3ToSnowflakeOperator to S3 operators
• Add LoadSharefileCustomUsersDag to automate Heimdall user creation from uploaded authenticated users files in ShareFile

Under the hood

• Code and error-handling improved in callable sharefile_to_disk. Arguments ds_nodash and ts_nodash are deprecated.
• Refactor SFTPToSnowflakeDAG and S3ToSnowflakeDAG to use new S3ToSnowflakeOperator
• Update callable ftp.download_all to accept either a remote directory or file.

Full Changelog: v0.3.1...v0.3.2

ea_airflow_util 0.3.1

New features

• Boolean argument is_manual_upload in S3ToSnowflakeDag rearranges S3 source pathing to easier structure for partners

Under the hood

• Copy statement in S3ToSnowflakeDag uses regex instead of string-splitting to infer pull-date and pull-timestamp

Fixes

• Fix bug in EACustomDAG where default_args were not passed to DAG super init.

Full Changelog: v0.3.0...v0.3.1

ea_airflow_util v0.3.0

New features

• Migrate FTP, ShareFile, casing, and ZIP utilities from Rally into ea_airflow_util
• New EACustomDAG factory to streamline DAG instantiation moving forward

Under the hood

• Move all Python callables out from /airflow/dags/dag_util into /airflow/callables
  • Note that all original imports are still valid and are secretly rerouted in __init__.py.
• Overload callables.airflow.xcom_pull_template to accept a task ID string or an Airflow Operator
• All DAGs use EACustomDAG to standardize initialization

Full Changelog: v0.2.6...v0.3.0

ea_airflow_util v0.2.6

What's Changed

• Feature/dbt_run_dag_trigger by @sleblanc23 in #37

Full Changelog: v0.2.5...v0.2.6

ea_airflow_util v0.2.5

What's Changed

• make changing header case optional by @ejoranlienea in #33
• Feature/db clean dag by @jayckaiser in #34

Full Changelog: v0.2.4...v0.2.5

ea_airflow_util v0.2.4

Bugfix release
Full Changelog: v0.2.3...v0.2.4

ea_airflow_util v0.2.3

What's Changed

• Feature/s3 dir to postgres by @ejoranlienea in #32

Full Changelog: v0.2.2...v0.2.3