feature/freshness-updates (#63)

* feature/freshness-updates

* pr ref fix

* Update packages.yml

Co-authored-by: Jamie Rodriguez <[email protected]>

---------

Co-authored-by: Jamie Rodriguez <[email protected]>

Author: fivetran-joemarkiewicz

.buildkite/pipeline.yml

@@ -3,7 +3,7 @@ steps:
     key: "run-dbt-postgres"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"
@@ -18,7 +18,7 @@ steps:
     key: "run_dbt_snowflake"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"
@@ -35,7 +35,7 @@ steps:
     key: "run_dbt_bigquery"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"
@@ -47,7 +47,7 @@ steps:
     key: "run_dbt_redshift"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"
@@ -62,7 +62,7 @@ steps:
     key: "run_dbt_databricks"
     plugins:
       - docker#v3.13.0:
-          image: "python:3.8"
+          image: "python:3.10.13"
           shell: [ "/bin/bash", "-e", "-c" ]
           environment:
             - "BASH_ENV=/tmp/.bashrc"

.buildkite/scripts/run_models.sh

@@ -17,6 +17,7 @@ echo `pwd`
 cd integration_tests
 dbt deps
 dbt seed --target "$db" --full-refresh
+dbt source freshness --target "$db" || echo "...Only verifying freshness runs…"
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
 dbt run --vars '{github__using_repo_team: false}' --target "$db" --full-refresh

CHANGELOG.md

@@ -1,9 +1,34 @@
-# dbt_github version.version
+# dbt_github v0.9.0
+
+[PR #63](https://github.com/fivetran/dbt_github/pull/63) includes the following updates:
+
+## Breaking Change for dbt Core < 1.9.6
+
+> *Note: This is not relevant to Fivetran Quickstart users.*
+
+Migrated `freshness` from a top-level source property to a source `config` in alignment with [recent updates](https://github.com/dbt-labs/dbt-core/issues/11506) from dbt Core ([GitHub Source v0.9.0](https://github.com/fivetran/dbt_github_source/releases/tag/v0.9.0)). This will resolve the following deprecation warning that users running dbt >= 1.9.6 may have received:
+
+```
+[WARNING]: Deprecated functionality
+Found `freshness` as a top-level property of `github` in file
+`models/src_github.yml`. The `freshness` top-level property should be moved
+into the `config` of `github`.
+```
+
+**IMPORTANT:** Users running dbt Core < 1.9.6 will not be able to utilize freshness tests in this release or any subsequent releases, as older versions of dbt will not recognize freshness as a source `config` and therefore not run the tests.
+
+If you are using dbt Core < 1.9.6 and want to continue running GitHub freshness tests, please elect **one** of the following options:
+  1. (Recommended) Upgrade to dbt Core >= 1.9.6
+  2. Do not upgrade your installed version of the `github` package. Pin your dependency on v0.8.1 in your `packages.yml` file.
+  3. Utilize a dbt [override](https://docs.getdbt.com/reference/resource-properties/overrides) to overwrite the package's `github` source and apply freshness via the previous release top-level property route. This will require you to copy and paste the entirety of the previous release `src_github.yml` file and add an `overrides: github_source` property.
 
 ## Documentation
 - Added Quickstart model counts to README. ([#60](https://github.com/fivetran/dbt_github/pull/60))
 - Corrected references to connectors and connections in the README. ([#60](https://github.com/fivetran/dbt_github/pull/60))
 
+## Under the Hood:
+- Updates to ensure integration tests use latest version of dbt.
+
 # dbt_github v0.8.1  
 This release contains the following updates:
 

README.md

@@ -61,7 +61,7 @@ Include the following github package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/github
-    version: [">=0.8.0", "<0.9.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.9.0", "<0.10.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 Do NOT include the `github_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
@@ -134,7 +134,7 @@ packages:
       version: [">=1.0.0", "<2.0.0"]
 
     - package: fivetran/github_source
-      version: [">=0.8.0", "<0.9.0"]
+      version: [">=0.9.0", "<0.10.0"]
     
     - package: dbt-labs/spark_utils
       version: [">=0.3.0", "<0.4.0"]

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'github'
-version: '0.8.1'
+version: '0.9.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   github:

integration_tests/ci/sample.profiles.yml

@@ -2,10 +2,6 @@
 # HEY! This file is used in the dbt package integrations tests with Buildkite.
 # You should __NEVER__ check credentials into version control. Thanks for reading :)
 
-config:
-    send_anonymous_usage_stats: False
-    use_colors: True
-
 integration_tests:
   target: redshift
   outputs:

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'github_integration_tests'
-version: '0.8.1'
+version: '0.9.0'
 config-version: 2
 profile: 'integration_tests'
 vars:
@@ -63,3 +63,6 @@ seeds:
 dispatch:
   - macro_namespace: dbt_utils
     search_order: ['spark_utils', 'dbt_utils']
+
+flags:
+  send_anonymous_usage_stats: False

packages.yml

@@ -1,3 +1,3 @@
 packages:
 - package: fivetran/github_source
-  version: [">=0.8.0", "<0.9.0"]
+  version: [">=0.9.0", "<0.10.0"]