Eliminate `Path(file)` to locate resource files #2210

asnare · 2025-12-30T15:01:33Z

Changes

What does this PR do?

This PR eliminates the use of Path(__file__) to locate file-based resources. In general:

In the main code, this is buggy: modern Python should use importlib.resources instead.
In tests, it's brittle and difficult to read because it's normally based on relative paths and traversal of many parent directories. It's also unnecessary, because pytest provides a mechanism for locating the project root. (Tests can also use importlib.resources, and probably should, but that's a bigger refactoring: this is a first step.)

Some incidental changes that are introduced:

Type hints in a few places.
Functional tests against Snowflake, Oracle and Presto have been consolidated into a single module. (The Snowflake functional tests have been moved into test_snowflake.py: test_databricks.py was misleading.)
The PipelineConfig (and Step) classes have been made immutable. (An unnecessary __post__init() method has also been eliminated.)

Tests

existing unit tests
existing integration tests

…ed for tests. This removes the need for `Path(__file__)`-based manipulation.

…urces.

github-actions · 2025-12-30T15:04:38Z

✅ 51/51 passed, 5 flaky, 4m13s total

Flaky tests:

🤪 test_transpiles_informatica_to_sparksql (23.663s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (19.583s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (4.363s)
🤪 test_transpile_teradata_sql (23.465s)
🤪 test_transpile_teradata_sql_non_interactive[True] (5.771s)

_{Running from acceptance #3354}

codecov · 2025-12-30T15:06:22Z

Codecov Report

❌ Patch coverage is 37.50000% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.05%. Comparing base (25312ad) to head (74d370e).

Files with missing lines	Patch %	Lines
.../labs/lakebridge/assessments/dashboards/execute.py	0.00%	13 Missing ⚠️
.../labs/lakebridge/assessments/profiler_validator.py	0.00%	3 Missing ⚠️
...databricks/labs/lakebridge/assessments/profiler.py	0.00%	2 Missing ⚠️
...cks/labs/lakebridge/assessments/profiler_config.py	80.00%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2210   +/-   ##
=======================================
  Coverage   64.05%   64.05%           
=======================================
  Files         100      100           
  Lines        8624     8625    +1     
  Branches      893      889    -4     
=======================================
+ Hits         5524     5525    +1     
  Misses       2928     2928           
  Partials      172      172

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

In addition, update tests: - No more Path(__file__) manipulation. - Test pipeline configurations now use paths relative to the test resources.

…ipulation.

…manipulation.

…ion.

…anipulation.

Note: the path being loaded doesn't seem to exist, so it's possible that this was broken before (and still is).

asnare · 2025-12-30T15:45:24Z

src/databricks/labs/lakebridge/assessments/dashboards/execute.py

+    # TODO: Verify this, I don't think it works? (These files are part of the test resources.)
+    schema_def = resources.files(assessment_resources).joinpath(f"{source_tech}_schema_def.yml")


I don't think this worked prior to this PR (or after): this is a pure refactoring.

But it does raise an interesting point: how is this tested?

@sundarshankar89: How can I check this?

These should be the source resources directory, not part of tests.

@goodwillpunning correct me if I'm wrong

@sundarshankar89, @goodwillpunning: The only resource I can find is: tests/resources/assessments/synapse_schema_def.yml

If this is not working, I propose creating an issue for this and fixing it independently of this PR.

tests/integration/assessments/test_pipeline.py

src/databricks/labs/lakebridge/assessments/dashboards/execute.py

src/databricks/labs/lakebridge/assessments/profiler.py

tests/integration/assessments/test_profiler_validator.py

src/databricks/labs/lakebridge/assessments/profiler.py

tests/unit/conftest.py

tests/unit/transpiler/test_databricks.py

sundarshankar89 · 2026-01-12T05:42:45Z

tests/unit/transpiler/test_snowflake.py

+from ..conftest import FunctionalTestFile, FunctionalTestFileWithExpectedException, get_functional_test_files
+
+
+def pytest_generate_tests(metafunc: pytest.Metafunc) -> None:


I see you moved test_databricks to test_snowflake

Indeed, the pattern for the other ones is to name these after the source system. So the changes here are:

The snowflake ones were moved from test_databricks.py to test_snowflake.py.

There were 2 sets of functional tests for each source system, in separate files: I've moved them into the same file.

asnare added 9 commits December 30, 2025 15:00

Use path traversal instead of string interpolation/parsing.

3ca7ce2

Type hint.

24926ba

Use class-method directly.

081afb6

Tidy up path manipulation to load version file.

8f8698d

Type hints.

39654af

Introduce a test fixture that provides the location of resources need…

8b57115

…ed for tests. This removes the need for `Path(__file__)`-based manipulation.

Refactor existing tests that use Path(__file__) to locate test reso…

d0b547d

…urces.

Drop unused global.

d58b0ab

Eliminate more Path(__file__) manipulation.

b6af977

asnare self-assigned this Dec 30, 2025

asnare added tech debt design flaws and other cascading effects internal technical pr's not end user facing labels Dec 30, 2025

asnare had a problem deploying to tool December 30, 2025 15:01 — with GitHub Actions Error

asnare had a problem deploying to tool December 30, 2025 15:01 — with GitHub Actions Failure

asnare added 2 commits December 30, 2025 16:08

Use pathlib instead of string interpolation.

f50ec68

Add missing import.

00261c9

asnare had a problem deploying to tool December 30, 2025 15:08 — with GitHub Actions Failure

Set defaults directly instead of during __post_init__().

2c48403

asnare had a problem deploying to tool December 30, 2025 15:18 — with GitHub Actions Error

Make PipelineConfig immutable.

2007cc3

In addition, update tests: - No more Path(__file__) manipulation. - Test pipeline configurations now use paths relative to the test resources.

asnare temporarily deployed to tool December 30, 2025 15:22 — with GitHub Actions Inactive

asnare added 7 commits December 30, 2025 16:23

Update the test LSP server to not use Path(__file__) manipulation.

4e0f733

Consolidate Presto functional tests, and eliminate Path(__file__) man…

2b4a8a2

…ipulation.

Update Oracle functional tests to eliminate Path(__file__) manipulation.

234e6f5

Consolidate Snowflake functional tests, and eliminate Path(__file__) …

6bd340c

…manipulation.

Update Morpheus integration tests to not use Path(__file__) manipulat…

b7e81ce

…ion.

Update tests that run our stub LSP server to not use Path(__file__) m…

d991934

…anipulation.

Remove unused code.

04e22c5

asnare added 2 commits December 30, 2025 16:29

TODO marker for future work.

b6b85a4

Update assessment pipeline to avoid Path(__file__) manipulation.

32a8519

Note: the path being loaded doesn't seem to exist, so it's possible that this was broken before (and still is).

asnare temporarily deployed to tool December 30, 2025 15:31 — with GitHub Actions Inactive

asnare marked this pull request as ready for review December 30, 2025 15:38

asnare requested a review from a team as a code owner December 30, 2025 15:38

asnare requested review from gueniai, m-abulazm and sundarshankar89 December 30, 2025 15:38

asnare commented Dec 30, 2025

View reviewed changes

Fix formatting.

a9faad1

asnare temporarily deployed to tool December 30, 2025 15:46 — with GitHub Actions Inactive

asnare mentioned this pull request Jan 2, 2026

Handle empty SQL resultsets gracefully in assessment pipeline #2172

Open

7 tasks

m-abulazm requested changes Jan 5, 2026

View reviewed changes

Merge branch 'main' into no-more-file-manip

87108f9

asnare temporarily deployed to tool January 5, 2026 13:11 — with GitHub Actions Inactive

asnare added 2 commits January 6, 2026 11:36

Type hints for factory used as test fixture.

0e6147d

Eliminate path_to_resources()

ab05369

asnare temporarily deployed to tool January 6, 2026 11:07 — with GitHub Actions Inactive

Mark a method to be renamed.

767eb51

asnare requested a review from m-abulazm January 6, 2026 11:42

asnare temporarily deployed to tool January 6, 2026 12:53 — with GitHub Actions Inactive

Merge branch 'main' into no-more-file-manip

98ccf1e

asnare temporarily deployed to tool January 7, 2026 13:18 — with GitHub Actions Inactive

m-abulazm approved these changes Jan 8, 2026

View reviewed changes

Merge branch 'main' into no-more-file-manip

d108e66

asnare temporarily deployed to tool January 9, 2026 12:58 — with GitHub Actions Inactive

sundarshankar89 reviewed Jan 12, 2026

View reviewed changes

Merge branch 'main' into no-more-file-manip

74d370e

asnare temporarily deployed to tool January 12, 2026 11:14 — with GitHub Actions Inactive

		# TODO: Verify this, I don't think it works? (These files are part of the test resources.)
		schema_def = resources.files(assessment_resources).joinpath(f"{source_tech}_schema_def.yml")

		from ..conftest import FunctionalTestFile, FunctionalTestFileWithExpectedException, get_functional_test_files


		def pytest_generate_tests(metafunc: pytest.Metafunc) -> None:

Eliminate Path(__file__) to locate resource files #2210

Are you sure you want to change the base?

Eliminate Path(__file__) to locate resource files #2210

Uh oh!

Conversation

asnare commented Dec 30, 2025

Changes

What does this PR do?

Tests

Uh oh!

github-actions bot commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

asnare Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

asnare Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sundarshankar89 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

asnare Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Eliminate `Path(file)` to locate resource files #2210

Eliminate `Path(file)` to locate resource files #2210

github-actions bot commented Dec 30, 2025 •

edited

Loading

codecov bot commented Dec 30, 2025 •

edited

Loading