Fix serverless compatibility by replacing cache() with conditional persistence #2218

BesikiML · 2026-01-07T03:16:12Z

🐛 Problem
Reconciliation fails on Databricks serverless compute with:
[NOT_SUPPORTED_WITH_SERVERLESS] PERSIST TABLE is not supported on serverless compute
🔍 Root Cause
The reconciliation process uses .cache() for performance optimization, but serverless compute does not support DataFrame caching operations.
✅ Solution
Implemented serverless detection and conditional caching strategy:
Changes Made

Added Serverless Detection Method
New _is_serverless() method checks for clusterNodeType config
Classic clusters: config exists → returns False
Serverless: config throws CONFIG_NOT_AVAILABLE → returns True
Conditional Caching Logic
Classic clusters: Uses .cache() for performance (existing behavior)
Serverless: Skips caching to avoid runtime errors
Technical Details
Detection Method:
node_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterNodeType")
✅ Classic: Returns node type (e.g., i3.2xlarge)
❌ Serverless: Throws AnalysisException with CONFIG_NOT_AVAILABLE

Fixed issue: #1438

Tests

manually tested
added unit tests
added integration tests

Use Unity Catalog volumes instead of .cache() for serverless. Auto-detects compute type. Fixes: [NOT_SUPPORTED_WITH_SERVERLESS]

github-actions · 2026-01-07T03:20:57Z

✅ 51/51 passed, 5 flaky, 4m27s total

Flaky tests:

🤪 test_transpiles_informatica_to_sparksql (23.854s)
🤪 test_transpile_teradata_sql_non_interactive[True] (22.807s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (4.164s)
🤪 test_transpile_teradata_sql (23.798s)
🤪 test_transpile_teradata_sql_non_interactive[False] (5.898s)

_{Running from acceptance #3364}

codecov · 2026-01-07T03:21:03Z

Codecov Report

❌ Patch coverage is 0% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.93%. Comparing base (25312ad) to head (8525599).

Files with missing lines	Patch %	Lines
...bricks/labs/lakebridge/reconcile/reconciliation.py	0.00%	17 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2218      +/-   ##
==========================================
- Coverage   64.05%   63.93%   -0.12%     
==========================================
  Files         100      100              
  Lines        8624     8640      +16     
  Branches      893      894       +1     
==========================================
  Hits         5524     5524              
- Misses       2928     2944      +16     
  Partials      172      172

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…on-serverless-compute

Use specific exception types instead of broad Exception catch to satisfy CI linter rules. Add cluster ID check for improved detection.

Avoids CONFIG_NOT_AVAILABLE exceptions by fetching all configs at once. Passes all linter checks.

m-abulazm · 2026-01-08T16:23:10Z

src/databricks/labs/lakebridge/reconcile/reconciliation.py

+        """Detect if running on serverless compute"""
+        try:
+            # Try to get compute type from Spark conf
+            compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterType", "")


can you link the documentation of this property?

spark.databricks.clusterUsageTags.clusterType is an internal Databricks metadata tag used to identify the compute type. It's not officially documented in public Databricks docs.

Suggested change

compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterType", "")

compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterNodeType", "")

It is possible to find out what is there by running spark.conf.getAll in a databricks notebook

For serveless:

Total configs: 3 spark.databricks.execution.timeout = 9000 spark.sql.ansi.enabled = true spark.sql.shuffle.partitions = auto

m-abulazm · 2026-01-09T08:40:33Z

src/databricks/labs/lakebridge/reconcile/reconciliation.py

+        """Detect if running on serverless compute"""
+        try:
+            # Try to get compute type from Spark conf
+            compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterType", "")


Suggested change

compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterType", "")

compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterNodeType", "")

It is possible to find out what is there by running spark.conf.getAll in a databricks notebook

m-abulazm · 2026-01-09T08:54:53Z

src/databricks/labs/lakebridge/reconcile/reconciliation.py

+        try:
+            # Try to get compute type from Spark conf
+            compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterType", "")
+            if compute_type is None:
+                compute_type = ""
+            return "serverless" in compute_type.lower()
+        except (AnalysisException, AttributeError, KeyError, RuntimeError):
+            # If detection fails (Spark config unavailable or invalid), assume serverless for safety
+            logger.warning("Unable to detect compute type, assuming serverless mode")
+            return True


Suggested change

try:

# Try to get compute type from Spark conf

compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterType", "")

if compute_type is None:

compute_type = ""

return "serverless" in compute_type.lower()

except (AnalysisException, AttributeError, KeyError, RuntimeError):

# If detection fails (Spark config unavailable or invalid), assume serverless for safety

logger.warning("Unable to detect compute type, assuming serverless mode")

return True

try:

compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterNodeType")

except (AnalysisException, SparkNoSuchElementException):

compute_type = "unknown"

return "standard" not in compute_type.lower()

I investigated this a bit on a standard cluster:
spark.conf.get("spark.databricks.clusterUsageTags.clusterNodeType") returns 'Standard_D8ads_v6'

on a serverless cluster it does not work, that is why on errors we can assume serverless

distinguishes serverless (CONFIG_NOT_AVAILABLE) from classic clusters.

…on-serverless-compute

…ss-compute' of github.com:databrickslabs/lakebridge into 1438-feature-remorph-reconcile-fails-to-run-on-serverless-compute

feat: fix serverless compute compatibility

fe944f2

Use Unity Catalog volumes instead of .cache() for serverless. Auto-detects compute type. Fixes: [NOT_SUPPORTED_WITH_SERVERLESS]

BesikiML requested a review from m-abulazm January 7, 2026 03:16

BesikiML requested a review from a team as a code owner January 7, 2026 03:16

BesikiML linked an issue Jan 7, 2026 that may be closed by this pull request

[Feature]: Remorph Reconcile fails to run on serverless compute #1438

Open

1 task

BesikiML temporarily deployed to tool January 7, 2026 03:16 — with GitHub Actions Inactive

BesikiML self-assigned this Jan 7, 2026

Fixed detection of running on serverless compute

06d7f61

BesikiML temporarily deployed to tool January 7, 2026 16:24 — with GitHub Actions Inactive

Merge branch 'main' into 1438-feature-remorph-reconcile-fails-to-run-…

60c46d2

…on-serverless-compute

BesikiML temporarily deployed to tool January 7, 2026 16:46 — with GitHub Actions Inactive

fix: remove pylint disable and use specific exceptions

2ab2352

Use specific exception types instead of broad Exception catch to satisfy CI linter rules. Add cluster ID check for improved detection.

BesikiML temporarily deployed to tool January 7, 2026 16:54 — with GitHub Actions Inactive

refactor: use getAll() instead of conf.get() for serverless detection

a049243

Avoids CONFIG_NOT_AVAILABLE exceptions by fetching all configs at once. Passes all linter checks.

BesikiML temporarily deployed to tool January 7, 2026 17:54 — with GitHub Actions Inactive

fixed dict issue

78f2253

BesikiML temporarily deployed to tool January 7, 2026 18:21 — with GitHub Actions Inactive

fixed getAll call

0d416c1

BesikiML temporarily deployed to tool January 7, 2026 18:53 — with GitHub Actions Inactive

Added AnalysisException in the exept block

670940b

BesikiML temporarily deployed to tool January 7, 2026 19:21 — with GitHub Actions Inactive

Optimised _is_serverless function

8fc57b5

BesikiML temporarily deployed to tool January 7, 2026 20:31 — with GitHub Actions Inactive

m-abulazm reviewed Jan 8, 2026

View reviewed changes

m-abulazm requested changes Jan 9, 2026

View reviewed changes

Replace clusterType check with clusterNodeType which reliably

d55fc20

distinguishes serverless (CONFIG_NOT_AVAILABLE) from classic clusters.

BesikiML temporarily deployed to tool January 9, 2026 21:31 — with GitHub Actions Inactive

Merge branch 'main' into 1438-feature-remorph-reconcile-fails-to-run-…

bb37933

…on-serverless-compute

BesikiML temporarily deployed to tool January 9, 2026 21:51 — with GitHub Actions Inactive

BesikiML and others added 2 commits January 12, 2026 10:59

Clead the code

7c5b49c

Merge branch 'main' into 1438-feature-remorph-reconcile-fails-to-run-…

b82cb7d

…on-serverless-compute

BesikiML had a problem deploying to tool January 12, 2026 16:01 — with GitHub Actions Error

Merge branch '1438-feature-remorph-reconcile-fails-to-run-on-serverle…

909a6d8

…ss-compute' of github.com:databrickslabs/lakebridge into 1438-feature-remorph-reconcile-fails-to-run-on-serverless-compute

BesikiML had a problem deploying to tool January 12, 2026 16:02 — with GitHub Actions Error

Changed the loger info to debug

8525599

BesikiML temporarily deployed to tool January 12, 2026 16:03 — with GitHub Actions Inactive

BesikiML changed the title ~~Add serverless compute support with Unity Catalog volume persistence~~ Fix serverless compatibility by replacing cache() with conditional persistence Jan 12, 2026

BesikiML requested a review from m-abulazm January 12, 2026 16:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix serverless compatibility by replacing cache() with conditional persistence #2218

Fix serverless compatibility by replacing cache() with conditional persistence #2218

Uh oh!

BesikiML commented Jan 7, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 7, 2026 •

edited

Loading

Uh oh!

codecov bot commented Jan 7, 2026 •

edited

Loading

Uh oh!

m-abulazm Jan 8, 2026

Uh oh!

BesikiML Jan 8, 2026

Uh oh!

m-abulazm Jan 9, 2026

Uh oh!

BesikiML Jan 9, 2026

Uh oh!

m-abulazm Jan 9, 2026

Uh oh!

m-abulazm Jan 9, 2026

Uh oh!

m-abulazm Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterType", "")
	compute_type = self._spark.conf.get("spark.databricks.clusterUsageTags.clusterNodeType", "")

Fix serverless compatibility by replacing cache() with conditional persistence #2218

Are you sure you want to change the base?

Fix serverless compatibility by replacing cache() with conditional persistence #2218

Uh oh!

Conversation

BesikiML commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests

Uh oh!

github-actions bot commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

m-abulazm Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

BesikiML Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

m-abulazm Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

BesikiML Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

m-abulazm Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

m-abulazm Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

m-abulazm Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BesikiML commented Jan 7, 2026 •

edited

Loading

github-actions bot commented Jan 7, 2026 •

edited

Loading

codecov bot commented Jan 7, 2026 •

edited

Loading