[Tracking] Arrow Flight SQL (ADBC): issues and improvements

[morningman]

This issue tracks Arrow Flight SQL-related problems in Apache Doris. If you encounter a problem, please open a new issue or leave a comment here.

Goal

Use this issue as the single index for user-visible bugs, protocol gaps, reliability work, and ecosystem improvements related to Doris Arrow Flight SQL and the ADBC or Flight SQL clients that use it.

ADBC is the client API and driver layer. Doris implements the Arrow Flight SQL server path that those clients call. This tracker covers both sides when a client report identifies a Doris interoperability or server behavior problem.

Foundational context: #25514 and #19812.

Status convention

• Open, unmerged, closed without a verified fix, duplicated but not closed, or otherwise needs follow-up.
• The referenced change was merged or the work item was confirmed complete. Backports are listed beside their primary PR.

Open or needs follow-up

Reliability, lifecycle, and correctness

• FE direct-memory leak during prepared-statement creation: [Bug] Arrow Flight SQL causes FE Direct Memory leak until OOM ("Cannot reserve ... direct buffer memory", Internal; Prepare) #65305. Current fix: [fix](arrow-flight-sql) Close temporary VectorSchemaRoot in createPreparedStatement to fix FE direct memory leak #65311. An earlier fix attempt, [fix](arrowflight) some resource unrelease bug #52245, was closed without merge. Add a repeat-query memory test and verify failed/cancelled statements and session teardown, not only the happy path.
• External-table split lifecycle across GetFlightInfo and DoGet: [Bug] Arrow Flight SQL + Iceberg: SplitSource released before DoGet (fetchSplitBatch crash) #62259 is closed, but its root-cause fix [fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan #64799 is still open. [fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan #64797 and its 4.1 backport branch-4.1: [fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan #64797 #64890 harden the error path so the BE does not crash; they explicitly do not keep the SplitSource alive. Do not consider the root cause complete until [fix](arrow-flight) Keep coordinator alive across GetFlightInfo/DoGet for external table scan #64799 or a replacement is merged and the required release branches are covered.
• Corrupted large string/JSON results: [Bug] Arrow Flight SQL: Corrupted StringArray serialization results in empty DataBuffer and invalid Offsets #61124 reports invalid StringArray offsets and an empty data buffer.
• DDL regression on 4.0.5: [Bug][Arrow Flight SQL] DDL statements via Flight SQL crash with getMysqlChannel RuntimeException after 4.0.5 CLIENT_DEPRECATE_EOF fix #62017. Proposed fix [fix](arrow-flight) Fix DDL via Flight SQL crashing with getMysqlChannel RuntimeException #62018 was closed without merge. Reproduce on supported 4.0/4.1 builds and land or replace the fix.
• BE crash with public_host and an ILB proxy: [Bug] BE segfault in Arrow Flight SQL DoGet when using public_host (ILB proxy) #62217. Separate the endpoint-routing problem from the Arrow/gRPC status-conversion crash path and verify both.
• LargeUtf8 offset handling can crash remote Doris reads: [bug](arrow) fix arrow type change cause read data coredump #62681.
• Complex-type fidelity: [fix](arrow-flight) Preserve array item nullability #64976 proposed preserving array child nullability and [test](be) Add Arrow array smallint layout tests #65044 added focused IPC tests, but both closed without merge. [test](regression) skip Arrow-incompatible complex outputs #65182 documents additional Arrow Map null-key and raw BITMAP incompatibilities by skipping those regression cases. Decide which cases require engine fixes and which require documented protocol limitations.
• Flight SQL JDBC RST_STREAM failures on 4.0.1: [Bug] Arrow Flight SQL Jdbc driver fails with RST_STREAM on Doris 4.0.1 #59304 was closed as stale without a linked fix. Reproduce against current supported versions or link the resolving change.
• Flight SQL with the distributed planner disabled: [Bug] FlightSQL throws IllegalArgumentException: null 4.0无法使用flightsql #57318 was closed as stale. [fix](fe) Fix remote Flight SQL result receiver initialization #63136 appears to address the same empty-result-receiver failure on master and branch-4.1: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 #63155 backports it to 4.1, but the 4.0 backport branch-4.0: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 #63154 closed without merge. Verify the version matrix and close the loop explicitly.
• Sessions remain in SHOW PROCESSLIST after an ADBC connection closes: [Bug] Arrow flight SQL process hanging after closing connection #36331 was closed as stale. [fix](connection)Fixes session-related data is not cleared when the connection is disconnected due to timeout. #55008/branch-3.1:[fix](connection)Fixes session-related data is not cleared when the connection is disconnected due to timeout. #55008 #55396 improve timeout cleanup and [fix](connection) Prevent timeout checker from stopping after an exception #65040 keeps the timeout checker alive, but explicit client-close behavior still needs a current-version verification.

Authentication, endpoint advertisement, and session semantics

• Duplicate authentication reports: FlightSQL Unable to authenticate user #61743, FlightSQL Unable to authenticate user #61744, and doris FlightSQL Unable to authenticate user #61757 all report authentication using user@0.0.0.0. Verify the merged fix [fix](auth)Fix arrow flight client ip auth #63506 and backports branch-4.0: [fix](auth)Fix arrow flight client ip auth #63506 #63591/branch-4.1: [fix](auth)Fix arrow flight client ip auth #63506 #63592 against the reported read-only user case, then close or deduplicate the three issues.
• FE cannot advertise a public Flight endpoint: [Bug] When arrow-flight is used and the FE private name name is not resolvable, the client cannot access the FE nodes #62538 requests FE equivalents of BE public_host and the Flight proxy port.
• FE/BE advertises 127.0.0.1:8050 to a remote JDBC client: [Bug] Connection refused: no further information: /127.0.0.1:8050 JDK1.8 connect error #59490 was closed as stale. Check whether it is the same endpoint-advertisement gap as [Bug] When arrow-flight is used and the FE private name name is not resolvable, the client cannot access the FE nodes #62538.
• Kubernetes and reverse-proxy topology: [Feature] Improving arrow flight sql with Kubernetes setup #44599 is an older, stale-closed precursor to [Bug] BE segfault in Arrow Flight SQL DoGet when using public_host (ILB proxy) #62217/[Bug] When arrow-flight is used and the FE private name name is not resolvable, the client cannot access the FE nodes #62538. Decide whether the remaining work belongs in Doris, Doris Operator, or both, and cross-link the owner issue.
• Doris Catalog session-variable passthrough in Arrow Flight mode: [fix](doris catalog)Support session passthrough for Doris catalog #59742.

Protocol coverage and architecture

• Variant over cross-cluster Arrow Flight: [Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type #61883. The discussion proposes a LargeUtf8 first step and leaves a more efficient representation as follow-up.
• Prepared-statement parameter binding: apache/arrow-java#1009 was closed upstream as a Doris server-side implementation gap. Create or link a Doris issue that defines the required Flight SQL parameter schema and execution behavior.
• Initial database/schema in a Flight SQL JDBC URL: apache/arrow-java#829 remains open. Determine client-versus-server ownership and document the supported connection option.
• ADBC ingestion: users requested adbc_ingest/DataFrame writes in [Feature] Doris support Arrow Flight SQL protocol #25514, but no dedicated Doris work item tracks scope, protocol methods, atomicity, or type coverage. Create one if ingestion remains on the roadmap.
• Protocol SPI extraction: [feat](protocol)extract protocol definitions (part 1) #60355.
• Optional BE Arrow Flight worker pool: [opt](arrow_flight) Add enable_arrow_flight configuration option #60388 adds an enable_arrow_flight switch so unused deployments do not allocate the pool.

Tests and ecosystem

• Arrow regression cleanup: [test](regression) Avoid selecting raw bitmap column in Arrow Flight suite #62536 remains open.
• Arrow JDBC close-time allocator leak: [fix](regression) Avoid prepared Arrow JDBC path in remote IP auth test #64024 changed the authentication regression test to avoid the prepared path, and [improvement](regression) Align regression framework arrow/netty with fe-core and target JDK17 #64677 explicitly says the intermittent close-time leak needs a separate fix. Create or link that follow-up and check whether newer Arrow/Netty versions resolve it.
• Spark column projection: apache/doris-spark-connector#333 reports that doris.read.fields is ignored in Arrow read mode.
• Spark array work cleanup: apache/doris-spark-connector#345 remains open although the narrower opt-in implementation #362 merged. Mark the older PR superseded or document the remaining element-type work.

Completed or merged during the review window

Core feature and query path

• Doris Catalog and cross-cluster query over Arrow Flight: [Feature] Support Doris Catalog #55274 and [Feature] doris cross-cluster query #57898.
• Connection cleanup and thread-local correctness: [fix](connection)Fixes session-related data is not cleared when the connection is disconnected due to timeout. #55008 with 3.1 backport branch-3.1:[fix](connection)Fixes session-related data is not cleared when the connection is disconnected due to timeout. #55008 #55396, plus [fix](connection)Fix kill connection will make current connection killed #55809.
• Non-destructive Flight ticket/query-id parsing: [fix] remove parse_id #57240 with 4.0 backport branch-4.0: [fix] remove parse_id #57240 #57330.
• Flight processor close ordering and noisy warning fix: [fix][arrow-flight-sql]Move ctx.clear() to the end of close method #57858 with 4.0 backport branch-4.0: [fix][arrow-flight-sql]Move ctx.clear() to the end of close method #57858 #58011, closing [Bug] many warn logs about arrow-flight-sql processor #57859.
• Queries containing & no longer produce malformed tickets: fix flight sql when contains & #58830 with 4.0 backport branch-4.0: fix flight sql when contains & #58830 #58884.
• Async result completion avoids dirty reads: [bugfix](arrowflight) should call done run in on_xxx method to make work in async mode #60282 with 4.0 backport branch-4.0: [bugfix](arrowflight) should call done run in on_xxx method to make work in async mode #60282 #60325.
• Remote Flight SQL result receiver initialization: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 with 4.1 backport branch-4.1: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 #63155.
• Arrow UTF8/String size limit corrected to the intended 2 GiB threshold: [fix](be) Correct Arrow UTF8/String size limit #63137 with backports branch-4.1: [fix](be) Correct Arrow UTF8/String size limit #63137 #63148 and branch-4.0: [fix](be) Correct Arrow UTF8/String size limit #63137 #63558.
• Timeout checking continues after one connection throws: [fix](connection) Prevent timeout checker from stopping after an exception #65040.

Authentication, diagnostics, and crash hardening

• Better malformed-ticket logging: log print flight sql ticket error #58811 with 4.0 backport branch-4.0: log print flight sql ticket error #58811 #58842.
• Workload-group values are populated in audit records for early query failures: [fix](audit log) fix empty workload_group value __internal_schema.audit_log #62651 with 4.1 backport branch-4.1: [fix](audit log) fix empty workload_group value __internal_schema.audit_log #62651 #62697.
• Arrow status initialization crash fix: [Fix](arrow flight) Fix arrow::Status inline static empty msg core #63191 with backports branch-4.0: [Fix](arrow flight) Fix arrow::Status inline static empty msg core (#63191) #63266/branch-4.1: [Fix](arrow flight) Fix arrow::Status inline static empty msg core (#63191) #63267. The build-mode root cause was also addressed by [Fix](thirdparty) Unify arrow cpp standard with be #64390 and documented upstream in apache/arrow#49781.
• Real client IP is used for Flight authentication: [fix](auth)Fix arrow flight client ip auth #63506 with backports branch-4.0: [fix](auth)Fix arrow flight client ip auth #63506 #63591/branch-4.1: [fix](auth)Fix arrow flight client ip auth #63506 #63592.
• External split-fetch failures now fail gracefully instead of crashing the BE: [fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan #64797 with 4.1 backport branch-4.1: [fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan #64797 #64890. This is only the robustness half of [Bug] Arrow Flight SQL + Iceberg: SplitSource released before DoGet (fetchSplitBatch crash) #62259; the lifecycle item remains open above.

Regression framework and ecosystem

• Remote-IP auth regression uses a stable JDBC statement path: [fix](regression) Avoid prepared Arrow JDBC path in remote IP auth test #64024 with backports branch-4.0: [fix](regression) Avoid prepared Arrow JDBC path in remote IP auth test #64024 #64047/branch-4.1: [fix](regression) Avoid prepared Arrow JDBC path in remote IP auth test #64024 #64049.
• Regression framework Arrow/Netty versions aligned with FE and JDK 17: [improvement](regression) Align regression framework arrow/netty with fe-core and target JDK17 #64677.
• JDBC complex values are materialized before result-set teardown: [fix](regression) Materialize JDBC array results #65084.
• Regression cases now account for Arrow-incompatible Map null keys and protocol-specific BITMAP bytes: [test](regression) skip Arrow-incompatible complex outputs #65182.
• Spark connector FE/BE load balancing: apache/doris-spark-connector#329.
• Spark connector Flight read logging: apache/doris-spark-connector#354.
• Spark connector opt-in native array reads: apache/doris-spark-connector#362, closing #341.
• Flink connector BackendClient typo that could affect Flight reads: apache/doris-flink-connector#622.
• Website corrected the Flink connector source.use-flight-sql default for current/4.x docs: apache/doris-website#3725.

Closed without merge or superseded attempts

Keep these for history so the same investigation is not repeated:

• Prepared-statement resource release: [fix](arrowflight) some resource unrelease bug #52245.
• Query profile and connection finalization: [fix](arrow-flight-sql) Fix arrow flight sql query profile and correct close #45416.
• Early Arrow regression experiments: [test] fix arrow flight regression-test #53284, [test] fix arrow flight regression-test 2 #53285, and [test] fix arrow flight regression-test 3 #53286.
• Alternative generic Arrow Flight Catalog implementation: [Feature](arrow_flight catalog) support arrow_flight catalog #59270. The merged Doris Catalog path is [Feature] Support Doris Catalog #55274/[Feature] doris cross-cluster query #57898.
• Logging experiment: test LOGs for arrow flight #61990.
• DDL regression fix: [fix](arrow-flight) Fix DDL via Flight SQL crashing with getMysqlChannel RuntimeException #62018.
• 4.0 result-receiver backport: branch-4.0: [fix](fe) Fix remote Flight SQL result receiver initialization #63136 #63154.
• Array child nullability and focused IPC tests: [fix](arrow-flight) Preserve array item nullability #64976 and [test](be) Add Arrow array smallint layout tests #65044.

Maintenance rules

• Add each new issue or PR under the closest topic and link it back to this tracker.
• Keep the primary fix and all required release-branch backports on the same line.
• Check an item only after the fix is merged and the affected supported versions are verified.
• If a stale-closed report still reproduces, reopen it or create a replacement and link both directions.
• Record user-visible protocol limitations in the Arrow Flight SQL documentation, not only in regression-test skips.