Repository navigation
fix(triple): release server DATA discarded before decoder handoff - #16585
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 3.3 #16585 +/- ##
============================================
+ Coverage 60.99% 61.14% +0.14%
- Complexity 11756 11762 +6
============================================
Files 1953 1953
Lines 89286 89348 +62
Branches 13478 13486 +8
============================================
+ Hits 54462 54628 +166
+ Misses 29220 29103 -117
- Partials 5604 5617 +13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
WebSocket regression fixed and locally revalidatedCommit 2abcce2 fixes the first CI run's real WebSocket regression: the pending-input wrapper hid FinalFragment metadata. The marker-preserving adapter now lives in the WebSocket listener, so the generic HTTP/gRPC listener does not acquire a mandatory dependency on the optional WebSocket module. Original-message close-once ownership is unchanged. Before pushing this update, I ran:
Updated GitHub CI remains pending; the previous failed run must not be treated as passed. |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
onDataFinally(message)is no longer guaranteed to run whenexecutor.execute(...)throws… · New The pending-input registry removes ownership before callingsuper.onDataMessage(...)(which… · NewclosePendingInputs()(and theinputClosed && processingInputs == 0path inonDataReceived)… · New The repeated fully-qualifiedorg.apache.dubbo.remoting.websocket.FinalFragmentreferences make… · New
What changed in this PR
Fixes server-side HTTP/2 DATA body ownership so reference-counted inputs aren’t leaked when executor submission is rejected or discarded before decoder handoff.
Changes:
- Add transport-level pending-input registry and lifecycle gating to release DATA bodies on close/cancel before decoder takes ownership.
- Extend server transport listener flow with
onDataReceived(...)/onDataMessage(...)hooks and explicitRejectedExecutionExceptionhandling. - Add comprehensive ownership/concurrency regression tests, plus WebSocket final-fragment coverage.
| File | Description |
|---|---|
| dubbo-rpc/dubbo-rpc-triple/src/main/java/org/apache/dubbo/rpc/protocol/tri/h12/AbstractServerTransportListener.java | Adds onDataReceived/onDataMessage hooks and catches executor rejection for DATA delivery. |
| dubbo-rpc/dubbo-rpc-triple/src/main/java/org/apache/dubbo/rpc/protocol/tri/h12/http2/GenericHttp2ServerTransportListener.java | Implements pending-input ownership registry and transport-driven cleanup on close/cancel. |
| dubbo-rpc/dubbo-rpc-triple/src/main/java/org/apache/dubbo/rpc/protocol/tri/websocket/DefaultWebSocketServerTransportListener.java | Wraps FinalFragment inputs so fragment metadata survives pending-input wrapping. |
| dubbo-rpc/dubbo-rpc-triple/src/test/java/org/apache/dubbo/rpc/protocol/tri/h12/http2/GenericHttp2ServerTransportListenerTest.java | New regression suite validating ref-count ownership across rejection/discard/close/cancel and WebSocket fragments. |
Review effort: Lite
| } | ||
| }); | ||
| } catch (RejectedExecutionException e) { | ||
| onError(message, e); |
There was a problem hiding this comment.
Confirmed and fixed in f643263. The rejection path now invokes onDataFinally(message) in a finally block around onError(message, e), so finalization is preserved even if the error handler throws. The rejection regression failed with 0 finalizations before the change and now asserts exactly 1, alongside exactly-once body release. The full Triple test package and real WebSocket checks passed before pushing; details are in the validation follow-up.
| synchronized (inputLock) { | ||
| if (inputClosed) { | ||
| return; | ||
| } | ||
| // The decoder owns accepted input, including any bytes buffered for later requests. | ||
| // Only inputs that have not crossed this boundary belong to the transport registry. | ||
| pendingInputs.remove((PendingInput) message.getBody()); | ||
| } | ||
| super.onDataMessage(message, listener); |
There was a problem hiding this comment.
I added listenerFailureBeforeConsumptionReleasesInputOnce to check this path directly. It also passes on the previous implementation: a callback that throws before consuming the body reaches the existing onError(message, ...) cleanup and releases the original body exactly once. I am retaining removal before the callback. The decoder can already own buffered input while the callback is still running; delaying removal until callback success would let concurrent transport cancellation close decoder-owned input. The accepted-partial and read-in-progress cancellation controls cover that distinction. The registry therefore tracks only inputs that have not been handed to the message listener, rather than claiming ownership until a successful callback return.
| private void closeInputs() { | ||
| synchronized (inputLock) { | ||
| inputClosed = true; | ||
| if (processingInputs == 0) { | ||
| closePendingInputs(); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| private void closePendingInputs() { | ||
| // An already-started callback may still be preparing its DATA message. Defer | ||
| // pre-delivery cleanup until it returns; accepted input belongs to the decoder. | ||
| for (PendingInput input : pendingInputs) { | ||
| closeInput(input); | ||
| } | ||
| } |
There was a problem hiding this comment.
Confirmed and fixed in f643263. Pending inputs are now detached into a snapshot under inputLock and closed after releasing the lock. Cleanup still waits until all started DATA callbacks leave the handoff, so it does not close a body being prepared for the decoder. The new closeCallbackCanWaitForConcurrentLateData regression times out on the prior lock-held implementation and passes with this change. I also added a red/green regression for unchecked close failures: each input is closed independently, so a RuntimeException from one close does not abandon the remaining detached inputs. Repeated close still releases each original body only once.
| @Override | ||
| protected PendingInput newPendingInput(Http2InputMessage message) { | ||
| return message.getBody() instanceof org.apache.dubbo.remoting.websocket.FinalFragment | ||
| ? new PendingFragmentInput(message) | ||
| : super.newPendingInput(message); | ||
| } | ||
|
|
||
| private final class PendingFragmentInput extends PendingInput | ||
| implements org.apache.dubbo.remoting.websocket.FinalFragment { | ||
| private PendingFragmentInput(Http2InputMessage message) { | ||
| super(message); | ||
| } | ||
|
|
||
| @Override | ||
| public boolean isFinalFragment() { | ||
| return ((org.apache.dubbo.remoting.websocket.FinalFragment) in).isFinalFragment(); | ||
| } | ||
| } |
There was a problem hiding this comment.
Applied in f643263: FinalFragment is imported once and the repeated fully qualified references are removed. The marker-preserving adapter remains confined to the WebSocket-specific listener; the generic HTTP/gRPC listener still has no dependency on the optional WebSocket module. Formatting checks and the real Netty/Tomcat WebSocket sample tests passed before pushing.
|
Follow-up in f643263 addresses the review comments after local validation:
Before pushing, the rejection-finalization and lock-held-close tests reproduced their failures; the snapshot close-failure test also failed before the extra fix. Final validation on OpenJDK 25: mvn -q -pl dubbo-rpc/dubbo-rpc-triple -am -Pjacoco \
'-Dtest=org.apache.dubbo.rpc.protocol.tri.**.*Test,GenericHttp2ServerTransportListenerTest' \
-Dsurefire.failIfNoSpecifiedTests=false -Dspotless.action=apply test jacoco:report
mvn -q -pl dubbo-rpc/dubbo-rpc-triple -am -DskipTests -Dspotless.action=check validate
git diff --checkTriple reports 486 tests, 0 failures, 0 errors, 2 existing disabled WriteQueueTest methods; the ownership test class has 18 passing tests. Formatting and whitespace checks passed. Independent review found no remaining blocking issue. The real dubbo-samples Triple WebSocket Netty/Tomcat clients also pass 16 tests each on JDK 21 and JDK 25, rerun after the final fix. These runs use the checkout core classes with released 3.3.6 Spring Boot bootstrap dependencies and no registry; they do not replace the upstream Nacos/dependency matrix or whole-repository CI. The previous head CI was green; new-head CI results are separate from these local checks. All four review threads have replies and remain available for reviewer confirmation. Could a maintainer review this update when available? AI assistance was used for the follow-up and validation. |
EarthChen
left a comment
There was a problem hiding this comment.
Thanks for the detailed report and the thorough test suite. I reproduced the leak on the base commit 1cf125c421 (8 of 18 tests fail with refcount 1) and confirmed the fix turns them all green, plus a full dubbo-rpc-triple module run.
Before this moves to a final review, I'd like to explore whether the ownership machinery can be simplified. As written it is hard to review: a registry set, a lifecycle lock, a closed flag and an in-flight counter interact across four methods, and it is not obvious from the code which parts are load-bearing.
One data point: grpc-java has the same class of bug (grpc/grpc-java#12890, fix PR grpc/grpc-java#12891, still unmerged — the maintainers want it extended into the full failure-notification design tracked in grpc/grpc-java#636). Their fix is a per-call-site catch (RejectedExecutionException) that closes the message producer and rethrows. That is sufficient there because (a) their SerializingExecutor always rethrows — the silent shutdown-drop is a dubbo-specific addition from #15122 — and (b) their deframer runs on the transport thread, so the executor task only ever holds one assembled message. Dubbo decodes inside the executor task, so a submission-site catch cannot cover the queued-data windows (I verified the rejection-only variant still fails 7 of these 18 tests); some registry remains necessary here.
That said, I prototyped a leaner variant that passes this same 18-test suite plus a full module run: replace the lock and the counter with a per-input state machine (QUEUED → DELIVERING → HANDED_OFF / CLOSED, one AtomicInteger per input, no shared lock):
stateDiagram-v2
[*] --> QUEUED: onDataReceived() registers input
QUEUED --> DELIVERING: task starts (CAS)
QUEUED --> CLOSED: close() / cancelByRemote()
DELIVERING --> HANDED_OFF: handoff to decoder (CAS)
DELIVERING --> CLOSED: task finally, not handed off
HANDED_OFF --> [*]: decoder owns input, transport never touches it
CLOSED --> [*]: released once
close() / cancelByRemote()
inputClosed = true
for each input still QUEUED: CAS -> CLOSED, close body once
executor task starts : CAS QUEUED -> DELIVERING (lose -> input was closed, skip)
handoff to message listener: CAS DELIVERING -> HANDED_OFF, leave registry
task finally : CAS DELIVERING -> CLOSED (lose -> decoder or drain owns it)
The line count is about the same, but the concurrency reasoning becomes local to each input, and the event-loop path no longer takes any lock. One behavioral nuance: the global lock in the current design guarantees that no delivery can start after close() returns; the per-input version has a benign race where a delivery that already passed its checks completes into a closing stream — the same behavior as before this PR, and the decoder/stream teardown releases it.
Could you either consider this direction, or add a short design note (class javadoc) explaining why the registry, the lock, the closed flag and the counter each need to exist? As a reviewer it is currently hard to tell which parts are load-bearing.
(For context: the rejection-only #16588 was closed in favor of this PR.)
|
Following the design feedback in #16585 (review), c698bf4 adopts the per-input state approach. The registry remains necessary for silent discard, while the shared lifecycle lock and in-flight counter are gone. Class javadoc now explains QUEUED / DELIVERING / HANDED_OFF / CLOSED ownership and registration racing with the terminal drain. Terminal cleanup claims only QUEUED inputs. Each started callback closes its own still-DELIVERING input in finally. Handoff removes transport ownership before invoking the listener; decoder-owned buffered inputs and original-message close-once behavior are preserved. A handoff that already passed its terminal check retains the existing decoder lifecycle; this update does not claim to fix every decoder-specific termination race. Before the refactor, the expanded 23-test fixture had exactly two failures: queued DATA was not released promptly on close/cancel while another callback was preparing. Afterward all 23 pass, including the original buffered-input/read-in-progress and post-handoff controls, plus registration-versus-close, preparation failure and error-handler failure. Before pushing, the Triple protocol package ran 491 tests with 0 failures/errors and 2 existing disabled methods. A final inventory audit then added TriRpcStatusTest, StatusRpcExceptionTest and StubInvocationUtilTest, which sit outside that package: complete-module verification after the push passes 518 tests, 0 failures/errors, with the same 2 disabled methods and no further code changes. Every annotated module test class has a corresponding report. Real Netty/Tomcat WebSocket clients pass 16 tests each on JDK 21 and 25 with the same disclosed no-registry/sample-bootstrap setup. Formatting, whitespace checks and independent review passed before pushing. The PR body now describes the final design and test scope. Local coverage covers the added generic-listener executable lines; one default base-hook forwarding line and two partial short-circuit conditions remain disclosed rather than claiming 100% branch coverage. New-head CI will be checked separately. Please review this update when available. |
EarthChen
left a comment
There was a problem hiding this comment.
Verified on base 1cf125c421 vs head c698bf42e2 (JDK 17):
- Base + this PR's test suite (only adapted to drop the
newPendingInputhook that does not exist on base):Tests run: 23, Failures: 11— exactly the claimed symptom: bodies still at refCnt 1 after executor rejection, silent discard bySerializingExecutoron a shut-down executor, queued bodies not released on close/cancel while another callback is preparing, and loss ofFinalFragmentmetadata after wrapping. Red at base, green at head (23/23); fulldubbo-rpc-triplerun: 518 tests, 0 failures, 2 pre-existing@Disabled. - Mechanism confirmed at the receiving call site:
AbstractServerTransportListener.onDatacloses the body only inside the executor task, so aRejectedExecutionExceptionat submission — or the silent task drop inSerializingExecutor.schedulewhen the backing executor is shut down (introduced by #15122) — leaves the refcountedByteBufInputStreambody (overframe.content()in the HTTP/2 and HTTP/3 codecs) unreleased. - The per-input QUEUED/DELIVERING/HANDED_OFF/CLOSED design addresses the earlier review feedback: no shared lifecycle lock on the event-loop path, terminal cleanup claims only QUEUED inputs, and each started callback closes its own DELIVERING input in finally; close-once is enforced by
getAndSet(CLOSED)/CAS before closing the original message. The remaining race windows are disclosed in the class javadoc and match pre-PR behavior. - Submission-site
RejectedExecutionExceptionhandling now finalizes DATA viaonError+onDataFinally; for the generic HTTP/2 listener this converts a raw REE propagation into a stream-level error response, consistent with the existing in-task error path. - Sibling sites: HTTP/1 and servlet DATA use non-refcounted byte-array bodies and inherit the rejection catch through the base class without needing the registry; the WebSocket adapter preserves
FinalFragmentso fragment accumulation keeps working. Client-side DATA handling is out of scope here and could be a follow-up if a similar window exists.
Looks good overall — approving.



Fixes #16584
What changes were proposed in this pull request?
Triple server DATA rejected or silently discarded before decoder execution can retain its reference-counted input even after stream closure. Register close-once input wrappers before executor submission and route explicit rejection through the existing message error cleanup.
The final design uses a registry, a volatile terminal flag and one atomic state per input: QUEUED -> DELIVERING -> HANDED_OFF, or CLOSED. Close/remote cancellation claims only QUEUED inputs. A started callback closes its own DELIVERING input in finally if it never handed off. At handoff, registry ownership ends before the message listener runs, so decoder-owned buffered input is not closed by the terminal drain. Closing the original message is guarded once. Registration rechecks the terminal flag after insertion to cover a concurrent drain.
The shared lifecycle lock and in-flight counter have been removed. The class javadoc explains the ownership transitions and the registration race. Original frame id, end-stream status and original-message close behavior are retained. The FinalFragment-preserving adapter stays in the WebSocket listener; the generic HTTP/gRPC listener has no optional WebSocket dependency. SerializingExecutor behavior and unrelated task APIs are unchanged.
Why are the changes needed?
A decode task is not a reliable owner until it runs. Saturation may reject it, while backing-executor shutdown may silently discard it. Decoder cleanup cannot release input it never received; a submission-site catch alone does not cover silent discard or queued input.
How was this patch tested?
Base: 3.3 at 1cf125c. The initial unchanged-base regression failed four of five tests with an unreleased reference; the accepted-partial control passed. Subsequent human review independently reproduced the expanded fixture failures and confirmed the fix.
Before the per-input refactor, the current 18 ownership tests passed. Five additional invocations brought the fixture to 23: two terminal-close/remote-cancel cases failed because an unrelated queued body waited for the preparing callback; the error-handler failure, preparation failure and registration-versus-close controls passed. After the refactor, all 23 pass. The original buffered-A/read-in-B cancellation, accepted-partial, handoff-before-decode, close-exception, WebSocket-fragment and exactly-once controls remain.
Before pushing c698bf4: Triple protocol package 491 tests, zero failures/errors, two existing disabled WriteQueueTest methods; ownership 23/23 pass. Formatting, whitespace checks and independent code review pass. Real Netty/Tomcat WebSocket sample clients pass 16 tests each on JDK 21 and JDK 25. Those sample runs use checkout core classes with released 3.3.6 Spring Boot bootstrap dependencies and no registry; they do not replace the upstream Nacos/dependency matrix. No snapshot artifacts were installed locally.
Local JaCoCo covers all added executable lines in the generic listener and WebSocket adapter. The base default onDataReceived forwarding line is unexercised, and two short-circuit conditions remain partially branch-covered; full patch branch coverage is not claimed. Updated-head GitHub CI is separate from these local results.
A final test-inventory audit found three module test classes outside the protocol package. The selector above was extended to include TriRpcStatusTest, StatusRpcExceptionTest and StubInvocationUtilTest. This additional complete-module verification, run after the push without further code changes, passes 518 tests with zero failures/errors and the same two existing disabled methods. Every annotated test class in the module has a corresponding report; the earlier 491 count refers specifically to the protocol-package run.
Scope
The ownership fixture uses real executors, a gRPC streaming decoder and response-observer lifecycle, with application routing isolated and an H2 channel test double. It is not a full TCP gRPC/load benchmark. Accepted input retains the existing decoder lifecycle, including decoder-specific termination/handoff races; this PR does not synchronize all decoder read/close behavior. Client timeout handling (#16415 / #16418), general executor redesign and HTTP/3 validation are separate scope. No configuration change is required.
AI assistance
AI assistance was used for the implementation, tests, independent review and description. The protocol-package and WebSocket checks were run before pushing; the additional complete-module check is identified above.