Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unobserved task exception thrown by an active client streaming operation when the server is killed #349

Open
tobi1449 opened this issue Nov 18, 2024 · 0 comments

Comments

@tobi1449
Copy link

When a server is killed while an client-server streaming operation is open, a GC run will cause an unobserved task exception, even if the exception thrown by the operation call itself is caught.

To reproduce:

  1. Start a streaming operation from a gRPC client to a gRPC server within a try-catch block
  2. Kill the server
  3. This results in a IncompleteSendRpcException from the gRPC call, caught properly by the try-catch block
  4. Force a GC run
  5. This results in an RpcException, only caught via TaskScheduler.UnobservedTaskException
Unobserved Task exception: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Strea
m removed", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1731967532.940000000","description":"Error received from peer ipv6:[::1]:5001","file":"..\..\..\src\core\lib\surface\call.cc","file_line":953,"grpc_message":"Stream removed","grpc_status":2}") ---> Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1731967532.940000000","description":"Error received from peer ipv6:[::1]:5001","file":"..\..\..\src\core\lib\surface\call.cc","file_line":953,"grpc_message":"Stream removed","grpc_status":2}
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
---> (Inner Exception #0) Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Stream removed", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1731967532.940000000","description":"Error received from peer ipv6:[::1]:5001","file":"..\..\..\src\core\lib\surface\call.cc","fil
e_line":953,"grpc_message":"Stream removed","grpc_status":2}") ---> Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1731967532.940000000","description":"Error received from peer ipv6:[::1]:5001","file":"..\..\..\src\core\lib\surface\call.cc","file_line":953,"grpc_message":"Stream removed","grpc_status":2}

I've created a small sample project at tobi1449/protobuf-net.grpc-bug.
To reproduce simply start NetFrameworkServer and NetFrameworkClient and kill the server.

As a workaround, I've found that setting the IgnoreStreamTermination flag, as described in the IncompleteSendRpcException, also suppresses the unobserved task exception.
It works for me, but the client not getting an exception or notification that the server is offline is a bit of a bummer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant