-
-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High memory usage from ISpan[] and SpanTracer #3303
Comments
This seems to have resolved the issue, or at least greatly reduced the impact. |
Hey @danzel , SentryHttpMessageHandler creates one span per requests here: sentry-dotnet/src/Sentry/SentryMessageHandler.cs Lines 85 to 97 in 0ee3904
As far as I'm aware, the only thing that holds a reference to those spans is the transaction itself: sentry-dotnet/src/Sentry/TransactionTracer.cs Line 302 in a448cc4
The transaction in an ASP.NET Core application would ordinarily be created by the Sentry Tracing middleware and represents the inbound HttpRequest to your server. So the spans get cleaned up and garbage collected once the Transaction ends.
I wonder if that's our issue. We probably wouldn't want to create a transaction and hold it open for every long lived websocket connection. I'm not sure if or how Sentry was designed to work with WebSockets though. @danzel what is creating the websockets? Are you using SignalR or something else? I'm wondering if we could try to reproduce this. |
We use web sockets without signalr. In our actual app we use a middleware to receive them See the first example here Here is a test app for you @jamescrosswell , it uses a controller to receive the websocket, and reproduces the same: Usage: Open Client.sln Client makes 100 websockets, connects to the server, periodically sends messages over them. Thanks! |
Awesome, thank you so much @danzel ! That should give us enough to go on 👍🏻 |
Hey @danzel, Sorry it took me a while to carve out some time to look into this. I'm not actually seeing any kind of memory leak. This is what I'm getting when running a memory profile in Rider: So yes, at first, the process appears to be allocating incrementally more and more memory... on my machine it capped out at about 1GB. However after a couple of minutes (once everything is warmed up) it actually released most of that memory and stabalised at ~460MB memory usage... including "unmanaged memory" (which I think is just memory that .NET set aside to be able to write to if/when it needed, to avoid fragmentation). Excluding the unmanaged memory, it's hovering at around 150-300MB of memory usage. |
So this is not even a question of whether we have a memory leak but about general memory consumption. We did some profiling last fall to ease up on the memory consumption (specifically lots of nested spans within one request). But afaik we have not done any benchmarking or profiling for this specific use case. |
I think that's probably true yes. If I change the client to create 10,000 web socket connections like @danzel suggested, even with Sentry disabled memory consumption on the web server continues to grow until the application simply falls over (which doesn't take long on my machine). The difference in memory consumption between Sentry enabled vs Sentry disabled with only 100 active web sockets seems significant although that might just be relative. Relative to everything else that's going on in this application (which isn't much) Sentry uses a lot of memory. A real world application might open database connections, perform linq queries etc. at which point the overhead that Sentry puts on memory consumption might seem relatively small. Still, it might be worth creating a benchmark that creates a test server and a bunch of web socket connections to the test server... we could then try to minimise Sentry's memory footprint in that scenario. |
What you've said lines up with my thoughts too, not a memory leak, just more memory usage than expected. |
Package
Sentry.AspNetCore
.NET Flavor
.NET Core
.NET Version
8
OS
Any (not platform specific)
SDK Version
4.2.1
Self-Hosted Sentry Version
No response
Steps to Reproduce
We have an aspnetcore app that has ~10k concurrent longlived websocket connections.
We're having memory usage issues that appear to be related to sentry.
We previously turned down
MaxBreadcrumbs
to 5 which helped, but we are still having issues.Output from
dotnet-dump analyze
dumpheap -stat
From my investigation sentry is creating SpanTracer instances when these websocket connections cause http requests to be made (When we receive certain messages on the websocket that causes an http request to be made). Keeping around all of these spans is eating up all the memory.
I'm going to try remove the
SentryHttpMessageHandlerBuilderFilter
service and see if that helps. Any other suggestions?Thanks!
Expected Result
Memory usage doesn't grow to such a large level.
Actual Result
Memory fills up and the app spends all its time trying to GC.
The text was updated successfully, but these errors were encountered: