Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky specs #2522

Open
solnic opened this issue Jan 24, 2025 · 0 comments
Open

Fix flaky specs #2522

solnic opened this issue Jan 24, 2025 · 0 comments
Assignees

Comments

@solnic
Copy link
Collaborator

solnic commented Jan 24, 2025

Issue Description

This issue gathers random failures we get from some flaky tests.

Failure 01

  • Ruby 3.3
Failures:

  1) Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error captures client report for transaction event with span counts
     Failure/Error: expect(client.transport).to have_recorded_lost_event(:network_error, 'transaction')
       expected #<Sentry::HTTPTransport:0x00007fad0197b5e8 @logger=#<Logger:0x00007fad0197b868 @level=0, @progname=ni...nt_reports=true, @discarded_events={}, @last_client_report_sent=2025-01-24 10:01:40.480909663 +0000> to have recorded lost event :network_error and "transaction"
     # ./spec/sentry/client/event_sending_spec.rb:492:in `block (5 levels) in <top (required)>'
     # ./spec/sentry/client/event_sending_spec.rb:359:in `block (4 levels) in <top (required)>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block (2 levels) in setup'

  2) Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error captures client report for error event
     Failure/Error: expect(client.transport).to have_recorded_lost_event(:network_error, 'error')
       expected #<Sentry::HTTPTransport:0x00007fad00bd46b0 @logger=#<Logger:0x00007fad00bd48e0 @level=0, @progname=ni...nt_reports=true, @discarded_events={}, @last_client_report_sent=2025-01-24 10:01:41.297596915 +0000> to have recorded lost event :network_error and "error"
     # ./spec/sentry/client/event_sending_spec.rb:486:in `block (5 levels) in <top (required)>'
     # ./spec/sentry/client/event_sending_spec.rb:359:in `block (4 levels) in <top (required)>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block (2 levels) in setup'

  3) Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error swallows and logs Sentry::ExternalError (caused by transport's networking error)
     Failure/Error: expect(string_io.string).to match(/Event sending failed: Failed to open TCP connection/)

       expected "D, [2025-01-24T10:01:41.915649 #2632] DEBUG -- sentry: Initializing the Sentry background worker wit...entry: [Transport] Sending envelope with items [event] 14d88737df1a4ab7b8581c7189f963e2 to Sentry\n" to match /Event sending failed: Failed to open TCP connection/
       Diff:
       @@ -1,3 +1,5 @@
       -/Event sending failed: Failed to open TCP connection/
       +D, [2025-01-24T10:01:41.915649 #2632] DEBUG -- sentry: Initializing the Sentry background worker with 2 threads
       +D, [2025-01-24T10:01:41.915735 #2632] DEBUG -- sentry: Sentry HTTP Transport will connect to http://sentry.localdomain/
       +D, [2025-01-24T10:01:41.919898 #2632] DEBUG -- sentry: [Transport] Sending envelope with items [event] 14d88737df1a4ab7b8581c7189f963e2 to Sentry
     # ./spec/sentry/client/event_sending_spec.rb:471:in `block (5 levels) in <top (required)>'
     # ./spec/sentry/client/event_sending_spec.rb:359:in `block (4 levels) in <top (required)>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/3.3.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block (2 levels) in setup'

Finished in 12.55 seconds (files took 0.73758 seconds to load)
1076 examples, 3 failures, 1 pending

Failed examples:

rspec ./spec/sentry/client/event_sending_spec.rb:489 # Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error captures client report for transaction event with span counts
rspec ./spec/sentry/client/event_sending_spec.rb:483 # Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error captures client report for error event
rspec ./spec/sentry/client/event_sending_spec.rb:467 # Sentry::Client integrated error handling testing with HTTPTransport #capture_event when sending events in background causes error swallows and logs Sentry::ExternalError (caused by transport's networking error)

Failure 02

  • jruby / rack3
  1) Sentry::Client#event_from_exception sets threads interface without stacktrace
     Failure/Error: expect(thread[:name]).to eq("Thread 1")

       expected: "Thread 1"
            got: nil

       (compared using ==)
     # ./spec/sentry/client_spec.rb:370:in `block in <main>'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/jruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/jruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/jruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/jruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block in setup'

Finished in 31.92 seconds (files took 4.12 seconds to load)

Failures 03

Ruby 2.6

Randomized with seed 4066

/home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport/http_transport.rb:69:in `rescue in send_data': end of file reached (Sentry::ExternalError)
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport/http_transport.rb:32:in `send_data'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport.rb:65:in `send_envelope'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport.rb:173:in `flush'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/client.rb:101:in `flush'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry-ruby.rb:271:in `close'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry-ruby.rb:245:in `block in init'
/opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/protocol.rb:225:in `rbuf_fill': end of file reached (EOFError)
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/protocol.rb:191:in `readuntil'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/protocol.rb:201:in `readline'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http/response.rb:40:in `read_status_line'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http/response.rb:29:in `read_new'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:1509:in `block in transport_request'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:1506:in `catch'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:1479:in `request'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/net/http.rb:35:in `request'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport/http_transport.rb:52:in `block in send_data'
	from /opt/hostedtoolcache/Ruby/2.6.10/x64/lib/ruby/2.6.0/net/http.rb:920:in `start'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport/http_transport.rb:49:in `send_data'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport.rb:65:in `send_envelope'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/transport.rb:173:in `flush'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry/client.rb:101:in `flush'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry-ruby.rb:271:in `close'
	from /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/lib/sentry-ruby.rb:245:in `block in init'
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
Coverage report generated for RSpec to /home/runner/work/sentry-ruby/sentry-ruby/sentry-ruby/coverage/coverage.xml. 10539 / 10898 LOC (96.71%) covered.
/opt/hostedtoolcache/Ruby/2.6.10/x64/bin/ruby -I/home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.6.0/gems/rspec-core-3.13.2/lib:/home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.6.0/gems/rspec-support-3.13.2/lib /home/runner/work/sentry-ruby/sentry-ruby/vendor/bundle/ruby/2.6.0/gems/rspec-core-3.13.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb  --exclude-pattern spec/isolated/\*\*/\*_spec.rb --order rand failed
@solnic solnic self-assigned this Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant