Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle analytics publishing errors more gracefully #10185

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

whummer
Copy link
Member

@whummer whummer commented Feb 6, 2024

Motivation

In a support channel, we've seen error stack trace related to SSL verification when connecting to the analytics service (can happen if the user is behind a corporate proxy/firewall).

This behavior can be reproduced by setting analytics.localstack.cloud to an invalid IP address (where the TLS certificate doesn't match our hostname), then shutting down the container:

$ DOCKER_FLAGS='--add-host analytics.localstack.cloud:1.1.1.1' DEBUG=1 localstack start
...
Ready.

[CTRL-C]
...
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: MyHTTPSConnectionPool(host='analytics.localstack.cloud', port=443): Max retries exceeded with url: /v1/events (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'analytics.localstack.cloud'. (_ssl.c:1006)")))
LocalStack supervisor: localstack process (PID 15) returned with exit code 0

These logs should not be presented to the user, and the error also causes the shutdown hooks not to be executed properly on shutdown in stop_infra().

Changes

Add a try/except block to catch the exception, and log it with an INFO log message.

Let me know if you'd like to see this covered with a test - happy to try and wire something up, if we think it could be useful.

@whummer whummer requested a review from thrau as a code owner February 6, 2024 14:29
@whummer whummer added the semver: patch Non-breaking changes which can be included in patch releases label Feb 6, 2024
Copy link

github-actions bot commented Feb 6, 2024

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 22m 7s ⏱️ -40s
2 623 tests ±0  2 374 ✅ ±0  249 💤 ±0  0 ❌ ±0 
2 625 runs  ±0  2 374 ✅ ±0  251 💤 ±0  0 ❌ ±0 

Results for commit 2d81292. ± Comparison against base commit e2e054a.

♻️ This comment has been updated with latest results.

Copy link
Member

@alexrashed alexrashed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for jumping on this! The fix is straight forward and minimal. 💯

@coveralls
Copy link

Coverage Status

coverage: 83.84% (-0.002%) from 83.842%
when pulling 9d504f6 on analytics-errors
into e2e054a on master.

Copy link
Member

@thrau thrau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for identifying the culprit! Before we merge I just want to understand why we are using a AnalyticsClientPublisher directly, rather than using localstack.utils.analytics.log, which already handles errors and logging.
@dominikschubert do you remember?

Comment on lines +123 to +125
publisher.publish(
[Event(name="ls:usage_analytics", metadata=metadata, payload=aggregated_payload)]
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason for not using the global analytics bus here? it already does buffering and graceful handling of errors

localstack/utils/analytics/usage.py Outdated Show resolved Hide resolved
@alexrashed alexrashed added this to the 3.4 milestone Mar 26, 2024
@alexrashed alexrashed modified the milestones: 3.4, 3.5 Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants