Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix serialization for sqs http calls #10732

Merged
merged 8 commits into from May 7, 2024
Merged

fix serialization for sqs http calls #10732

merged 8 commits into from May 7, 2024

Conversation

baermat
Copy link
Member

@baermat baermat commented Apr 26, 2024

Motivation

When performing API calls for sqs like "ReceiveMessage" directly via HTTP outside the CLI, the response was malformed. It was malformed in general, and had an additional error for json messages. This PR addresses these issues.

Changes

  • fixes the json parsing by taking into account the escaping of a potential json dump

TODO

What's left to do:

  • clarify if the asymmetry between the marker conversions has a reason. It seems odd that the code goes to great lengths explaining the reason for all the conversion, only to have these two ways differ -> not doing this results in strange conversion errors for \r on non-json messages.
  • while tackling this, I found another bug: the response for ReceiveMessage is potentially further malformed. A usual cli call returns {Messages:[{<msg_1>}, {<msg_2>}, ...]} but we return {Message:{<msg_1>}} if there is one message, and {Message:[{<msg_1>}, {<msg_2>}, ...]} if there are more than one (note the missing plural s of Message). AWS' response is also different with a "direct" http call, but it is only the case that is different ("Messages" vs "messages"). How to proceed here? It seems to be an error in the automatic response creation of LS

@baermat baermat added this to the 3.5 milestone Apr 26, 2024
@baermat baermat added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Apr 26, 2024
Copy link

github-actions bot commented Apr 26, 2024

S3 Image Test Results (AMD64 / ARM64)

  2 files  ±0    2 suites  ±0   3m 21s ⏱️ +20s
398 tests ±0  346 ✅ ±0   52 💤 ±0  0 ❌ ±0 
796 runs  ±0  692 ✅ ±0  104 💤 ±0  0 ❌ ±0 

Results for commit 38a9aad. ± Comparison against base commit 0f7d49f.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Apr 26, 2024

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 36m 56s ⏱️ - 1m 52s
2 928 tests +2  2 634 ✅ ±0  294 💤 +2  0 ❌ ±0 
2 930 runs  +2  2 634 ✅ ±0  296 💤 +2  0 ❌ ±0 

Results for commit 38a9aad. ± Comparison against base commit 0f7d49f.

This pull request removes 5 and adds 7 tests. Note that renamed tests count towards both.
tests.aws.services.events.test_events.TestEvents ‑ test_event_pattern
tests.aws.services.events.test_events.TestEvents ‑ test_put_event_without_source
tests.aws.services.events.test_events.TestEvents ‑ test_put_events_with_nested_event_pattern
tests.aws.services.events.test_events.TestEvents ‑ test_put_events_with_values_in_array
tests.aws.services.events.test_events.TestEvents ‑ test_put_target_id_validation
tests.aws.services.events.test_event_patterns ‑ test_event_pattern_source
tests.aws.services.events.test_events.TestEventPattern ‑ test_put_events_pattern_nested
tests.aws.services.events.test_events.TestEventPattern ‑ test_put_events_pattern_with_values_in_array
tests.aws.services.events.test_events.TestEventTarget ‑ test_put_target_id_validation
tests.aws.services.events.test_events.TestEvents ‑ test_put_events_without_source
tests.aws.services.sqs.test_sqs.TestSqsProvider ‑ test_marker_serialization_json_protocol["{\\"foo\\": \\"ba\\rr\\"}"]
tests.aws.services.sqs.test_sqs.TestSqsProvider ‑ test_marker_serialization_json_protocol[{"foo": "ba\rr", "foo2": "ba&quot;r&quot;"}]
This pull request skips 1 test.
tests.aws.services.events.test_events.TestEvents ‑ test_put_event_without_detail

♻️ This comment has been updated with latest results.

@baermat baermat added semver: patch Non-breaking changes which can be included in patch releases and removed semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases labels Apr 29, 2024
@baermat baermat marked this pull request as ready for review May 7, 2024 13:21
Copy link
Member

@alexrashed alexrashed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoof, that must have been terrible to pin down! Really great catch! Thanks a lot for tackling this! 🦸🏽
The implementation looks good (as good as it can be on my initial hacky solution), the AWS validated test is great to avoid a regression in the future, and to have a kick-start when tackling the other issue you detected. 🚀

Copy link
Member

@thrau thrau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and works great, also via the dev endpoint. Thanks @baermat, nice job tracking this down! 💯

@thrau thrau merged commit 9d0093a into master May 7, 2024
37 checks passed
@thrau thrau deleted the sqs-json-parsing branch May 7, 2024 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants