Skip to content

fix(kafka): Handle message indices in proto data also for Glue Schema Registry #1907

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 20, 2025

Conversation

phipag
Copy link
Contributor

@phipag phipag commented Jun 20, 2025

Issue #, if available: N/A

Description of changes:

Copy of #1906

Thanks to @karthikpswamy for contributing this logic!

Description of changes:

* Adding support for different schema registry types (Confluent and Glue)

* Improving the handling of message indices in protobuf data

* Fixing deserialization logic for different schema registry formats

* Adding appropriate tests for both Confluent and Glue schema registry scenarios

Schema Registry Type Detection and Deserialization Logic:

No Schema Registry Integration:

* When KafkaEvent contains no key/value metadata

* Uses entire byte array as raw data for deserialization

* AWS Glue Schema Registry:

Identified by 16-byte schema ID in key/value metadata

* Skips first byte of data

* Uses remaining bytes for Protobuf deserialization

* Confluent Schema Registry:

Default case when neither of above conditions are met

* Follows Confluent wire format specification

* Dynamically skips 1 or 2 bytes based on message index

* Uses remaining bytes for Protobuf deserialization

Checklist

@phipag phipag self-assigned this Jun 20, 2025
Copy link

@phipag phipag merged commit 4a8511d into main Jun 20, 2025
15 checks passed
@phipag phipag deleted the phipag/karthikpsqamy-kafka-fixes branch June 20, 2025 11:29
@github-project-automation github-project-automation bot moved this from Pending review to Coming soon in Powertools for AWS Lambda (Java) Jun 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Coming soon
Development

Successfully merging this pull request may close these issues.

2 participants