Kafka source (v0.51.0) fails SCRAM-SHA-512 SASL authentication when connecting to AWS MSK (Kafka 4.0)

[devsecopsaio]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Vector 0.51.0 fails SASL authentication with SCRAM-SHA-512 when connecting to an Amazon MSK Kafka cluster running Apache Kafka 4.0. The failure occurs during the SCRAM handshake in the AUTH_REQ state after a successful socket + TLS handshake (~318ms).

The credentials used (username/password) have been independently validated outside Vector using Kafka CLI tooling (kcat, kafka-metadata-shell, kafka-metadata-shell) and confirmed to:

Authenticate successfully against the same MSK brokers

Return cluster metadata without errors

Confirm the SASL user is active and password is correct

Confirm no ACL, SCRAM user, or broker-side credential rejection issues

This indicates the authentication failure is specific to Vector’s Kafka source client (librdkafka), not the MSK cluster or IAM/SCRAM credentials.

Configuration

Steps to reproduce the behavior:

Configure a Vector Kafka source with SCRAM-SHA-512 + TLS enabled.

Point bootstrap_servers to an AWS MSK TLS/SASL endpoint (port 9096 or configured secure port).

Start Vector and observe authentication failure.

Configuration snippet:

[sources.msk_source]
type = "kafka"
bootstrap_servers = "<MSK_BROKER>:9096"
sasl.enabled = true
sasl.mechanism = "SCRAM-SHA-512"
sasl.username = "<USERNAME>"
sasl.password = "<PASSWORD>"
tls.enabled = true


Error log captured:

vector | 2025-11-24T12:35:03.000615Z ERROR source{component_kind="source" component_id=msk_source component_type=kafka}: kafka_source:
librdkafka: FAIL [thrd:sasl_ssl://<MSK_BROKER>:9096/bootstrap]: SASL authentication error:
Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (after 318ms in state AUTH_REQ)

Expected behavior

Vector should authenticate successfully using the provided SCRAM-SHA-512 credentials over TLS and start consuming Kafka topic data from the MSK cluster, consistent with behavior exhibited by other Kafka clients.

Version

0.51.0

Debug Output

vector | 2025-11-24T12:35:03.000615Z ERROR source{component_kind="source" component_id=msk_source component_type=kafka}: kafka_source:
librdkafka: FAIL [thrd:sasl_ssl://<MSK_BROKER>:9096/bootstrap]: SASL authentication error:
Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (after 318ms in state AUTH_REQ)

Example Data

No response

Additional Context

we implemented the config changes recommended in issue 23385 :#23385 but its did work for us.

References

No response