KafkaJSGroupCoordinatorNotFound: Failed to find group coordinator

[arupsarkar-sfdc]

Describe the bug
I am getting an error when trying to execute the consumer. I am able to publish the messages without any issues. The error I am encountering is "Not authorized to access group: Group authorization failed"

To Reproduce
https://sfdc-ctx.herokuapp.com/
Go to kafka tab, click Start Producer and then click on Start Consumer. Start Producer is publishing messages, Start Consumer should be able to consume the message and console log it. At that point it is erroring out.

1. Repository with an example project reproducing the issue
2. Look at the startConsumer function line 76.

Detail error message:
The reason you have 8 error messages below because I have 8 brokers.

{"level":"ERROR","timestamp":"2023-12-21T05:03:47.974Z","logger":"kafkajs","message":"[Connection] Response GroupCoordinator(key: 10, version: 2)","broker":"ec2-54-90-120-75.compute-1.amazonaws.com:9096","clientId":"my-app","error":"Not authorized to access group: Group authorization failed","correlationId":0,"size":49}
2023-12-21T05:03:47.986876+00:00 app[web.1]: {"level":"ERROR","timestamp":"2023-12-21T05:03:47.986Z","logger":"kafkajs","message":"[Connection] Response GroupCoordinator(key: 10, version: 2)","broker":"ec2-3-82-39-76.compute-1.amazonaws.com:9096","clientId":"my-app","error":"Not authorized to access group: Group authorization failed","correlationId":0,"size":49}
2023-12-21T05:03:47.999074+00:00 app[web.1]: {"level":"ERROR","timestamp":"2023-12-21T05:03:47.998Z","logger":"kafkajs","message":"[Connection] Response GroupCoordinator(key: 10, version: 2)","broker":"ec2-3-82-39-220.compute-1.amazonaws.com:9096","clientId":"my-app","error":"Not authorized to access group: Group authorization failed","correlationId":0,"size":49}
2023-12-21T05:03:48.010682+00:00 app[web.1]: {"level":"ERROR","timestamp":"2023-12-21T05:03:48.010Z","logger":"kafkajs","message":"[Connection] Response GroupCoordinator(key: 10, version: 2)","broker":"ec2-3-82-39-93.compute-1.amazonaws.com:9096","clientId":"my-app","error":"Not authorized to access group: Group authorization failed","correlationId":0,"size":49}
2023-12-21T05:03:48.021977+00:00 app[web.1]: {"level":"ERROR","timestamp":"2023-12-21T05:03:48.021Z","logger":"kafkajs","message":"[Connection] Response GroupCoordinator(key: 10, version: 2)","broker":"ec2-3-82-39-32.compute-1.amazonaws.com:9096","clientId":"my-app","error":"Not authorized to access group: Group authorization failed","correlationId":0,"size":49}
2023-12-21T05:03:48.033654+00:00 app[web.1]: {"level":"ERROR","timestamp":"2023-12-21T05:03:48.033Z","logger":"kafkajs","message":"[Connection] Response GroupCoordinator(key: 10, version: 2)","broker":"ec2-3-82-39-245.compute-1.amazonaws.com:9096","clientId":"my-app","error":"Not authorized to access group: Group authorization failed","correlationId":0,"size":49}
2023-12-21T05:03:48.034835+00:00 app[web.1]: {"level":"ERROR","timestamp":"2023-12-21T05:03:48.034Z","logger":"kafkajs","message":"[Connection] Response GroupCoordinator(key: 10, version: 2)","broker":"ec2-3-82-39-56.compute-1.amazonaws.com:9096","clientId":"my-app","error":"Not authorized to access group: Group authorization failed","correlationId":2,"size":49}

{"level":"ERROR","timestamp":"2023-12-21T05:03:48.045Z","logger":"kafkajs","message":"[Consumer] Crash: KafkaJSGroupCoordinatorNotFound: Failed to find group coordinator","groupId":"gr2-1703135027851","stack":"KafkaJSGroupCoordinatorNotFound: Failed to find group coordinator\n at Cluster.findGroupCoordinatorMetadata (/app/node_modules/kafkajs/src/cluster/index.js:420:11)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /app/node_modules/kafkajs/src/cluster/index.js:346:33\n at async [private:ConsumerGroup:join] (/app/node_modules/kafkajs/src/consumer/consumerGroup.js:167:24)\n at async /app/node_modules/kafkajs/src/consumer/consumerGroup.js:335:9\n at async Runner.start (/app/node_modules/kafkajs/src/consumer/runner.js:84:7)\n at async start (/app/node_modules/kafkajs/src/consumer/index.js:243:7)\n at async Object.run (/app/node_modules/kafkajs/src/consumer/index.js:304:5)\n at async Object.startConsumer (/app/server/kafka-server.js:76:11)\n at async /app/server/app.js:477:5"}

Expected behavior
The expected behavior should be to view the messages in server console of what is being published.

Observed behavior
A clear and concise description of what did happen. Please include any relevant logs with the log level set to debug.

Environment:

• OS: [e.g. Linux {Heroku}]
• KafkaJS version [e.g. 2.2.4]
• Kafka version [on Heroku]
• NodeJS version [18.x]

Additional context
Add any other context about the problem here.

[ekozilforce]

@arupsarkar-sfdc did you find a way around this? I'm running into similar errors running kafka on heroku and will probably have to swap to a different kafka node lib if there's still no path to resolution here.

[arupsarkar-sfdc]

Elliot is working fine, what I did not have was a consumer group in my kafka in heroku, once I created it all working fine. I can show you how if you are interested. ARUP SARKAR Principal Technical Architect | Salesforce Mobile: 646-269-7876 -- <https://smart.salesforce.com/sig/arup.sarkar//us_mb/default/link.html>

…

On Tue, Jan 16, 2024 at 10:33 AM Elliot Kozil ***@***.***> wrote: @arupsarkar-sfdc <https://github.com/arupsarkar-sfdc> did you find a way around this? I'm running into similar errors running kafka on heroku and will probably have to swap to a different kafka node lib if there's still no path to resolution here. — Reply to this email directly, view it on GitHub <#1654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APZ7LGQQYLNVWNBPBI7BTTLYO2MULAVCNFSM6AAAAABA52YL7KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJTHE4DMNZZGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>