Fetch consumer group metadate after a rebalance

[erikvanoosten]

As discussed in #1464 (comment) we can simplify the code that fetches consumer group metadata by fetching it immediately after a rebalance. However, we first need to explore why errors from the call to consumer.groupMetadata() are suppressed, and whether it makes sense to keep retrying after a failure.

This issue is done when:

• we know if we can ignore errors from consumer.groupMetadata()
• we know retrying consumer.groupMetadata() is useless
• if the answer is yes to both, fetching the group metadata was moved to directly after the rebalance

[vermas7988]

the given method consumer.groupMetadata() may return InvalidGroupIdException (see line 1717: KafkaConsumer.java).
It can happen if consumer group id is misconfigured from user's side, and it looks like there is very less we can do from library's perspective. WDYT?

[erikvanoosten]

the given method consumer.groupMetadata() may return InvalidGroupIdException

It depends on what they mean by 'if consumer does not have a group'. We already do not call the method when no group id is configured. Is that sufficient?

[vermas7988]

I read a bit more. So, also, when users don't want to store offsets on kafka, they might not configure consumer groups.

If the Runloop is not crashing because consumer group metadata is None while we received the current event, and we still process the current record/event successfully, it feels safe to make the change.
But hard to take decision, if we are retrying indefinitely on current event when the metadata is not present.

[erikvanoosten]

We're going to make release candidates for zio-kafka 3. We could make the change for the release candidate. Then when no-one complains we can keep it :)

[vermas7988]

okay I will submit a PR