consuming: reset to nearest if we receive OOOR while fetching #628
Conversation
If we receive OOOR while fetching after a fetch was previously successful, something odd happened in the broker. Either what we were consuming was truncated underfoot, which is normal and expected (for very slow consumers), or data loss occurred without a leadership transfer. We will reset to the nearest offset after our prior consumed offset (by time!) because, well, that's what's most valid: we previously had a valid offset, and now it is invalid. Closes #621.
|
@twmb may I ask why we prefer resetting by time? Say the consumer did consume offset=10, now it does a fetch offset and gets offset_out_of_range error with log start offset = 15, why it doesn't just move to offset 15 and instead has to do the timequery dance? |
|
@nvartolomei this dance is to address data loss at the end, not the start:
The best to reset to is the new 13, which is, by time, the first thing after 12. Any other offset-reset scheme can only pick the start or end; picking the start is a whole-topic-reconsume we want to avoid, picking the end misses the new 13, 14, 15. |
|
But for brokers conforming to KIP-320 that shouldn’t be a problem. From
what I recall, broker is allowed to return only: a) not a leader, b) leader
epoch fence.
If so then, this timequery dance shouldn’t be applied for a broker with
KIP-320 support as OOOR can only indicate a prefix truncation from which we
can advance to the earliest available offset.
…On Tue, 30 Apr 2024 at 06:55, Travis Bischel ***@***.***> wrote:
@nvartolomei <https://github.com/nvartolomei> this dance is to address
data loss at the end, not the start:
- Broker has 11, 12, 13, 14, 15
- Client consumes 11, 12
- Broker loses 13, 14, 15
- Client asks for 13
- Broker returns OOOR
- Second client produces 13, 14, 15
The best to reset to is the new 13, which is, by time, the first thing
after 12. Any other offset-reset scheme can only pick the start or end;
picking the start is a whole-topic-reconsume we want to avoid, picking the
end misses the new 13, 14, 15.
—
Reply to this email directly, view it on GitHub
<#628 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAEETWP74FOGEJ4VL3VURWLY74W6RAVCNFSM6AAAAABAFDV3M6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBUGQZTMMBRGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Correct, if there are no bugs, this code should not be necessary. This PR was added as a bit of a paranoia guard if a broker was buggy and lost data but did NOT bump the epoch. There was worry that this happened in RP once, which caused a client to reset to the beginning. Now, at least, the client will not reset to the beginning even if the broker has a bug. |
If we receive OOOR while fetching after a fetch was previously successful, something odd happened in the broker. Either what we were consuming was truncated underfoot, which is normal and expected (for very slow consumers), or data loss occurred without a leadership transfer. We will reset to the nearest offset after our prior consumed offset (by time!) because, well, that's what's most valid: we previously had a valid offset, and now it is invalid.
Closes #621.