Bug Report

Describe the bug

offset_key in the tail input plugin reports offsets starting from 0 for newly appended lines after Fluent Bit restart, even though the SQLite database stores the correct absolute offset. The file content is read from the correct position, but the offset_key value attached to records is wrong.

This causes data corruption in downstream systems that use offset_key for positional writes.

To Reproduce

1. Configure tail input with offset_key and db enabled
2. Let Fluent Bit read a file partially (e.g. DB stores offset 9,840,308)
3. Restart Fluent Bit
4. Append new lines to the file
5. Observe offset_key values in output records: 0, 143, 286... instead of 9840308, 9840451, ...

Expected behavior

offset_key should report absolute byte offset from the beginning of the file, consistent with the offset stored in the database.

Your Environment

• Fluent Bit version: v5.0.0, v5.0.1, v4.2.0 (confirmed on all three)
• Operating System: Ubuntu 22.04 (Docker)
• Configuration file:

pipeline:
  inputs:
  - name: tail
    tag: test
    path: /mnt/input/*.log
    path_key: file_path
    offset_key: file_offset
    read_from_head: true
    refresh_interval: 5
    inotify_watcher: false
    skip_empty_lines: false
    skip_long_lines: false
    mem_buf_limit: 30Mb
    buffer_chunk_size: 32Kb
    buffer_max_size: 2Mb
    threaded: true
    static_batch_size: 5Mb
    event_batch_size: 5Mb
    db: /fluent-bit/data/test.db

  outputs:
  - name: tcp
    match: test
    host: receiver
    port: 5170
    format: json_lines
    workers: 1
    retry_limit: false
    net.keepalive: true

Evidence (captured via tcpdump on TCP output)

DB stores the correct absolute offset:

sqlite> SELECT * FROM in_tail_files;
3|/mnt/input/test.log|9840308|33|1774994106|0

Before fix (bug) -- offsets start from 0 instead of 9,840,308 after restart:

{"file_path":"/mnt/input/test.log","file_offset":0,"log":"..."}
{"file_path":"/mnt/input/test.log","file_offset":143,"log":"..."}
{"file_path":"/mnt/input/test.log","file_offset":286,"log":"..."}

After fix -- offsets match the DB position:

{"file_path":"/mnt/input/test.log","file_offset":9840308,"log":"..."}
{"file_path":"/mnt/input/test.log","file_offset":9840451,"log":"..."}
{"file_path":"/mnt/input/test.log","file_offset":9840594,"log":"..."}