Follow-up to #4821, which adds direct encoding into the write buffer for fitting fixed-size journal records. Investigate whether applying this approach to variable journals produces enough performance benefit to justify the additional implementation and public API surface. No variable-journal speedup has been measured yet.
Evaluate two candidates independently:
- Single-offset appends in contiguous variable journals. When
write_encoded publishes a batch containing one item, append base_offset directly to the fixed offsets journal. This avoids the temporary absolute-offset vector and, when the fixed write buffer has room, the fixed journal's prepared encoding allocation and copy. Preserve data-before-offset publication. Account for the internal metrics change from append_many to append.
- Direct variable-frame encoding. Investigate extending the writer primitive from
FixedSize + Write to EncodeSize + Write, with a frame adapter that caches the checked payload length and total frame size and emits the existing length prefix followed by the payload. Start with uncompressed single-item appends in contiguous and segmented variable journals, with the existing owned-buffer path as the fallback. Preserve the wire format, size validation, and the writer's exact-size and unwind guarantees.
Measure both buffered append CPU cost and complete caller workloads using their normal sync cadence. Include representative record sizes, buffer-boundary fallbacks, allocation counts, and throughput/latency. Compare each candidate with the baseline; keep batched, prepared, and compressed appends as controls so a narrow single-item improvement is not mistaken for a general journal improvement.
Choose workloads from actual callers: QMDB's bulk path batches records, while consensus WALs use segmented variable journals and therefore do not benefit from the contiguous offsets-only change. Check recovery and publication ordering for any prototype using the supported journal lifecycle.
The outcome should be reproducible benchmark results and a recommendation on whether to implement either candidate. Keeping direct encoding limited to fixed journals is a valid outcome if the measured gains do not justify the complexity.
Follow-up to #4821, which adds direct encoding into the write buffer for fitting fixed-size journal records. Investigate whether applying this approach to variable journals produces enough performance benefit to justify the additional implementation and public API surface. No variable-journal speedup has been measured yet.
Evaluate two candidates independently:
write_encodedpublishes a batch containing one item, appendbase_offsetdirectly to the fixed offsets journal. This avoids the temporary absolute-offset vector and, when the fixed write buffer has room, the fixed journal's prepared encoding allocation and copy. Preserve data-before-offset publication. Account for the internal metrics change fromappend_manytoappend.FixedSize + WritetoEncodeSize + Write, with a frame adapter that caches the checked payload length and total frame size and emits the existing length prefix followed by the payload. Start with uncompressed single-item appends in contiguous and segmented variable journals, with the existing owned-buffer path as the fallback. Preserve the wire format, size validation, and the writer's exact-size and unwind guarantees.Measure both buffered append CPU cost and complete caller workloads using their normal sync cadence. Include representative record sizes, buffer-boundary fallbacks, allocation counts, and throughput/latency. Compare each candidate with the baseline; keep batched, prepared, and compressed appends as controls so a narrow single-item improvement is not mistaken for a general journal improvement.
Choose workloads from actual callers: QMDB's bulk path batches records, while consensus WALs use segmented variable journals and therefore do not benefit from the contiguous offsets-only change. Check recovery and publication ordering for any prototype using the supported journal lifecycle.
The outcome should be reproducible benchmark results and a recommendation on whether to implement either candidate. Keeping direct encoding limited to fixed journals is a valid outcome if the measured gains do not justify the complexity.