reporting: Serialize batches as they're constructed

[sharnoff]

Part of #1220.

For scaling events, the size of a gzip-compressed batch is much less than the uncompressed version, so it's actually worth it to compress the JSON-encoded contents of the batch as we go, instead of buffering the events as a []ScalingEvent in memory.

Implementation

This change primarily adds the reporting.BatchBuilder[E] type (and some implementations), which abstracts over the process of gradually building the serialized bytes from a batch of events.

Two implementations of BatchBuilder are provided in pkg/reporting - one for large JSON arrays, and one for the JSON lines format.

To allow continuous gzip compression, both of these JSON batch builders are constructed with a new IOBuffer type, which is just an io.Writer with a method to collect the bytes at the end. There's implementations of IOBuffer for normal bytes and for a gzip-compressed buffer.

All that together means that batches of billing and scaling events should now be GZIP-compressed (where possible) as events are added to the batches, dramatically reducing the memory footprint.

---

Note: This PR builds on #1226, and must not be merged before it.

[github-actions]

No changes to the coverage.

HTML Report

Click to open

[sharnoff]

Needs to be updated after rebasing #1226. Final commit is still ok-ish to review in the meantime, but ok to leave until after I'm done.

[petuhovskiy]

(non-blocking) I don't really like panics when handling errors from io.Writer, but should be fine as long as we know that byte buffer and gzip implementation should not return errors.

[sharnoff]

I don't really like panics when handling errors from io.Writer (non-blocking), but should be fine as long as we know that byte buffer and gzip implementation should not return errors.

That's fair — my thinking is that there isn't really anything we can do if we get an error there: it's a buffer in memory, so retrying won't help - should we just drop the events? IMO it's kind of better to panic so that we bubble the error up into something more noisy, hoping that restarting the autoscaler-agent will resolve the issue.

Probably worthwhile to at least clarify that in the code — WDYT?

[petuhovskiy]

there isn't really anything we can do if we get an error there

There are always two options:

1. propagate the error up
2. handle the error

In terms of handling, it's true that we can't do much here. The only thing we can do is to drop the event and collect context about the error (what was the event, what was the write that failed, etc).

In this particular case, I'd probably changed the definition of BatchBuilder to this:

type BatchBuilder[E any] interface {
	// Add appends an event to the in-progress batch. If it fails, there are no guarantees
	// about the state of the batch and it should be dropped.
	Add(event E) error
	// Finish completes the in-progress batch, returning the events serialized as bytes.
	Finish() []byte
}

Then, I'd propagated the error up, probably up to EventSink. The error handling there would be to drop the batch, log the error, maybe increase the metric for the errors. I don't like panic in JSONLinesBuilder and JSONArrayBuilder because it might actually get triggered if someone adds new BufferIO implementation in future PRs. And if it gets triggered, it might break things unexpectedly, if there is no explicit recover().

But for now we know that there is only ByteBuffer and GZIPBuffer so I'm ok with merging as is.