Add gzip compression on reaper report #242
Conversation
| url(url) | ||
| post(reportString.toRequestBody("application/json; charset=utf-8".toMediaTypeOrNull())) | ||
| post(compressedReport.toRequestBody("application/json; charset=utf-8".toMediaTypeOrNull())) |
There was a problem hiding this comment.
I think okhttp has a method for this: https://square.github.io/okhttp/5.x/okhttp/okhttp3/-request-body/-companion/gzip.html
Something like:
val body = reportString.toRequestBody("application/json; charset=utf-8".toMediaTypeOrNull());
val request = Request.Builder().apply {
header("Authorization", "Bearer $apiKey")
header("Content-Encoding", "gzip")
url(url)
post(body.gzip())
};
I think
There was a problem hiding this comment.
Appears to only be in the 5.0 snapshot, which we're on 4.11. I'll see if 4.x has one, as I'd prefer to not leverage a snapshot release.
There was a problem hiding this comment.
Follow up, did some reading and despite 5.x-snapshot state, it sounds like it actually is quite stable/no breaking changes.
We need to land backend support for this, but I can follow up with updating OkHTTP to 5.x and leverage the helper function.
| import java.io.DataInputStream | ||
| import java.io.EOFException | ||
| import java.io.File | ||
| import java.io.FileInputStream | ||
| import java.io.FileOutputStream | ||
| import java.io.IOException | ||
| import java.util.zip.GZIPOutputStream | ||
| import kotlin.coroutines.resumeWithException | ||
|
|
||
| internal class ReaperReportUploadWorker( |
There was a problem hiding this comment.
I thought sorting the records might help with compression, and it does, but not as much as I had hoped:
$ # Without sorting:
$ python3 -c 'import json; import random; import base64; print(json.dumps([base64.b64encode(random.randbytes(8)).decode("utf8") for i in range(100000)]))' | gzip --verbose >/dev/null
41.7%
$ # With sorting:
$ python3 -c 'import json; import random; import base64; print(json.dumps(sorted([base64.b64encode(random.randbytes(8)).decode("utf8") for i in range(100000)])))' | gzip --verbose >/dev/null
46.3%
(Number is compression ratio)
So maybe not worth it.
No description provided.