Memory issues Decompressing a large compressed array

[brianreinhold]

Memory allocation is a big problem on the PICO because of fragmentation. There is plenty of memory, but all in small scattered chunks.

I need to make an HL7 FHIR Bundle message which is a huge JSON string. So to do that I preallocate a buffer as a memoryview and create the bytearray piecemeal and load that into the buffer.

However, some messages can still get too large so I use a preallocated BytesIO(fixed_size) and use deflate.DeflateIO to write a compressed version of this message. Now what I would like to do is check it.

I tried an approach like this:

def handle_fhir(bundle: BytesIO):
    total_size: int = bundle.tell()
    logger.info(f'Got to the application with compressed JSON bytearray of size {total_size}')
    bundle.seek(0)
    chunk_size: int = 128  # Adjust the chunk size as needed
    #total_size: int = 0
    import deflate
    with deflate.DeflateIO(bundle, deflate.GZIP) as decompressor:
        
        while True:
            chunk: bytes = decompressor.read(chunk_size)
            logger.debug(chunk.decode('utf-8'))
            if not chunk:
                break

but at runtime I get that the line chunk: bytes = decompressor.read(chunk_size) requires an allocation of 32768 bytes and of course it fails. Why so large? How can I decompress piecemeal to check the output?

[Carglglz]

requires an allocation of 32768 bytes and of course it fails. Why so large?

it may be related to the window size, see https://docs.micropython.org/en/latest/library/deflate.html#deflate-window-size

[brianreinhold]

Thanks. That should do it. I forgot about that and did not set a value in the compression algorithm. I have set a buffer size BytesIO of 0x4000 into which the compressed data should be stored. Given that I did not set the window size during compression, it is using the default compression size of 256.