how read metadata only?

[il-giza]

Hi!

I am trying to read only the metadata. But I see that when reading with the parameter metadataonly = True, all the data is still read. This can be seen in the reading time and traffic. My data is 10GB and I can see that 10GB is loading.
The same thing happens when using row_limit=1, all 10GB is loading. Although the result is correct, i see only 1 row.

How read metadata only?

Thanks!

[ofajardo]

The C library behind pyreadstat (Readstat) reads all the bytes from the file and puts them in RAM. This is unavoidable. Once it is in RAM, I use the API to parse those bytes and first get the metadata, and then parse and extract the data. When you use metadataonly I stop after parsing the metadata, avoid parsing the numerical data, this saves time and some extra memory while creating the python objects for the data, but this is as good as we can get. So, there is no way to read only a fraction of the bytes and from there parse the metadata. Does that answer the question?

[il-giza]

Thanks for the quick response!!!

That is, I have to read all 10 GB to get the metadata?
Did I understand the answer correctly?

[ofajardo]

yes, that is correct.

IN the other hand you will see that if you read all the data the RAM consumption will be (transiently) higher than 10GB while you have in parallel all the bytes and the python objects are being created, so still reading only the metadata saves memory.

[il-giza]

ОК, Many thanks!