Inquiry Regarding Compatibility Issue with Reading SPSS Version 25 Files

[TCcodemaster]

Dear all,
I'm reaching out regarding an issue I encountered while using pyreadstat with SPSS files of different versions.
I've noticed that pyreadstat seems to handle SPSS files from version 25 correctly, but when attempting to read files from version 22, I encounter the following error:
Traceback (most recent call last): pyreadstat._readstat_parser.ReadstatError: Invalid file, or file has unsupported features
I find it perplexing that SPSS files of version 22 are not supported, while those of version 25 are. Could someone please shed some light on this issue? Is there a specific reason for this limitation, and is there any plan to extend support to older versions of SPSS files in future releases?

Thank you in advance for any insights or assistance provided.

[ofajardo]

Hi, thanks for the report. I think it should not be the case that older versions are not read. So the explanation can be:

• The files are corrupt. Can you open them in SPSS?
• The files have some feature Pyreadstat cannot read. If you generate a simple file in SPSS 22, is that read correctly?
• The file was generated by an application different from SPSS.
• Indeed there is a regression bug in [ReadStat]((https://github.com/WizardMac/ReadStat) precluding from reading older versions. In this case an issue gas to be opened in ReadStat. Fir that, the other options have to be checked and a sample file has to be provided.

[TCcodemaster]

Thank you for your prompt response. I've conducted further testing based on your suggestions, and it appears that SPSS 22 is indeed able to read files created within its environment, either manually inputted or copy-pasted. However, issues arise when attempting to import data from CSV or Excel files. I've attached two sample files for your reference: one created via copy-pasting and the other imported from a CSV file.

Upon reflection, I'm considering whether the problem might stem from SPSS's own conversion process of CSV files to .sav format, potentially causing compatibility issues with external packages. Could this be a plausible explanation?

Your insights on this matter would be greatly appreciated.
spss_sample.zip

[ofajardo]

Thanks for the example files. I opened them in a hex editor (you can try yourself in hexed.it) and the files look different. The bycopy one says in the header that it was created in SPSS 25, while the csv one says that it was created in SPSS 22. So it is not clear to me if the problem is that pyreadstat cannot read files from SPSS 22 or not as you initially proposed. I also opened them in SPSS, and It also looks like the csv one is not in UTF-8 but in some chinese local encoding. In SPSS 26, a file generated by coping pasting or generated by csv import both are fine to read. So not sure what is going on here. The file can be opened in PSPP suggesting that it should be possible to read it.

I would suggest that you try with a simpler file without chinese characters first (for example in the folder https://github.com/Roche/pyreadstat/tree/master/test_data/basic, there is sample.csv and sample.sav) trying both to import the csv and copying pasting in SPSS 22 to see if that generates an issue. If not, then try to add some chinese characters to see if that makes a difference. Also try to define the encoding as UTF-8 maybe? That already would point at some workarounds, and then you can also open an issue in Readstat for them to fix it (will take long probably).

[TCcodemaster]

Your insight was instrumental in uncovering the root cause, which appears to be related to special Chinese characters. It's fascinating to note that transitioning to SPSS 25 resolves the issue.
Your suggestion to explore simpler files initially and gradually introduce complexities is spot-on. I'll follow your guidance, experimenting with different encodings, and exploring potential workarounds
thank you sincerely for your support and expertise.