@raprasad Starting on this tonight - prototyping a class that will handle SPSS (.sav) and Stata (.dta) files in a single interface and return a Pandas dataframe (based on https://github.com/Roche/pyreadstat)

@raprasad After researching the differences between using pandas and pyreadstat, I'm finding that pandas supports a much smaller subset of file version numbers. The two files I am using for testing (one .dta and one .sav) both pass tests using pyreadstat, but pandas gives the following error:

Version of given Stata file is 36. pandas supports importing versions 105, 108, 111 (Stata 7SE), 113 (Stata 8/9), 114 (Stata 10/11), 115 (Stata 12), 117 (Stata 13), 118 (Stata 14/15/16),and 119 (Stata 15/16, over 32,767 variables)

Given this restriction, it seems preferable to keep pyreadstat as the SPSS / Stata reading library.

@raprasad Currently getting the following error as the variable info page loads:

Failed to open file due to UnicodeDecodeError. ('utf-8' codec can't decode byte 0x96 in position 6: invalid start byte)

From the logs:

celery-queue_1 | [tasks.py:62 - profile_dataset_info() ] profile_dataset_info: Failed to open file due to UnicodeDecodeError. ('utf-8' codec can't decode byte 0x96 in position 6: invalid start byte) celery-queue_1 | [2022-08-04 16:08:08,215: ERROR/ForkPoolWorker-6] profile_dataset_info: Failed to open file due to UnicodeDecodeError. ('utf-8' codec can't decode byte 0x96 in position 6: invalid start byte)

This needs a redo as per PR notes