Missing_ranges for a raw spss .sav data file

[steinunns]

I'm working with a raw .sav file. I recoded all the missing values as 99. I am able to export the dataframe with the defined missing_ranges. However, I'm unable to read in the new .sav export unless I don't export the missing_ranges with the new .sav file. I'm guessing there is something wrong with how I'm creating the missing_ranges dictionary.

I've tried
missing_ranges = {} for item in column_list: missing_ranges[item] = [{'lo': 99, 'hi': 99}]
output:
{'birth': [{'lo': 99, 'hi': 99}],
'grade': [{'lo': 99, 'hi': 99}]}

I have also tried having the dictionary look like:
{'birth': [99],
'grade': [99]}

But I always get the same error when I try to read in the new .sav export:

I've made sure there are no NaN values in the dataframe .

I've been able to export .sav files before with the missing ranges but that time the .sav file I was working with had already defined missing ranges and I just added a new item to the missing ranges dictionary after I added a new variable to the dataframe.

Can someone help me?

[ofajardo]

very strange, there is something special in your data that causes the error I guess, because I have missing ranges in my tests and it works, see here. It looks very similar to what you are doing ... so I really don't know what is going on ... so some random question to check:

have you tried for example writing missing ranges for only one variable and see if that works?
Does it work if you use take away that meta_dict when writing?. Try to find what is the minimum of arguments to reproduce the issue.
Which version of pyreadstat are you using? the latest? in any case could you try another version to see if the problem is also there?
I guess it should not matter, I wrote 'hi' first and 'lo' second in my dictionary and you the opposite, but that shouldn't matter.

[ofajardo]

In case you are using the latest version of pyreadstat, it seems they have done some changes in the latest version of Readstat that may have something to do with this, it worth trying therefore the previous version of pyreadstat to see if the problem has been introduced in the most recent version. The opposite also holds true, maybe the issue is solved in the latest version, so you should try it if you haven't yet.

[steinunns]

That worked! Just had to update the pyreadstat package and restart my Jupyter notebook and everything works :)

[ofajardo]

wonderful!

[steinunns]

Now I noticed something strange, not sure if it happened in previous modules sine I'm pretty new working with .sav data.

In my dataframe (same as above), the first 3 columns are descriptive (date, language and IDs) and are therefore not part of the metadata. The rest of the columns in the dataframe are of type integer and have variable value labels in the metadata except for 2 columns that hold answers for open ended questions and are of type object. These object columns are causing trouble. In the raw data these columns are the two last ones. When working with the data I created two new columns ('school', 'country'), 'school' of type object, and 'country' that's type int and added its information to the metadata dictionary and missing ranges.

The trouble I'm getting is when I export the dataframe and try importing the new .sav file the new dataframe has 2 more columns than the dataframe that was exported and it has changed the names of my new columns and wiped out their data. It happens even though I export the dataframe without the variable_value_labels and missing ranges. I tried exporting it as a CSV and it worked fine so it was not the dataframe itself causing the problem.

I tried rearranging the dataframe columns so that all non-integer columns appear first in the dataframe and exporting it that way along with the value labels and missing ranges. That works fine! I thought the variable_value_labels and missing_ranges uses the column names to match its properties and not the indices? Is this normal or something strange going on?

Screenshot of the end of the original dataframe

Screenshot of the dataframe I get when I import the new .sav file

The rearranged dataframe

The imported .sav file from the rearranged dataframe

[ofajardo]

very strange indeed, this one looks like a bug. If you isolate the error and do have a file you can share, you can create an issue to track the problem. It would be extremely useful if the data you share has the minimum number of variables and observations to reproduce the issue, i.e. the simpler the data, the better.

Do you have any very long string somewhere in your data, or do you have a variable with a very long name? Those seem to be causing problems some time. IF you have such things, try to remove them and see if that also fixes the issue.

[ofajardo]

oh, and could you please check if the issue is happening only in the last version of pyreadstat or also before?

[steinunns]

There was a very long string in one of string columns. When I removed that record I don't experience this error anymore. The longest string was 2800 characters, the second longest was around 500. Do you know if there is a max of "allowed" number of characters before it starts breaking the data?
I tried to see if the error also happens in previous versions so I tried using pyreadstat version 1.1.8 and got the same error

[ofajardo]

OK, probably it is a variant of a known bug (see for example), other people have reported somethig similar but only when opening the file in SPSS while it is fine in python (here.
Good thing is that it is tracked, bad thing is that it is not known when it is going to be solved.
The limit I think is 756 C characters, but that may translate to lower number in python, since python uses unicode and some characters (non-ascii) are double.