Unusual behaviour defining variables within function call

[elsie-h]

I've noticed some strange behaviour using the minimum_of function. It's easy enough to work around, but I can't work out exactly why it's happening. It seems to be related to defining variables within function calls for multiple variables, and might not be specific to the combination of satisfying and minimum_of given in the example below, but I haven't checked. I've included some examples below to illustrate when the error does and doesn't appear.

Here's what I want to do:

study = StudyDefinition(
    default_expectations={
        "date": {"earliest": "1900-01-01", "latest": "today"},
        "rate": "uniform",
        "incidence": 0.5,
    },

    index_date="2020-01-01",

    population=patients.satisfying(
        """
        registered
        AND
        NOT died
        """,
    registered=patients.registered_as_of(
      "index_date",
    ),
    died=patients.died_from_any_cause(
      on_or_before="index_date",
      returning="binary_flag",
    ),
    ),

 covid_vax_1_date=patients.minimum_of(
    covid_vax_disease_1_date=patients.with_tpp_vaccination_record(
                target_disease_matches="SARS-2 CORONAVIRUS",
                on_or_after="index_date",
                find_first_match_in_period=True,
                returning="date",
                date_format="YYYY-MM-DD",
            ),
     covid_vax_pfizer_1_date=patients.with_tpp_vaccination_record(
            product_name_matches="COVID-19 mRNA Vaccine Pfizer-BioNTech BNT162b2 30micrograms/0.3ml dose conc for susp for inj MDV",
            on_or_after="index_date",
            find_first_match_in_period=True,
            returning="date",
            date_format="YYYY-MM-DD",
        ),
    ),
)

But it returns the following error:

File "/app/cohortextractor/expressions.py", line 155, in insert_implicit_comparisons
empty_value = empty_value_map[column_name]
KeyError: 'registered'

Note that if I conditioned on died first, then the error would read KeyError: 'died', i.e. it's nothing to do with the name 'registered'.

However, if I select the population like this instead (and then create a died variable), it works:

population=patients.registered_as_of(
      "index_date",
 ),

Or, if I keep the original method for selecting the population, but define the variables that I'm minimising outside the minimum_of function like this, it also works:

  covid_vax_disease_1_date=patients.with_tpp_vaccination_record(
      target_disease_matches="SARS-2 CORONAVIRUS",
      on_or_after="index_date",
      find_first_match_in_period=True,
      returning="date",
      date_format="YYYY-MM-DD",
      ),
            
  covid_vax_pfizer_1_date=patients.with_tpp_vaccination_record(
      product_name_matches="COVID-19 mRNA Vaccine Pfizer-BioNTech BNT162b2 30micrograms/0.3ml dose conc for susp for inj MDV",
      on_or_after="index_date",
      find_first_match_in_period=True,
      returning="date",
      date_format="YYYY-MM-DD",
      ),

 covid_vax_1_date=patients.minimum_of(
   "covid_vax_disease_1_date", "covid_vax_pfizer_1_date",
    ),

So it's straightforward to work around with either of the above solutions, but I thought I'd ask in case there was an obvious explanation that I'm missing!

[wjchulme]

FWIW I recall encountering a similarish problem a while ago -- something like:

var1 = patients.satisfying(
   "a OR b",
   a = patients.some_function(),
   b = patients.some_other_function()
),


var2 = patients.satisfying(
   "a AND c",
   a = patients.some_function(),
   c = patients.yet_another_function()
)

and it didn't like that a was defined twice.

It's not clear whether this is happening in your case as I can't see the full study definition -- can you link to it? (here or offline). Either way, it's some unexpected behaviour related to temporary variables. But not sure whether it's a bug in your script, or a bug in cohort extractor itself.

[elsie-h]

Thanks, @wjchulme. It's in a private repo on my personal GitHub at the moment, as the project is not yet approved, but I've sent you an invite to collaborate. Let me know if any issues accessing it.

The script I've been using for testing is study_definition_test.py, which only includes the variables in the chunk posted in my question, so the error shouldn't be related to anything else in my study definition.

[evansd]

Hi @elsie-h, thanks for the very thorough issue description. It's definitely a bug with cohortextractor and not an issue with your code. I was in the process of writing up a ticket for this and explaining that we probably wouldn't be able to look at this for a while when I realised there was actually a quick fix for it!

I've created a PR but it will need to get reviewed and merged and then you'll need to run opensafely pull cohortextractor before you can use the new version of the code:
opensafely-core/cohort-extractor#651

It should be live by the end of the day though.

[evansd]

Update: fix is live now

[elsie-h]

Thanks @evansd - that's working fine now!!