Retrospective forecast

[Inglezos]

Let's say we want an analysis where last_date for a country scenario is set to be 31Dec2020 (end analysis for 2020) using _update_range(), but we want to make forecast using as base/last date the 31Dec2020 and forecasting values up to [31Dec2020 + delay] date. Then we want to plot the actual (up-to-now) against the past forecasted variables (C, CI, F, R, Rt). It is like a kind of retrospective forecast analysis. The final plot would be actual & forecasted target variables for [01Jan2020, 31Dec2020 + delay] date AND only actual variables for [31Dec2020 + delay + 1, today] ranges. Is something like that possible at all? What needs to be done?

[lisphilar]

Did you try Scenario.disable() method? Please share the current script you tried.

[Inglezos]

It is about forecast accuracy. What I mean is how well the forecasted cases compare to the actual cases. With current implementation, the forecast is about the future where we don't know yet the actual cases. If the forecast had a retrospective option/ability additionally, that would mean we could make forecast for past dates and compare these forecasted cases with the actual, since now we know about these past cases. The key factor about this would be that the forecast would consider only a subset of phases up to the desired past date. This could work if we disabled manually the phases that exceed the past date we want but this is not correct, because we also want to plot all the phases up to today as well, in order to compare both actual and forecasted cases in a common plot.

[lisphilar]

I'm not a coder in your company.
You MUST share the scripts you tried as much as possible.
Without codes, problems will not progress towards resolution at all.

https://gist.github.com/lisphilar/6a14ef325f1fe93e8a79c442e2523d1f

• Scenario.history() works because this shows actual values and simulated values in one figure.
• Scenario.delete() (not .disable()) deletes selected phases as used in Scenario.retrospective().
• For user-interface, please update .retrospective() to accept the arguments of .fit(), including oxcgrt_data, with a new issue and a pull request.
• Score (determination coefficient) of fitting can be retrieved with .fit()["score_train"] and .fit()["score_test"].
• With Japan data and start date "01Nov2020" for retrospective analysis, theta values were predicted as negative values. This will be a next issue for prediction.

[Inglezos]

I'm not a coder in your company.

What do you mean by that? Sorry I didn't mean to cause any troubles. I hope there is no misunderstanding.

The reason I am not providing any codes is simply because I do not know what code to write to try. My question is theoretical at its basis, since you know better the supported methods and functionality regarding the phases and I thought to ask you if something like that is currently possible at all or needs implementation. My initial goal was to make you understand the scenario I am describing, not ask you to provide me with a solution/code right away.

Commenting on the above notebook example, I am not sure if the topic I describe is indeed in there.

To put it in a specific example, I would like the forecast to consider as last day a past day, let's say 31-Dec-20. So we forget all the days after that (01-Jan-21 until today) and equal today = 31-Dec-20. Then we run the scenario analysis as you do (using as COVID records from 01-Jan-20 up to 31-Dec-20) and make forecast prediction about the parameters and thus for the cases (with the OxCGRT analysis). Assuming delay period = 21 days, the forecasted cases begin on 01-Jan-21 and end on 22-Jan-21. The interesting part now is that we actually do know the actual cases for all this period. So we could estimate the forecast accuracy by comparing these forecasted cases with the actual ones.

The final plot output from this would be common for Main and Forecast line up to 31-Dec-20, then probably forecast would deviate a little bit between 01Jan21 - 21Jan21 and then for 22Jan21 - today (26Jan21) the forecast line would either disappear or continue simulating cases by keeping the last forecasted parameters.

In the notebook example for 01Nov20 (assuming Japan delay = 14), you delete the next phases and call fit() but then the prediction somehow considers all the phases again. The desired predict output summary should be ...Past 15Oct20 - 31Oct20, Future 01Nov20 - 14Nov20. After that no forecast is valid (since we forecast only for delay days into the future) and the next simulated cases should be based on the last forecasted parameters. Then, the deleted phases shall reappear in order to compare their actual cases with the forecasted ones.

Did I clarify it a little bit or made it worse?

[lisphilar]

My initial goal was to make you understand the scenario I am describing, not ask you to provide me with a solution/code right away.

How should I reply as the first comment on this question? I said "Did you try Scenario.disable() method?"

The desired predict output summary should be ...Past 15Oct20 - 31Oct20, Future 01Nov20 - 14Nov20. After that no forecast is valid (since we forecast only for delay days into the future)

I don't think so because forecast after 14Nov20 is also valid.
In the notebook, we have records of

• estimated parameter values from 06Feb2020 to 31Oct2020 and
• actual records of OxCGRT indicators from 06Feb2020 to 23Jan2021.

With .fit() and delay=14, linear regression model learns the relasionship with [parameters 20Feb2020 - 31Oct2020] and [OxCGRT 06Feb2020 - 17Oct2020].
Then, with [OxCGRT 18Oct2020 - 23Jan2021], the model predict [parameters 01Nov2020 - 07Feb2021].
Because we know OxCGRT to 23Jan2021, we can predict paramers to 07Feb2021 beyond "31Oct2020 + delay".

If this is not desired, users need to set property snl.last_date = 31Oct2020 (self._last_date internally). This makes the actual record range of OxCGRT scores as 06Feb2020 - 31Oct2020. The predicted parameters will be to "21Oct2020 + delay".

However, at the current version, this causes another problem. The actual record range of the number of cases will be 06Feb2020 - 31Oct2020 as well. The last date of "Actual" of Scenario.history() will be 31Oct2020.

To overcome this issue, we need to create .today property (self._today), replace some self._last_date with self._today and edit line 1099-1100 of ._fit_create_data() method.
Users set snl.today = "31Oct2020" (to set the last date of OxCGRT data and set the reference date to difine Past/Future). I need to investigate the codes of Scenario.track() method, but I think users can skip snl.last_date = 14Nov2020" (= 31Oct2020 + delay).