This project will rigorously test static spatial microsimulation models in a range of conditions. We will begin with IPF-based approaches, as these are simple, deterministic and readily available. The project is open access: all the code and data sets used to conduct the testing can be downloaded and modified. We encourage others to fork the entire project using the command git clone for testing on home computers.
The research problem tackled is lack of rigorous and repeatable model evaluation and testing. Data and code used in published microsimulation research is not always publicly available. This can make it difficult for readers to reproduce results, test models in new ways and adapt the code for their own purposes.
The project aims to test each IPF model in a range of realistic conditions, changing only one parameter at a time. This will allow factors affecting IPF performance to be isolated and analysed.
Our objectives in creatng a framework for model evaluation are exactly the same as Holm and Malika's (2013) 'wish list', a set of principles that should be considered before developing new models:
- Using the most modern software
- Using standard methods, shared by many users
- Backward compatibility (so keeping our old models and subsystems running)
- Avoiding relearning
- Developing solutions that are theoretically well designed
- Transferring knowledge and know-how to new colleagues
This data-driven approach requires a variety of input data sets. These will be grouped into scenarios that are designed to be realistic: simple to complex, small to large. The next stage is model preparation, which involves tailoring each to produce small area microdata in each of the scenarios. The final stage is testing: systematically altering specific aspects of each model to observe the factors affecting model performance.
This is the current list of priorities, and will be updated as the project progresses:
- Discuss the modus operandi and structure of project
- Upload publicly available test data scenarios
- Add models, and modify them to generate spatial microdata for each scenario
- Run the models under a variety of scenarios, changing one parameter at a time. These parameters will include:
- The initial weights
- Number of iterations
- Order of constraints
- Integerisation
- The use of cross-tabulated vs uni-variate categories (where available)
- Evaluate the results using quantitative methods