Download web data dynamically

[ivan-aksamentov]

Currently, the data is made available to the application through the synchronous ES6 import, processed by webpack, which bundles JSON files into resulting JS files along with the application code.

An import was an easy and quick solution in the beginning

• it's synchronous, meaning the data is always available to the code immediately, in a form of object
• it's infallible, meaning import cannot fail

but it's no longer great nowadays, because data size have grown substantially, blowing up the app bundle, making the app very slow and requiring more resources for CI builds.

An alternative would be to host data separately from the app bundle and to fetch it dynamically from the network.

Fetching data from a network location have some additional disadvantages:

• it's asynchronous, meaning it arrives after a delay, and the code using the data should also be converted to asynchronous code, recursively
• it's fallible, meaning the fetch can fail (e.g. no internet connection), so the code using the data should now handle potential errors
• fetched data is untyped, i.e. arrived data may not be in the format that's expected (due to a mistake somewhere). The app and the data will be disconnected from each other. So an additional validation will be needed upon arrival.
• data needs to be handled specifically during CI build, and hosted somewhere

Steps:

• in the app, remove JSON file imports
• on CI, deploy files into public directory during build, along with other static data
• in the app, fetch data with a network request, asynchronous and fallible
• adjust app state handling, data-dependent code to be asynchronous and fallible, recursively

[ivan-aksamentov]

This would also allow to switch data URLs dynamically, to be able to fetch data from a non-default location.

This will be handy to organize experiments and to preview freshly generated data, especially if we Move data out of git repository #301. For example newly generated data can be stored on a preview S3 bucket and the web app can fetch data from it instead of from production bucket.