Using external data (APIs) as an optional replacement for CSV files

[brockfanning]

@philipashlock @Kali2017SDG
I'm starting this issue to correspond to the "Database Linking" item in the development priorities. There has been discussion of allowing for external data, through API calls in javascript, as an optional replacement for the CSV file approach. This would allow the use of this platform along with a completely decoupled data management system.

To get started on development, we'd need some real-world use-case to development against - preferably a public API that's already out there. In #852 @soho501 mentioned the SDG API, which has a non-final test version that seems like a good first candidate.

[brockfanning]

@SmithersA ^^

[brockfanning]

@philipashlock @SmithersA On second thought I shifted over to a build-time approach, rather than run-time with javascript. It seemed like a re-architecture of all of the data visualizations (tables, line graphs, bar graphs, etc.) to use an abstracted data-grabbing function would have been a lot of work, and also carry a performance cost. Instead, I thought I'd first go with a build-time approach of importing data with a server-side script and saving it as CSV data in whatever way the platform already expects.

I've put up an example script as a pull request: #861

@soho501 FYI this example script used the SDGAPI as a data source. Hopefully I didn't hammer the API servers too much while testing it. :)

[brockfanning]

@philipashlock pointed out an inherent limitation of the build-time approach: the data could frequently be out-of-date in between site builds. Let's explore the run-time approach too.

[brockfanning]

@philipashlock @SmithersA #887 is meant to provide the basis for a run-time approach to external data sources.

[brockfanning]

@philipashlock @SmithersA #901 is a next step here - showing a proof-of-concept of getting data from a CKAN service. More details are in the pull request.

[brockfanning]

@philipashlock @SmithersA I've updated #901 with a demo of the CKAN data source here.