-
Notifications
You must be signed in to change notification settings - Fork 92
Technical Call: October 14, 2019
Open SDG Tech call 14 Oct 2019
Present: Brock Fanning, Lucy Gwilliam, Savvas Stephanides, Ann Corp
Headline data not being recognised as headline data eg area. Brock noted SDG Build has new feature to alter data: ‘Add data alteration’ method. Same issue found with Cambodia POC. Brock will do an example of something for Cambodia and send to us.
Also issue with Git indicator name which needs further investigation. Brock advised could have a plug-ins folder for Jekyll to load. And could try using this to override one of the layouts and help identify where issue arising. Lucy to work with Brock on this.
General principle is to avoid writing our own code where possible and Brock advised there is a Python library we could explore – it seems well supported. Aim to convert import function eventually too. SDG Build library when data is imported, internally its in format of Pandas data frame, so it's there.
Action: Lucy find out from contacts at SDMX conference what they use for SDMX export. Ditto re Brock finding out what used to export.
Brock thinks could do in SDB Build as a new feature, so that when site gets built, can zip them up and save them in a predictable place. Use some form of library. Create ticket. Note: SDG build has newer style using python classes as well as the older more procedural style. Suggest move towards the newer style. Could be another ‘output’ – currently only have ‘Open SDG’ format
Action: UK Add info re this to issue in Open SDG
Storing data in relational database rather than the static files. Would be easier to implement advanced searching, and other features.
Brock: with input strategy (SDMX, .CSV .YAML, CKAN). Open SDG currently coded to expect a certain format: json. Open SDG tightly coupled to specific JSON format. So our strategy has been to convert the various input formats into the required .json format – therefore don’t have to change Open SDG.
So two approaches to moving to relational db:
- Queries data from MySQL at "build time" and convert to .json. Open SDG not dynamic, but SDG Build pulls in and creates files.
- Dynamic querying of MySQL at "run time". If there are features that need dynamic querying, then Open SDG would need to change.
Potential benefits of moving to relational DB that we discussed:
- Ease of data management – could be done without changing Open SDG
- Ability to dynamically query – would require changes to Open SDG, need to discuss in more detail what user needs are (searching by metadata, disaggregation, natural language)
Agreed our custom-built search feature is quite rigid. Savvas thinks database backend would make search improvements easier. Brock acknowledges that there are issues (costs) relating to hosting. There is a ticket in Open SDG that could work to improve search – possibly Lunar.js