More automated process for indicator status updates

[brockfanning]

@Kali2017SDG You mentioned it would be helpful to get more automation into the platform regarding indicator status updates, I believe. To kick things off, could you describe the manual process that you're currently doing? For example, what URL do you go to, what do you click on, etc.

[Kali2017SDG]

After Jen (I guess I'll be doing them now) does pull requests, I go onto the staging site and check for a few things: data complete?, graph o.k.?, metadata updated? anything else pending?... assuming all goes well, I then make a list of indicators that should receive "stars" ... Jen or Angela would manually put it in (not sure if it is via editing the code) to show the indicator is done updating for this round.

What I was thinking-- similar to the existence of an URL in the metadata triggering a graph, can we create an entry in the metadata (e.g. Is update complete? Y/N) to "toggle" the star?

Thanks for looking into this.

[brockfanning]

@Kali2017SDG I've given this a try on #855. I'm not sure of an easy way to test prose.io fields, but in theory this should add a checkbox for "Is update complete?" when you're editing the metadata.

[brockfanning]

@Kali2017SDG @philipashlock Just noting a recap of our discussion today about this:

• The PR I submitted earlier will be closed, in favor of a more automatic date-calculation approach (already closed)
• We will get a date field in the metadata converted to a machine-readable format, going forward (if not already done)
  • Question: would that be scheduled_update_by_national_source or date_metadata_updated (or something else)?
• Criteria for "completeness" of data updates: (Kali please correct me if I got this wrong or missed something)
  • Metadata was updated
  • Graphing is enabled and working
  • Data includes certain range of years
• Possible approach for automatically identifying complete data updates:
  • Programmatically check that metadata was updated (this possible?)
  • Programmatically check that graphing is enabled
  • Programmatically check the date field against a platform-wide "last data call" date.
    • Or maybe a flat 6 months as an alternative

[brockfanning]

Looping in @SmithersA. Angela, see some notes above about trying to automate the display of the "star" next to indicators that are "complete" after a data call.

[SmithersA]

@brockfanning & @Kali2017SDG ~ probably added the "updated" tag regardless of working graph (because sometimes there wasn't a graph)

[philipashlock]

I have a demo of using the logic to key off of a date_metadata_updated field that is within the past 6 months. You can see this at the bottom of 1.5.3 and here in the code. If we want to take this approach we can add this logic to _layouts/goal.html

This will require a well formatted date (e.g. 2018-02-02) and will obviously require that field to be actually updated when the metadata is updated. The six month calculation will be based on the last time the website is compiled which currently is only done whenever an update is pushed to the site, but we can configure automatic builds every day in the future if needed.

There may be more automated ways (with a plugin or pre-commit script) that we can explore so this field doesn't have to be manually edited, but I'm not sure if those solutions will be available to us with the current workflow. We could also use javascript to calculate the last six months date rather than calculate using the last site build date.

[brockfanning]

This seems like a good direction to me. Would a reasonable next step be to write a one-off script that tries to convert all the invalid date_metadata_updated values into the proper format?

[philipashlock]

I think we may need to do some manual review and updates that wouldn't be super easy to do with a script unless you wanted to query the GitHub API for commit dates. I don't think we have a lot to update though so I figured I'd create a new view to better understand where we are. The new table view here lists all indicators, their status, and the last updated date entered into the metadata. Here's how it works:

• The gold star indicates where we currently hard code the status into the template
• The text "STAR" indicates whether the date_metadata_updated could be read and was within the past 6 months
• The next column shows the date for date_metadata_updated if it was provided. It's displayed here in a standardized format (it tries to convert from month names to dates as needed)
• There are also links to view the history of edits to the CSV and MD files for easy reference of the last dates they were each updated

If the star image is present, the text "STAR" is present, and the date is present then we probably don't need to make any change. The one thing we might want to update is if the date is on the first of the month, since that implies that a specific day of the month wasn't specified for the date_metadata_updated and we should update the date with the exact date in the standardized format. 1.2.1 is an example of this situation.

If the star image is present and the text "STAR" is not present it either means no valid date was provided and we'll need to update the metadata with date_metadata_updated in the standardized format - or if a good date was provided but is not within the past 6 months then everything is behaving as we expect and the star to no longer be visible when we roll out this new automated approach. 3.4.2 is an example where no valid date was provided and 3.c.1 is an example where a date was provided, but it's more than 6 months ago.

If the star image is not present and the text "STAR" is present that should mean that the star image has just not been manually added for this indicator, but we shouldn't need to take any action since it will be displayed automatically when we roll out this new automated approach. 1.5.3 is an example of this scenario.

So really the main thing we want to review here is to make sure that we have a good date entered for date_metadata_updated on anything that's been updated in the past 6 months (ideally for everything, but the priority will be for all changes we want to use to trigger this star to display). So any place in this table where there's a star image, but no "STAR" text and no date listed is an example where we'll definitely need to add a date in the metadata, e.g. 3.1.2, 3.2.1, 3.2.2, 3.4.2, etc.

[brockfanning]

@philipashlock @Kali2017SDG @SmithersA Who would know the correct dates to add for the indicators in question? (3.1.2, 3.2.1, 3.2.2, 3.4.2, etc.) Or should we just use the last time the file was changed in Github?

Also meanwhile, I've submitted a pull request that implements date validation for date_metadata_updated via CircleCI: #902