Skip to content

inventory.data.gov

Jump to bottom
Fuhu Xia edited this page Sep 7, 2023 · 22 revisions

a.k.a Inventory is used by federal agencies to manage metadata for their datasets. Inventory is used to generate the agency's data.json which must be hosted on the agency's website (e.g. agency.gov/data.json). Inventory is a CKAN instance and can be used to host datasets in addition to metadata.

Access

Access to Inventory is historically confusing. There are several mechanisms referring to public/private and each means something different. Since Inventory contains only open data, all data within Inventory can be publicly accessible. However for historical reasons, datasets are only visible to authenticated users while resources are publicly visible.

  • CKAN private: true a property on the dataset which is seen on the organization datasets listing and affects visibility within CKAN. Only members of an organization can see private datasets within their own organizations. Since Inventory only contains open data and any metadata is published on catalog.data.gov, there is no reason to mark a dataset as private. The CKAN private field is ignored by Inventory's data.json export.
  • DCAT-US “public access level” doesn’t mean anything to CKAN and does not affect how the dataset is displayed within Inventory. A dataset with "accessLevel": "non-public" will still appear in the data.json inventory.
  • Publishing status (draft or published) is an Inventory concept which affects whether or not the dataset is included in the data.json when exporting from Inventory. It does not affect visibility within CKAN. Any authenticated users will still be able to see draft datasets. Public users be able to see resources of draft datasets. TODO: is this defined in ckanext-datajson?

Resources, the actual data files uploaded to individual datasets, do not have a concept of private and inherit visibility settings from the dataset. Any dataset that includes resource files hosted on Inventory must be marked private: false, otherwise the resource files will not be accessible to anonymous users. This includes some of GSA's hosted datasets that are available by download or the datastore API.

Why are datasets not visible to the public? 🤷 maybe because the confusion around Publishing Status. See GSA/data.gov#2095.

Environments

Instance Url
Production inventory.data.gov
Staging inventory-stage-datagov.app.cloud.gov
sandbox inventory-dev-datagov.app.cloud.gov

Dependencies

Sub-components:

  • ckan

Services:

  • nginx
  • PostgreSQL
  • redis
  • S3
  • SOLR

Common tasks

Updating Publishers List

Inventory organizations have a pre-approved list of "publishers" that can be listed in the metadata for a dataset. This list is then uploaded to the inventory organizations, and is used to create a list for the dataset creator. The list is currently here To update the list, you can use the icon in the top right (pencil icon) or clone it and make the necessary changes locally. Once the edits/additions have been made, create a pull request for the changes (similar to https://github.com/GSA/inventory-app/pull/469). Add a team member (or the whole team) as a reviewer. Once the PR is merged, then an automated deployment of this csv occurs (after deployment). You can verify the deployment in GitHub Actions.

Importing from data.json

ckanpyimport is used in onboarding new agencies to inventory.data.gov. This tool imports datasets from a data.json file.

The import script will happily create duplicates, so if there are any existing datasets in the organization, you probably should delete them all first.

Run this from the jumpbox using nohup or tmux so that disconnecting your session does not interrupt the script. The script can take a while depending on how many packages need to be imported (~2 hours for 1000 datasets). You should also test against staging before running against production.

Adding a User

Once a user has registered via google form and application has been approved by admins, a CKAN user can be added with an Editor role for the agency with the following steps:

  1. Given user's name and email (lowercase), log into cloud.gov CLI and run cf command:
cf run-task inventory --command "ckan user add firstname-lastname [email protected] password=\'\$(cat /proc/sys/kernel/random/uuid)\'" -k 1500M -m 2G
  1. Monitor the output and make sure user is successfully created:
# task-name is printed from previous command
cf tail -f -t log inventory| grep some-task-name
  1. Log into Inventory web UI, go to the Agency's organization page and add user as an Editor.

In case there is an error in step 2 complaining the email address has been used by another registered user, check the email in the user table in the DB. If it belongs to a deleted user, the user has to be set active with a DB query.

Alert conditions

Health Checks

This is configured with cloud.gov, see current configuration as code here.

Database Initialization

This should be handled automatically: https://github.com/GSA/inventory-app/blob/main/.profile#L90-L95

Clone this wiki locally