Noaadata-web-server-metrics enables anyone to use the scripts that show the metrics for the NSIDC NOAA dataset downloads. The data is hosted in this repo: noaadata-web-server
- This repository is not actively supported by NSIDC but we welcome issue submissions and pull requests in order to foster community contribution.
See the LICENSE for details on permissions and warranties. Please contact nsidc@nsidc.org for more information.
Docker + docker-compose
OR
Python + Conda
Install a pre-built image from DockerHub:
docker pull nsidc/noaadata-web-server-metrics
There are two cli functions to run.
-
Ingest: The ingest function will run daily to read in the download logs and then output daily json files to /share/logs with necessary information for the report. Use
--help
to learn more. -
Report The report function generates the CSV report that will be mailed to recipients. Use
--help
to learn more. To send to multiple emails put-m
before each email.
source VERSION.env
.
./scripts/cli.sh ingest -s 2023-01-01 -e 2023-04-01
.
./scripts/cli.sh report -s 2023-01-01 -e 2023-04-01 -m email@colorado.edu -m email@email.com
.
PYTHONPATH=. python noaa_metrics/cli.py ingest -s 2023-01-01 -e 2023-04-07
.
PYTHONPATH=. python noaa_metrics/cli.py report -s 2023-01-01 -e 2023-04-07 -m email@email.com
.
- Deploy app with Garrison.
- Set the version properly.
source /opt/deploy/noaadata-web-server-metrics/VERSION.env
- Run ingest daily.
/opt/deploy/noaadata-web-server-metrics/scripts/cli.sh ingest -s 2023-01-01 -e 2023-01-01
- Run report on specified schedules or adhoc.
/opt/deploy/noaadata-web-server-metrics/scripts/cli.sh report -s 2023-01-01 -e 2023-04-01 -m roma8902@colorado.edu
Make sure that share/logs/noaa-web/ingest
and share/logs/noaa-web/report
are created in each environment.
parse_nginx_logs.sh
was created in response to an issue listed in PSS-677
The script is in scripts
directory and can be run on the access.log
if for some reason the download.log
does not get created for a day or various days. An example of how this was run to recover the missing December days for PSS-667 is below:
grep "01/Dec/2024" /share/logs/noaa-web-all/production/access.log > ~/noaadata-web-server-metrics/scripts/01dec2024_access.log
./parse_nginx_lines.sh 01dec2024_access.log > 01dec2024_download.log
In that scenario if you then want to run the ingest locally you will need to change the input path in noaa_metrics/ingest_logs.py
See LICENSE.
See Code of Conduct.
This software was developed by the National Snow and Ice Data Center with funding from multiple sources.