This repository aims to provide a platform for thinking about, and developing, a unified view of metadata elements required to describe climate indices (aka climate indicators).
Currently, the main component is the spreadsheet file, master_table.xls, that contains several tables as separate sheets. The .xls format should [hopefully] be readable/editable by several common desktop office programs.
The repository also contains a small python package that can convert the
spreadsheet into a set of equivalent yaml files that are
better suited for automatic processing. For more information about the python
program, take a look at the python sub-directory.
We encourage contributions via discussions in issues. If you have a suggestion for the improvement of the metadata or the distribution of this work, please have a look at the issues. To help you starting a new issue there are three issue templates:
- Request new climate index --- this template is divided into two parts,
one for a general description of the new index, and one part for more precisely
specifying entries in the
index_definitionspeadsheet (see below). There is no need to complete all parts, just fill out those parts that you feel are relevant for your proposal. - Correction to index definition --- this template specifically
refers to the
index_definitionspreadsheet (see below). Here we would like you to specify which table entries are wrong and how they should be changed. - General issue --- any other issue, question or discussion.
Don't hesitate to open a new one to initiate a discussion on any question or topic related to this effort!
Standard github and git approaches apply when using the green button "Code"
(above right), which does not include yaml files of the tables. To the right,
under the heading "Releases"
packaged versions of the repository are available as compressed files.
These include yaml files that are up-to-date with the corresponding tables of
the spreadsheet file. The compressed files are available both for MS-DOS/Windows
line endings (.zip) and Linux line endings (.gz).
To facilitate data exchange and dissemination the metadata should, as far as possible, follow the Climate and Forecasting (CF) Conventions. Considering the very rich and diverse flora of climate indices this is however not always possible. By collecting a wide range of different indices it is easier to discover any common patterns and features that are currently not well covered by the CF Conventions.
With respect to climate index metadata the CF Conventions are in a sense
both permissive and restrictive. This has some bearing on which indices
are at all included in the list, and which metadata elements are present,
as well as the (somewhat subjective) classification as ready or not.
Further details here (click here)
CF is permissive in the sense that only the bare essential information for understanding what the data represents are mandatory (cf. here, especially paragraph 8), and that any additional information can be included with few limitations. Thus, almost any climate index dataset can be published in a CF compliant way, but only with the bare minimum of standardised metadata. For such free (i.e., non-managed) information there are no rules, which means that it is difficult or intractable to develop common standardised workflows that would depend on this particular metadata information. But the CF Conventions also includes a range of more detailed metadata components that are managed according to specific rules. With these managed components it is possible to provide a richer and much more detailed description of the dataset. For these managed components CF is however often restrictive in that the rules are not always well suited to handle climate index metadata.
At the 2021 CF Workshop a plenary presentation outlined the links between metadata requirements for describing the climate indices and the CF Conventions (version 1.9).
In the index definition table the column ready (second left) subjectively
indicates how complete the metadata description is. In general terms, indices
marked as ready ("1") either have full metadata description, or there are
advanced plans for what needs to be done. In particular, the focus is on the
following elements of the CF Conventions:
-
standard_name(recommended if available). -
long_name(free text, recommended). -
unit(required if standard name is used, else recommended). -
cell_methods(recommended if available).
Some relevant open issues at the CF github repository (click here)
-
Issue 101: Clarifying the temperature unit associated with some cell methods and standard names is relevant for all indices that involves temperature differences, either directly (e.g.
dtrandetr), or indirectly (e.g. "percentile indices", liketn10pandwsdi, because they involve anomalies with respect to a reference period). -
Issue 110: Standard names for selected climate indices/indicators based on thresholds covers a lot of ground related to the linkages between cell methods, the somewhat complex CF concept of
climatologicaltime coordinate, and standard names. -
Issue 131: Standard names: *_threshold, allow for percentile based thresholds focusses on how to extend CF and the standard name table to allow thresholds based on percentiles instead of a fixed value.
More details (as per late 2020) regarding the approach towards structuring the climate index metadata is available in an IS-ENES3 project report M10.3 - Climate indicators/indices and file metadata specifications and tools.
This repository is in active development, and the content will frequently change.
The metadata is contained in the spreadsheet file master_table.xls and its
sheets are as follows:
-
README --- explains contents and formatting of the spreadsheet file itself.
-
index_definitions --- the main table holding the metadata for the individual indices. Most of the indices developed by the ETCCDI and ET-SCI are included, as are the ones produced by ECA&D. However, some of the more complex indices remain to be included.
-
variables --- specification of input variables (following CMIP5/6 and CORDEX rules). This sheet also gives non-exhaustive lists of common aliases for the variable names. Its role is to provide a link from the actual variable name in the input file to the standardised variable names used in the index_definitions table. It is not intended to be prescriptive or restrictive regarding what variable names can be used or are suitable.
-
index_functions --- contains details about the calculation methods used for the indices. This is referred to in the index_definitions sheet.
-
ECA&D --- list of indices produced by ECA&D. Many of these are already covered by existing entries in the index_table sheet.
The spreadsheets can be converted to a set of yaml files using the python package found in this repository. These are automatically generated and distributed alongside the spreadsheet in the metadata distributions linked to above.
The purpose of the yaml files is to give the same information as in the spreadsheet in a format that allows for a richer, hierarchical structure that also lends itself better to automatic processing. The downside is, that this makes it slightly more difficult to get a quick overview of all the available information.
At the moment only the index_definitions and variables sheets of the
spreadsheet file are transformed into the corresponding index_definitions.yml
and variables.yml files.
The README sheet, insofar it gives information beyond the present document,
only applies to the spreadsheet. The ECA&D sheet mainly exists to track the
open indices of that collection until they are fully integrated into the main
table index_definitions. Consequently, those two sheets are not likely to be
converted in the future.
The index_functions sheet will be added to the yaml files in the future.
This work is supported by the European Horizon 2020 programme through the projects IS-ENES3 (2019-2023) and CLIMATEUROPE2 (2022-2027), as well as by the SMHI Rossby Centre.
Clix-meta(c) 2020-2023 by Lars Bärring and Klaus Zimmermann, Rossby Centre, Swedish Meteorological and Hydrological Institute (SMHI).
You should have received a copy of the license along with this work. If not, see http://creativecommons.org/licenses/by-sa/4.0/.