Documentation updates

conf.py

@@ -45,16 +45,16 @@
 
 # General information about the project.
 project = u'HazImp'
-copyright = u'2020, Geoscience Australia'
+copyright = u'2021, Geoscience Australia'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = '0.6'
+version = '1.0'
 # The full version, including alpha/beta/rc tags.
-release = '0.6'
+release = '1.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

code.rst → docs/code.rst

@@ -1,30 +1,33 @@
-Auto Generated Documentation
-============================
-
-
-Pipeline
-========
-.. automodule:: hazimp.pipeline
-   :members:
-
-Context
-=======
-.. automodule:: hazimp.context
-   :members:
-
-Templates
-=========
-.. automodule:: hazimp.templates
-   :members:
-
-Aggregate
-=========
-.. automodule:: hazimp.aggregate
-   :members:
-
-
-
-Miscellaneous
-=============
-.. automodule:: hazimp.misc
+Auto Generated Documentation
+============================
+
+
+Pipeline
+--------
+.. automodule:: hazimp.pipeline
+   :members:
+
+Context
+-------
+.. automodule:: hazimp.context
+   :members:
+
+Templates
+---------
+.. automodule:: hazimp.templates
+   :members:
+
+Aggregate
+---------
+.. automodule:: hazimp.aggregate
+   :members:
+
+Raster
+------
+.. automodule:: hazimp.raster
+   :members:
+
+Miscellaneous
+-------------
+.. automodule:: hazimp.misc
    :members:

docs/contributing.rst

@@ -36,7 +36,11 @@ Finally, go to the web page of the your fork of the hazimp repo,
 and click 'Pull request' to send your changes to the maintainers for
 review. This will send an email to the committers. **Pull requests should
 only be made against the 'develop' branch of the repository.** Pull requests
-against the 'master' branch will not be approved.
+against the 'master' branch will be refused and requested to resubmit against
+the 'develop' branch.
+
+A demonstration of the git workflow used for HazImp can be seen at
+https://leanpub.com/git-flow/read.  
 
 (If any of the above seems like magic to you, then look up the 
 `Git documentation <http://git-scm.com/documentation>`_ on the web.)

docs/history.rst

@@ -0,0 +1,60 @@
+.. _history:
+
+History
+=======
+
+1.0.0 (2021-03-12)
+------------------
+
+* Add provenance records
+* Add categorisation, tabulation and aggregation
+* Implemented AWS S3 download and upload funcitonality. Also allow the configuration file to be stored on S3 as well. 
+* Update hazimp-tests.yml - add coveralls, remove flake8 and pytest install
+* Calculate percentages of assets in damage state tabulation
+* Updates to documentation (user guide and in-code)
+* Add NRML schema from https://github.com/gem/oq-nrmllib/tree/master/openquake/nrmllib/schema
+* Validate xml-format vulnerability files prior to execution
+* Add support for configuring choropleth aggregation fields via the configuration file
+* Add aggregation functions to provenance statement
+* Add support for generic 'hazard_raster' to wind templates
+* Switched template instantiation to use config dictionary, removed use of config list
+* Update find_attributes to use config dictionary for locating attributes, and support lookup via a ordered list of keys to ease deprecation
+* add domestic eq vulnerability curves in MMI (#35)
+* Increase unit test coverage (#36)
+* Update to NRML 0.5 schema
+* Add EQ template and example
+* Split templates into new module separated by hazard type
+* Enable hazard_raster attribute_label to be configured
+* Improve memory usage for large raster input: No longer reads the entire raster into memory, reads only the cells defined in the exposure data, removed the ability to pass an in-memory array via 'from_array' and added a 'ThreadPoolExecutor' for some performance improvement when reading hazard data for large exposure datasets
+* Change default aggregate output format to GeoJSON (#42)
+
+
+0.6 (2020-08-13)
+----------------
+
+* WIP: First steps on provenance - Initially only implemented in the wind_nc and wind_v5 templates, but several of the jobs include provenance statements
+* Add provenance for aggregation file
+* Add categorisation and tabulation (pivot table)
+* Adding some basic documentation for new functions
+* Add updated v5 template
+* Template file for tcimpact processing
+* WIP: PEP8 conformance
+* TCRM-47: Implemented S3 download and upload functionality.
+* TCRM-47: Allow config file to be from S3 as well.
+* TCRM-88: Moving logger.INFO output to stdout from stderr. Fixed compiler warning.
+* Documentation updates
+* TCRM-90: Reduce memory usage which require 2 times memory for source and destination arrays. (#18)
+* Set raster dtype to GDT_Float32: May not necessarily always be correct, however the previous default was integer which worked for the test cases, except when replacing missing values with `numpy.NaN` (which is ostensibly a float value).
+* Fix unittest suite by mocking prov module: Patching various methods under `prov.model.ProvDocument` to allow the test suite to run without error. The tests set up a dummyy context, but missed the provenance module, so we're using `mock` to patch the methods that are called in some tests.
+
+
+0.5 (2020-07-09)
+----------------
+
+* Implement S3 access
+
+
+0.4 (2020-06-29)
+----------------
+
+* Provenance tracking and aggregation added

docs/index.rst

@@ -14,6 +14,8 @@ hazard information. It is command line based and can be executed in parallel.
    vulnerability
    categorise
    aggregate
+   tabulate
+   history
 
 
 .. code

docs/install.rst

@@ -10,13 +10,13 @@ You can either download a zip file containing
 the HazImp code, or clone the repository (if you have `git`
 installed) as follows:
 
-Using ssh:: 
+Using ssh:
 
 .. code-block:: bash
 
   git clone [email protected]:GeoscienceAustralia/hazimp.git
 
-Using HTTPS::
+Using HTTPS:
 
 .. code-block:: bash
 
@@ -43,7 +43,9 @@ set of compatible packages. conda_ is available for Linux, MacOS and Windows
 environments. 
 
 Once you have installed miniconda_, create a new environment with a command such
-as::
+as:
+
+.. code-block:: bash
 
   conda create -f hazimp.yml 
 
@@ -68,7 +70,7 @@ in `hazimp.yml`.
 System python install
 ^^^^^^^^^^^^^^^^^^^^^
 
-On Ubuntu systems, the following requires system administrator privileges.::
+On Ubuntu systems, the following requires system administrator privileges.
 
 .. code-block:: bash
 
@@ -80,17 +82,18 @@ On Ubuntu systems, the following requires system administrator privileges.::
 Install HazImp
 --------------
 
-Install HazImp into your python environment::
+Install HazImp into your python environment:
 
 .. code-block:: bash
 
   python setup.py install
 
 Or, if you are interested in modifying HazImp, the following alternative
 install command will instead provide your python environment with links to
-the location where you have downloaded the HazImp source::
+the location where you have downloaded the HazImp source:
 
 .. code-block:: bash
+
   python setup.py develop
 
 Please read the `Contributing code`_ notes if you wish to modify HazImp. 

docs/templates.rst

@@ -1,4 +1,4 @@
-.. _templates::
+.. _templates:
 
 Templates
 ---------
@@ -13,11 +13,12 @@ Templates are used to set out the flow of processing invoked in separate
 :class:`Job` functions that are then connected into a :class:`PipeLine` that is
 subsequently executed.
 
-NOTE:: Because some of the :class:`Job` functions in the templates are essential,
-the order of key/value pairs in the configuration files is important. The code
-will  raise a `RuntimeError` if the order is incorrect, or if a mandatory
-configuration key is missing. This is even more important if not using the
-pre-defined templates
+.. NOTE:: 
+  Because some of the :class:`Job` functions in the templates are essential, the
+  order of key/value pairs in the configuration files is important. The code
+  will raise a `RuntimeError` if the order is incorrect, or if a mandatory
+  configuration key is missing. This is even more important if not using the
+  pre-defined templates.
 
 
 We take the example of the `wind_v4` template. It sets up the following job
@@ -26,14 +27,9 @@ sequence in a specific order::
 #. LOADCVSEXPOSURE - load the exposure data
 #. LOADRASTER - load the wind raster data
 #. LOADXMLVULNERABILITY - load the vulnerability functions
-#. SIMPLELINKER - select a group of vulnerability functions - some vulnerability
-   files may have multiple sets of curves identified by `vulnerabilitySetID`
-#. SELECTVULNFUNCTION - link the selected vulnerability function set (specified
-   by the `vulnerabilitySetID` option) to each exposure asset
-#. LOOKUP - do a table lookup to determine the damage index for each asset,
-   based on the intensity measure level (i.e. the wind speed)
-#. CALCSTRUCTLOSS - combine the calculated damage index with the building value
-   to calculate $$$ loss
+#. SIMPLELINKER - select a group of vulnerability functions - some vulnerability files may have multiple sets of curves identified by `vulnerabilitySetID`
+#. SELECTVULNFUNCTION - link the selected vulnerability function set (specified by the `vulnerabilitySetID` option) to each exposure asset
+#. LOOKUP - do a table lookup to determine the damage index for each asset, based on the intensity measure level (i.e. the wind speed)
+#. CALCSTRUCTLOSS - combine the calculated damage index with the building value to calculate $$$ loss
 #. SAVE - should speak for itself - saves the building level loss data
-#. SAVEPROVENANCE - saves provenance data (like the version of HazImp, source of
-   the hazard data, etc.)
+#. SAVEPROVENANCE - saves provenance data (like the version of HazImp, source of the hazard data, etc.)

docs/user_guide.rst

@@ -23,29 +23,29 @@ Quick how-to
 ------------
 Follow the install notes in the README.md file.
 
-A configuration file can be used to define a HazImp simulation.  The
+A configuration file can be used to define a HazImp simulation. The
 configuration file is described using yaml, a data serialisation
-format.  HazImp can also be used by another Python application, by
+format. HazImp can also be used by another Python application, by
 passing the configuration infomation in as a dictionary. 
 
-For example, to run a wind example do;::
+For example, to run a wind example do::
 
      cd examples/wind
      python ../../hazimp/main.py  -c wind_v5.yaml
 
 
 The -c specifies the configuration file.
 
-HazImp can also be ran in parallel, using mpirun.  For example;::
+HazImp can also be ran in parallel, using mpirun. For example::
 
      mpirun -np 4 python ../../hazimp/main.py  -c wind_v5.yaml
  
 
 There are a suite of HazImp tests to test the install and code during
-software developemnt.  To run these, in the root HazImp directory
+software developemnt. To run these, in the root HazImp directory
 do;::
 
-    ./all_tests     
+    ./all_tests
 
 
 
@@ -57,9 +57,9 @@ a wind template and a flood template. Templates take into account internal
 vulnerability curves and the data flow needed to produce loss information,
 simplifying the configuration file.
 
-NOTE:: The order of key/value pairs in the sample configuration files is 
-important. The code will raise a `RuntimeError` if the order is incorrect.
-
+.. NOTE::
+   The order of key/value pairs in the sample configuration files is important.
+   The code will raise a `RuntimeError` if the order is incorrect.
 
 
 Wind Template
@@ -69,7 +69,9 @@ Given gust information from TCRM and point exposure data the loss
 associated with each site is calculated using the wind template.  
 
 Here is the example wind configuration file (from examples/wind),
-which uses the wind template.::
+which uses the wind template.
+
+.. code-block:: yaml
 
      #  python hazimp.py -c wind_nc.yaml
       - template: wind_nc
@@ -99,7 +101,7 @@ The rest of the file can be understood by the following key value pairs;
     The type of template to use.  This example describes the *wind_nc* template.
 
 *load_exposure*
-    This loads the exposure data. It has 3 sub-sections::
+    This loads the exposure data. It has 3 sub-sections:
 
     *file_name*
         The name of the csv exposure file to load. The first row of the csv
@@ -265,15 +267,17 @@ Without Templates
 Vulnerability functions
 -----------------------
 
-See the :ref:`Preparing vulnerability curves` section for guidance on
+See the :ref:`Preparing vulnerability curves`_ section for guidance on
 preparing vulnerability functions for use in HazImp.
 
 
 Provenance tracking
 -------------------
 
 The provenance of information used in generating an impact analysis is tracked
-using the :ref:`prov` module. 
+using the Prov_ module. 
 
 Contributions to the code base should incorporate appropriate provenance
 statements to ensure consistency.
+
+.._Prov: https://prov.readthedocs.io/en/latest/

docs/vulnerability.rst

@@ -1,3 +1,5 @@
+.. _vulnerability:
+
 Preparing vulnerability curves
 ==============================
 
@@ -6,7 +8,7 @@ provide information on the vulnerability of different building types
 to the hazard measure used. The vulnerability describes the level of
 damage, given a specific intensity of the hazard. The level of damage is given
 as a value between 0 and 1, and represents the ratio between cost of repair of
-an building, and the total replacement cost for that building. Typically, a
+a building, and the total replacement cost for that building. Typically, a
 value of 1 indicates the building will require complete replacement.
 
 .. figure:: ../examples/diagrams/example_vuln_curve.png
@@ -35,6 +37,12 @@ language (NRML) 0.5, with the schema
         </vulnerabilityModel>
     </nrml>
 
+Vulnerability files are validated against the NRML 0.5 schema definition. If the
+file is not a vaild NRML file, an error will be raised. Note this validation
+does not do a complete check of the logical consistency of the values in the
+file. For example, NRML includes `meanLRs` and `covLRs`. The value of `covLRs`
+must be zero where the `meanLRs` values are zero. These internal consistency
+checks are not done by the validation check.
 
 Structuring vulnerability functions
 -----------------------------------

index.rst

@@ -17,10 +17,12 @@ hazard information. It is command line based and can be executed in parallel.
    Installation <docs/install>
    Vulnerability <docs/vulnerability>
    Templates <docs/templates>
+   Aggregation <docs/aggregate>
    Categorise <docs/categorise>
    Tabulate <docs/templates>
    Function documentation <docs/code>
    Contributing <docs/contributing>
+   History <docs/history>
 
 .. code