diff --git a/HazImp.pdf b/HazImp.pdf index bd375da..d34e7dc 100755 Binary files a/HazImp.pdf and b/HazImp.pdf differ diff --git a/conf.py b/conf.py index 4104533..baa3949 100755 --- a/conf.py +++ b/conf.py @@ -45,16 +45,16 @@ # General information about the project. project = u'HazImp' -copyright = u'2020, Geoscience Australia' +copyright = u'2021, Geoscience Australia' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. -version = '0.6' +version = '1.0' # The full version, including alpha/beta/rc tags. -release = '0.6' +release = '1.0' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/code.rst b/docs/code.rst old mode 100755 new mode 100644 similarity index 75% rename from code.rst rename to docs/code.rst index f32dd4f..6a7852b --- a/code.rst +++ b/docs/code.rst @@ -1,30 +1,33 @@ -Auto Generated Documentation -============================ - - -Pipeline -======== -.. automodule:: hazimp.pipeline - :members: - -Context -======= -.. automodule:: hazimp.context - :members: - -Templates -========= -.. automodule:: hazimp.templates - :members: - -Aggregate -========= -.. automodule:: hazimp.aggregate - :members: - - - -Miscellaneous -============= -.. automodule:: hazimp.misc +Auto Generated Documentation +============================ + + +Pipeline +-------- +.. automodule:: hazimp.pipeline + :members: + +Context +------- +.. automodule:: hazimp.context + :members: + +Templates +--------- +.. automodule:: hazimp.templates + :members: + +Aggregate +--------- +.. automodule:: hazimp.aggregate + :members: + +Raster +------ +.. automodule:: hazimp.raster + :members: + +Miscellaneous +------------- +.. automodule:: hazimp.misc :members: \ No newline at end of file diff --git a/docs/contributing.rst b/docs/contributing.rst index 04ee79c..ea801c5 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -36,7 +36,11 @@ Finally, go to the web page of the your fork of the hazimp repo, and click 'Pull request' to send your changes to the maintainers for review. This will send an email to the committers. **Pull requests should only be made against the 'develop' branch of the repository.** Pull requests -against the 'master' branch will not be approved. +against the 'master' branch will be refused and requested to resubmit against +the 'develop' branch. + +A demonstration of the git workflow used for HazImp can be seen at +https://leanpub.com/git-flow/read. (If any of the above seems like magic to you, then look up the `Git documentation `_ on the web.) diff --git a/docs/history.rst b/docs/history.rst new file mode 100644 index 0000000..d4c4c7c --- /dev/null +++ b/docs/history.rst @@ -0,0 +1,60 @@ +.. _history: + +History +======= + +1.0.0 (2021-03-12) +------------------ + +* Add provenance records +* Add categorisation, tabulation and aggregation +* Implemented AWS S3 download and upload funcitonality. Also allow the configuration file to be stored on S3 as well. +* Update hazimp-tests.yml - add coveralls, remove flake8 and pytest install +* Calculate percentages of assets in damage state tabulation +* Updates to documentation (user guide and in-code) +* Add NRML schema from https://github.com/gem/oq-nrmllib/tree/master/openquake/nrmllib/schema +* Validate xml-format vulnerability files prior to execution +* Add support for configuring choropleth aggregation fields via the configuration file +* Add aggregation functions to provenance statement +* Add support for generic 'hazard_raster' to wind templates +* Switched template instantiation to use config dictionary, removed use of config list +* Update find_attributes to use config dictionary for locating attributes, and support lookup via a ordered list of keys to ease deprecation +* add domestic eq vulnerability curves in MMI (#35) +* Increase unit test coverage (#36) +* Update to NRML 0.5 schema +* Add EQ template and example +* Split templates into new module separated by hazard type +* Enable hazard_raster attribute_label to be configured +* Improve memory usage for large raster input: No longer reads the entire raster into memory, reads only the cells defined in the exposure data, removed the ability to pass an in-memory array via 'from_array' and added a 'ThreadPoolExecutor' for some performance improvement when reading hazard data for large exposure datasets +* Change default aggregate output format to GeoJSON (#42) + + +0.6 (2020-08-13) +---------------- + +* WIP: First steps on provenance - Initially only implemented in the wind_nc and wind_v5 templates, but several of the jobs include provenance statements +* Add provenance for aggregation file +* Add categorisation and tabulation (pivot table) +* Adding some basic documentation for new functions +* Add updated v5 template +* Template file for tcimpact processing +* WIP: PEP8 conformance +* TCRM-47: Implemented S3 download and upload functionality. +* TCRM-47: Allow config file to be from S3 as well. +* TCRM-88: Moving logger.INFO output to stdout from stderr. Fixed compiler warning. +* Documentation updates +* TCRM-90: Reduce memory usage which require 2 times memory for source and destination arrays. (#18) +* Set raster dtype to GDT_Float32: May not necessarily always be correct, however the previous default was integer which worked for the test cases, except when replacing missing values with `numpy.NaN` (which is ostensibly a float value). +* Fix unittest suite by mocking prov module: Patching various methods under `prov.model.ProvDocument` to allow the test suite to run without error. The tests set up a dummyy context, but missed the provenance module, so we're using `mock` to patch the methods that are called in some tests. + + +0.5 (2020-07-09) +---------------- + +* Implement S3 access + + +0.4 (2020-06-29) +---------------- + +* Provenance tracking and aggregation added \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index 7f525b9..b37fd4c 100755 --- a/docs/index.rst +++ b/docs/index.rst @@ -14,6 +14,8 @@ hazard information. It is command line based and can be executed in parallel. vulnerability categorise aggregate + tabulate + history .. code diff --git a/docs/install.rst b/docs/install.rst index 1eda824..f555783 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -10,13 +10,13 @@ You can either download a zip file containing the HazImp code, or clone the repository (if you have `git` installed) as follows: -Using ssh:: +Using ssh: .. code-block:: bash git clone git@github.com:GeoscienceAustralia/hazimp.git -Using HTTPS:: +Using HTTPS: .. code-block:: bash @@ -43,7 +43,9 @@ set of compatible packages. conda_ is available for Linux, MacOS and Windows environments. Once you have installed miniconda_, create a new environment with a command such -as:: +as: + +.. code-block:: bash conda create -f hazimp.yml @@ -68,7 +70,7 @@ in `hazimp.yml`. System python install ^^^^^^^^^^^^^^^^^^^^^ -On Ubuntu systems, the following requires system administrator privileges.:: +On Ubuntu systems, the following requires system administrator privileges. .. code-block:: bash @@ -80,7 +82,7 @@ On Ubuntu systems, the following requires system administrator privileges.:: Install HazImp -------------- -Install HazImp into your python environment:: +Install HazImp into your python environment: .. code-block:: bash @@ -88,9 +90,10 @@ Install HazImp into your python environment:: Or, if you are interested in modifying HazImp, the following alternative install command will instead provide your python environment with links to -the location where you have downloaded the HazImp source:: +the location where you have downloaded the HazImp source: .. code-block:: bash + python setup.py develop Please read the `Contributing code`_ notes if you wish to modify HazImp. diff --git a/docs/templates.rst b/docs/templates.rst index cdbb9ce..becf782 100644 --- a/docs/templates.rst +++ b/docs/templates.rst @@ -1,4 +1,4 @@ -.. _templates:: +.. _templates: Templates --------- @@ -13,11 +13,12 @@ Templates are used to set out the flow of processing invoked in separate :class:`Job` functions that are then connected into a :class:`PipeLine` that is subsequently executed. -NOTE:: Because some of the :class:`Job` functions in the templates are essential, -the order of key/value pairs in the configuration files is important. The code -will raise a `RuntimeError` if the order is incorrect, or if a mandatory -configuration key is missing. This is even more important if not using the -pre-defined templates +.. NOTE:: + Because some of the :class:`Job` functions in the templates are essential, the + order of key/value pairs in the configuration files is important. The code + will raise a `RuntimeError` if the order is incorrect, or if a mandatory + configuration key is missing. This is even more important if not using the + pre-defined templates. We take the example of the `wind_v4` template. It sets up the following job @@ -26,14 +27,9 @@ sequence in a specific order:: #. LOADCVSEXPOSURE - load the exposure data #. LOADRASTER - load the wind raster data #. LOADXMLVULNERABILITY - load the vulnerability functions -#. SIMPLELINKER - select a group of vulnerability functions - some vulnerability - files may have multiple sets of curves identified by `vulnerabilitySetID` -#. SELECTVULNFUNCTION - link the selected vulnerability function set (specified - by the `vulnerabilitySetID` option) to each exposure asset -#. LOOKUP - do a table lookup to determine the damage index for each asset, - based on the intensity measure level (i.e. the wind speed) -#. CALCSTRUCTLOSS - combine the calculated damage index with the building value - to calculate $$$ loss +#. SIMPLELINKER - select a group of vulnerability functions - some vulnerability files may have multiple sets of curves identified by `vulnerabilitySetID` +#. SELECTVULNFUNCTION - link the selected vulnerability function set (specified by the `vulnerabilitySetID` option) to each exposure asset +#. LOOKUP - do a table lookup to determine the damage index for each asset, based on the intensity measure level (i.e. the wind speed) +#. CALCSTRUCTLOSS - combine the calculated damage index with the building value to calculate $$$ loss #. SAVE - should speak for itself - saves the building level loss data -#. SAVEPROVENANCE - saves provenance data (like the version of HazImp, source of - the hazard data, etc.) +#. SAVEPROVENANCE - saves provenance data (like the version of HazImp, source of the hazard data, etc.) diff --git a/docs/user_guide.rst b/docs/user_guide.rst index 397b26f..6dcd0ca 100755 --- a/docs/user_guide.rst +++ b/docs/user_guide.rst @@ -23,12 +23,12 @@ Quick how-to ------------ Follow the install notes in the README.md file. -A configuration file can be used to define a HazImp simulation. The +A configuration file can be used to define a HazImp simulation. The configuration file is described using yaml, a data serialisation -format. HazImp can also be used by another Python application, by +format. HazImp can also be used by another Python application, by passing the configuration infomation in as a dictionary. -For example, to run a wind example do;:: +For example, to run a wind example do:: cd examples/wind python ../../hazimp/main.py -c wind_v5.yaml @@ -36,16 +36,16 @@ For example, to run a wind example do;:: The -c specifies the configuration file. -HazImp can also be ran in parallel, using mpirun. For example;:: +HazImp can also be ran in parallel, using mpirun. For example:: mpirun -np 4 python ../../hazimp/main.py -c wind_v5.yaml There are a suite of HazImp tests to test the install and code during -software developemnt. To run these, in the root HazImp directory +software developemnt. To run these, in the root HazImp directory do;:: - ./all_tests + ./all_tests @@ -57,9 +57,9 @@ a wind template and a flood template. Templates take into account internal vulnerability curves and the data flow needed to produce loss information, simplifying the configuration file. -NOTE:: The order of key/value pairs in the sample configuration files is -important. The code will raise a `RuntimeError` if the order is incorrect. - +.. NOTE:: + The order of key/value pairs in the sample configuration files is important. + The code will raise a `RuntimeError` if the order is incorrect. Wind Template @@ -69,7 +69,9 @@ Given gust information from TCRM and point exposure data the loss associated with each site is calculated using the wind template. Here is the example wind configuration file (from examples/wind), -which uses the wind template.:: +which uses the wind template. + +.. code-block:: yaml # python hazimp.py -c wind_nc.yaml - template: wind_nc @@ -99,7 +101,7 @@ The rest of the file can be understood by the following key value pairs; The type of template to use. This example describes the *wind_nc* template. *load_exposure* - This loads the exposure data. It has 3 sub-sections:: + This loads the exposure data. It has 3 sub-sections: *file_name* The name of the csv exposure file to load. The first row of the csv @@ -265,7 +267,7 @@ Without Templates Vulnerability functions ----------------------- -See the :ref:`Preparing vulnerability curves` section for guidance on +See the :ref:`Preparing vulnerability curves`_ section for guidance on preparing vulnerability functions for use in HazImp. @@ -273,7 +275,9 @@ Provenance tracking ------------------- The provenance of information used in generating an impact analysis is tracked -using the :ref:`prov` module. +using the Prov_ module. Contributions to the code base should incorporate appropriate provenance statements to ensure consistency. + +.._Prov: https://prov.readthedocs.io/en/latest/ \ No newline at end of file diff --git a/docs/vulnerability.rst b/docs/vulnerability.rst index cab89fc..ddb606f 100644 --- a/docs/vulnerability.rst +++ b/docs/vulnerability.rst @@ -1,3 +1,5 @@ +.. _vulnerability: + Preparing vulnerability curves ============================== @@ -6,7 +8,7 @@ provide information on the vulnerability of different building types to the hazard measure used. The vulnerability describes the level of damage, given a specific intensity of the hazard. The level of damage is given as a value between 0 and 1, and represents the ratio between cost of repair of -an building, and the total replacement cost for that building. Typically, a +a building, and the total replacement cost for that building. Typically, a value of 1 indicates the building will require complete replacement. .. figure:: ../examples/diagrams/example_vuln_curve.png @@ -35,6 +37,12 @@ language (NRML) 0.5, with the schema +Vulnerability files are validated against the NRML 0.5 schema definition. If the +file is not a vaild NRML file, an error will be raised. Note this validation +does not do a complete check of the logical consistency of the values in the +file. For example, NRML includes `meanLRs` and `covLRs`. The value of `covLRs` +must be zero where the `meanLRs` values are zero. These internal consistency +checks are not done by the validation check. Structuring vulnerability functions ----------------------------------- diff --git a/index.rst b/index.rst index fb80339..2699c48 100755 --- a/index.rst +++ b/index.rst @@ -17,10 +17,12 @@ hazard information. It is command line based and can be executed in parallel. Installation Vulnerability Templates + Aggregation Categorise Tabulate Function documentation Contributing + History .. code