Skip to content

Commit

Permalink
Merge pull request #246 from bessagroup/pr/1.4.6
Browse files Browse the repository at this point in the history
Pr/1.4.6
  • Loading branch information
mpvanderschelling committed Dec 7, 2023
2 parents 7495fc4 + 506d0ae commit 7f96e7e
Show file tree
Hide file tree
Showing 43 changed files with 1,833 additions and 674 deletions.
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.4.5
1.4.6
3 changes: 2 additions & 1 deletion docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
sphinx
sphinx_rtd_theme
sphinxcontrib-bibtex
sphinx_autodoc_typehints
sphinx_autodoc_typehints
sphinx-tabs==3.4.4
7 changes: 4 additions & 3 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@
project = 'f3dasm'
author = 'Martin van der Schelling'
copyright = '2022, Martin van der Schelling'
version = '1.4.5'
release = '1.4.5'
version = '1.4.6'
release = '1.4.6'


# -- General configuration ----------------------------------------------------
Expand All @@ -42,7 +42,8 @@
'sphinx.ext.autosummary',
'sphinx.ext.intersphinx',
'sphinx.ext.viewcode',
'sphinx_autodoc_typehints']
'sphinx_autodoc_typehints',
'sphinx_tabs.tabs']

# Source: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-source_suffix
source_suffix = {'.rst': 'restructuredtext', }
Expand Down
31 changes: 15 additions & 16 deletions docs/source/rst_doc_files/classes/datageneration/datagenerator.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ Datagenerator
=============

The :class:`~f3dasm.datageneration.DataGenerator` class is the main class of the :mod:`~f3dasm.datageneration` module.
It is used to generate :attr:`~f3dasm.design.ExperimentData.output_data` for the :class:`~f3dasm.design.ExperimentData` by taking a :class:`~f3dasm.design.ExperimentSample` object.
It is used to generate :attr:`~f3dasm.ExperimentData.output_data` for the :class:`~f3dasm.ExperimentData` by taking a :class:`~f3dasm.ExperimentSample` object.

The :class:`~f3dasm.datageneration.DataGenerator` can serve as the interface between the
:class:`~f3dasm.design.ExperimentData` object and any third-party simulation software.
:class:`~f3dasm.ExperimentData` object and any third-party simulation software.

.. image:: ../../../img/f3dasm-datageneration.png
:width: 70%
Expand All @@ -17,27 +17,27 @@ The :class:`~f3dasm.datageneration.DataGenerator` can serve as the interface bet
|
Use the simulator in the data-driven process
------------------------------------------
--------------------------------------------

In order to run your simulator on each of the :class:`~f3dasm.design.ExperimentSample` of your :class:`~f3dasm.design.ExperimentData`, you follow these steps:
In order to run your simulator on each of the :class:`~f3dasm.ExperimentSample` of your :class:`~f3dasm.ExperimentData`, you follow these steps:
In this case, we are utilizing a one of the :ref:`benchmark-functions` to mock a simulator.

We provide the datagenerator to the :meth:`~f3dasm.design.ExperimentData.evaluate` function with the :class:`~f3dasm.datageneration.DataGenerator` object as an argument.
We provide the datagenerator to the :meth:`~f3dasm.ExperimentData.evaluate` function with the :class:`~f3dasm.datageneration.DataGenerator` object as an argument.

.. code-block:: python
experimentdata.evaluate(data_generator="Ackley", method='sequential', kwargs={'some_additional_parameter': 1})
.. note::

Any key-word arguments that need to be passed down to the :class:`~f3dasm.datageneration.DataGenerator` can be passed in the :code:`kwargs` argument of the :meth:`~f3dasm.design.ExperimentData.evaluate` function.
Any key-word arguments that need to be passed down to the :class:`~f3dasm.datageneration.DataGenerator` can be passed in the :code:`kwargs` argument of the :meth:`~f3dasm.ExperimentData.evaluate` function.


There are three methods available of handeling the :class:`~f3dasm.design.ExperimentSample` objects:
There are three methods available of handeling the :class:`~f3dasm.ExperimentSample` objects:

* :code:`sequential`: regular for-loop over each of the :class:`~f3dasm.design.ExperimentSample` objects in order
* :code:`parallel`: utilizing the multiprocessing capabilities (with the `pathos <https://pathos.readthedocs.io/en/latest/pathos.html>`_ multiprocessing library), each :class:`~f3dasm.design.ExperimentSample` object is run in a separate core
* :code:`cluster`: utilizing the multiprocessing capabilities, each :class:`~f3dasm.design.ExperimentSample` object is run in a separate node. After completion of an sample, the node will automatically pick the next available sample. More information on this mode can be found in the :ref:`cluster-mode` section.
* :code:`sequential`: regular for-loop over each of the :class:`~f3dasm.ExperimentSample` objects in order
* :code:`parallel`: utilizing the multiprocessing capabilities (with the `pathos <https://pathos.readthedocs.io/en/latest/pathos.html>`_ multiprocessing library), each :class:`~f3dasm.ExperimentSample` object is run in a separate core
* :code:`cluster`: utilizing the multiprocessing capabilities, each :class:`~f3dasm.ExperimentSample` object is run in a separate node. After completion of an sample, the node will automatically pick the next available sample. More information on this mode can be found in the :ref:`cluster-mode` section.


Implement your simulator
Expand Down Expand Up @@ -65,16 +65,16 @@ Once you have created the ``data_generator`` object, you can plug-in a pre-proc

pre-processing
^^^^^^^^^^^^^^
The preprocessing method is used to create a simulator input file from the information in the :class:`~f3dasm.design.ExperimentSample`.
The preprocessing method is used to create a simulator input file from the information in the :class:`~f3dasm.ExperimentSample`.


This method should adhere to a few things:

* The first argument of the function needs to be ``experiment_sample`` of type :class:`~f3dasm.design.ExperimentSample`.
* The first argument of the function needs to be ``experiment_sample`` of type :class:`~f3dasm.ExperimentSample`.
* The method should return None.
* The method should create the input file ready for the simulator to process with the job_number as name (``experiment_sample.job_number``)

You can retrieve the parameters of the :class:`~f3dasm.design.ExperimentSample` object by calling the :meth:`~f3dasm.design.ExperimentSample.get` method.
You can retrieve the parameters of the :class:`~f3dasm.ExperimentSample` object by calling the :meth:`~f3dasm.ExperimentSample.get` method.

You can add the ``pre-process-function`` to the :class:`~f3dasm.datageneration.DataGenerator` object by passing it through the :meth:`~f3dasm.datageneration.DataGenerator.add_pre_process` method:

Expand All @@ -90,13 +90,12 @@ You can add the ``pre-process-function`` to the :class:`~f3dasm.datageneration.D
post-processing
^^^^^^^^^^^^^^^

* :meth:`~f3dasm.datageneration.DataGenerator.execute` - The main functon call to run the simulation. Returns None.
The post-processing method converts the output of the simulator to a ``results.pkl`` `pickle <https://docs.python.org/3/library/pickle.html>`_ file.
This ``results.pkl`` is then loaded into the :class:`~f3dasm.design.ExperimentData` object.
This ``results.pkl`` is then loaded into the :class:`~f3dasm.ExperimentData` object.

This method should adhere to a few things:

* The first argument of the function needs to be ``experiment_sample`` of type :class:`~f3dasm.design.ExperimentSample`.
* The first argument of the function needs to be ``experiment_sample`` of type :class:`~f3dasm.ExperimentSample`.
* The method should return None.
* The method read the output of the simulator (it's name is ``experiment_sample.job_number``) and convert it to a ``results.pkl`` file.
* This pickle file is stored in the current working directory.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ In order to augment the benchmark functions, you can provide 4 keyword arguments
* ``seed``: Seed for the random number generator for the ``noise`` and ``offset`` calculations.

Benchmark functions can substitute the expensive simulation in the
:class:`~f3dasm.design.ExperimentData` object by providing the name of the function as the ``data_generator`` argument:
:class:`~f3dasm.ExperimentData` object by providing the name of the function as the ``data_generator`` argument:

.. code-block:: python
Expand Down
12 changes: 6 additions & 6 deletions docs/source/rst_doc_files/classes/design/domain.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Domain

.. _domain:

The :class:`~f3dasm.design.Domain` is a set of :class:`f3dasm.design.Parameter` instances that make up the feasible search space.
The :class:`~f3dasm.design.Domain` is a set of parameter instances that make up the feasible search space.

.. image:: ../../../img/f3dasm-domain.png
:width: 100%
Expand All @@ -23,7 +23,7 @@ To start, we instantiate an empty domain object:

.. code-block:: python
from f3dasm import Domain
from f3dasm.design import Domain
domain = Domain()
Expand Down Expand Up @@ -108,12 +108,12 @@ and a dictionary with the parameter type (:code:`type`) and the corresponding ar
domain:
param_1:
type: float
lower_bound: -1.0
upper_bound: 1.0
low: -1.0
high: 1.0
param_2:
type: int
lower_bound: 1
upper_bound: 10
low: 1
high: 10
param_3:
type: category
categories: ['red', 'blue', 'green', 'yellow', 'purple']
Expand Down
44 changes: 22 additions & 22 deletions docs/source/rst_doc_files/classes/design/experimentdata.rst
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
Experiment Data
===============

The :class:`~f3dasm.design.ExperimentData` object is the main object used to store implementations of a design-of-experiments,
The :class:`~f3dasm.ExperimentData` object is the main object used to store implementations of a design-of-experiments,
keep track of results, perform optimization and extract data for machine learning purposes.

All other processses of f3dasm use this object to manipulate and access data about your experiments.

The :class:`~f3dasm.design.ExperimentData` object consists of the following attributes:
The :class:`~f3dasm.ExperimentData` object consists of the following attributes:

- :ref:`domain <domain-format>`: The feasible :class:`~f3dasm.design.Domain` of the Experiment. Used for sampling and optimization.
- :ref:`input_data <input-data-format>`: Tabular data containing the input variables of the experiments as column and the experiments as rows.
Expand All @@ -25,7 +25,7 @@ The :class:`~f3dasm.design.ExperimentData` object consists of the following attr

Users of :mod:`f3dasm` are advised to not directly manipulate the attributes of the ExperimentData object. Instead, the methods of ExperimentData should be used to manipulate the data.

The :class:`~f3dasm.design.ExperimentData` object can be constructed in several ways:
The :class:`~f3dasm.ExperimentData` object can be constructed in several ways:

* :ref:`By providing your own data <experimentdata-own>`
* :ref:`Reconstructed from the project directory <experimentdata-file>`
Expand All @@ -37,15 +37,15 @@ The :class:`~f3dasm.design.ExperimentData` object can be constructed in several
ExperimentData from your own data
---------------------------------

You can construct a :class:`~f3dasm.design.ExperimentData` object by providing it :ref:`input_data <input-data-format>`, :ref:`output_data <output-data-format>`, a :ref:`domain <domain-format>` object and a :ref:`filename <filename-format>`.
You can construct a :class:`~f3dasm.ExperimentData` object by providing it :ref:`input_data <input-data-format>`, :ref:`output_data <output-data-format>`, a :ref:`domain <domain-format>` object and a :ref:`filename <filename-format>`.

.. code-block:: python
>>> from f3dasm import ExperimentData
>>> data = ExperimentData(domain, input_data, output_data)
The following sections will explain how to construct a :class:`~f3dasm.design.ExperimentData` object from your own data.
The following sections will explain how to construct a :class:`~f3dasm.ExperimentData` object from your own data.

.. _domain-format:

Expand Down Expand Up @@ -154,7 +154,7 @@ Several datatypes are supported for the ``output_data`` argument:
>>> domain = Domain({'x0': ContinuousParameter(0., 1.)}, 'x1': ContinuousParameter(0., 1.)})
>>> data = ExperimentData(input_data="my_experiment_data.csv", domain=domain)

If you don't have output data yet, you can also construct an :class:`~f3dasm.design.ExperimentData` object without providing output data.
If you don't have output data yet, you can also construct an :class:`~f3dasm.ExperimentData` object without providing output data.


.. _filename-format:
Expand All @@ -172,7 +172,7 @@ You can provide a string or a path to a directory. If the directory does not exi
>>> project_dir = "folder/to/my_project_directory"
>>> data = ExperimentData(project_dir=project_dir)
You can also set the project directoy manually after creation with the :meth:`~f3dasm.design.ExperimentData.set_project_dir` method"
You can also set the project directoy manually after creation with the :meth:`~f3dasm.ExperimentData.set_project_dir` method"

.. code-block:: python
Expand All @@ -187,7 +187,7 @@ You can also set the project directoy manually after creation with the :meth:`~f
ExperimentData from project directory
-------------------------------------

If you already have constructed the :class:`~f3dasm.design.ExperimentData` object before, you can retrieve it from disk by calling the :meth:`~f3dasm.design.ExperimentData.from_file`
If you already have constructed the :class:`~f3dasm.ExperimentData` object before, you can retrieve it from disk by calling the :meth:`~f3dasm.ExperimentData.from_file`
classmethod with the path of project directory:

.. code-block:: python
Expand All @@ -200,7 +200,7 @@ classmethod with the path of project directory:
ExperimentData from a sampling
------------------------------

You can directly construct an :class:`~f3dasm.design.ExperimentData` object from a sampling strategy by using the :meth:`~f3dasm.design.ExperimentData.from_sampling` method.
You can directly construct an :class:`~f3dasm.ExperimentData` object from a sampling strategy by using the :meth:`~f3dasm.ExperimentData.from_sampling` method.
You have to provide the following arguments:

* A sampling function. To learn more about integrating your sampling function, please refer to :ref:`this <integrating-sampling>` section.
Expand Down Expand Up @@ -247,9 +247,9 @@ ExperimentData from a `hydra <https://hydra.cc/>`_ configuration file
---------------------------------------------------------------------
If you are using `hydra <https://hydra.cc/>`_ for configuring your experiments, you can use it to construct
an :class:`~f3dasm.design.ExperimentData` object from the information in the :code:`config.yaml` file with the :meth:`~f3dasm.design.ExperimentData.from_yaml` method.
an :class:`~f3dasm.ExperimentData` object from the information in the :code:`config.yaml` file with the :meth:`~f3dasm.ExperimentData.from_yaml` method.
You can create an experimentdata :class:`~f3dasm.design.ExperimentData` object in the same ways as described above, but now using the hydra configuration file.
You can create an experimentdata :class:`~f3dasm.ExperimentData` object in the same ways as described above, but now using the hydra configuration file.
.. code-block:: yaml
Expand All @@ -276,7 +276,7 @@ You can create an experimentdata :class:`~f3dasm.design.ExperimentData` object i
The :class:`~f3dasm.design.Domain` object will be constructed using the :code:`domain` key in the :code:`config.yaml` file. Make sure you have the :code:`domain` key in your :code:`config.yaml`!
To see how to configure the :class:`~f3dasm.design.Domain` object with hydra, see :ref:`this <domain-from-yaml>` section.
Inside your python script, you can then create the :class:`~f3dasm.design.ExperimentData` object with the :meth:`~f3dasm.design.ExperimentData.from_yaml` method:
Inside your python script, you can then create the :class:`~f3dasm.ExperimentData` object with the :meth:`~f3dasm.ExperimentData.from_yaml` method:
.. code-block:: python
Expand All @@ -289,9 +289,9 @@ Inside your python script, you can then create the :class:`~f3dasm.design.Experi
.. note::
Make sure to pass the full :code:`config` to the :meth:`~f3dasm.design.ExperimentData.from_yaml` constructor!
Make sure to pass the full :code:`config` to the :meth:`~f3dasm.ExperimentData.from_yaml` constructor!
To create the :class:`~f3dasm.design.ExperimentData` object with the :meth:`~f3dasm.design.ExperimentData.from_sampling` method, you can use the following configuration:
To create the :class:`~f3dasm.ExperimentData` object with the :meth:`~f3dasm.ExperimentData.from_sampling` method, you can use the following configuration:
.. code-block:: yaml
:caption: config.yaml for from_sampling
Expand Down Expand Up @@ -320,7 +320,7 @@ To create the :class:`~f3dasm.design.ExperimentData` object with the :meth:`~f3d
To see how to configure the :class:`~f3dasm.design.Domain` object with hydra, see :ref:`this <domain-from-yaml>` section.
To create the :class:`~f3dasm.design.ExperimentData` object with the :meth:`~f3dasm.design.ExperimentData.from_file` method, you can use the following configuration:
To create the :class:`~f3dasm.ExperimentData` object with the :meth:`~f3dasm.ExperimentData.from_file` method, you can use the following configuration:
.. code-block:: yaml
:caption: config.yaml for from_file
Expand All @@ -331,7 +331,7 @@ To create the :class:`~f3dasm.design.ExperimentData` object with the :meth:`~f3d
Adding data after construction
------------------------------
If you have constructed your :class:`~f3dasm.design.ExperimentData` object, you can add ``input_data``, ``output_data``, a ``domain`` or the ``filename`` using the :meth:`~f3dasm.design.ExperimentData.add` method:
If you have constructed your :class:`~f3dasm.ExperimentData` object, you can add ``input_data``, ``output_data``, a ``domain`` or the ``filename`` using the :meth:`~f3dasm.ExperimentData.add` method:
.. code-block:: python
Expand All @@ -345,7 +345,7 @@ If you have constructed your :class:`~f3dasm.design.ExperimentData` object, you
.. warning::
You can only add data to an existing :class:`~f3dasm.design.ExperimentData` object if the domain is the same as the existing domain.
You can only add data to an existing :class:`~f3dasm.ExperimentData` object if the domain is the same as the existing domain.
Exporting
Expand All @@ -356,7 +356,7 @@ Exporting
Storing the ExperimentData object
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The :class:`~f3dasm.design.ExperimentData` object can be exported to a collection of files using the :meth:`~f3dasm.design.ExperimentData.store` method.
The :class:`~f3dasm.ExperimentData` object can be exported to a collection of files using the :meth:`~f3dasm.ExperimentData.store` method.
.. code-block:: python
Expand All @@ -369,7 +369,7 @@ Inside the project directory, a subfolder `experiment_data` will be created cont
- :code:`output.csv`: The :attr:`~f3dasm.design.ExperimentData.output_data` table
- :code:`jobs.pkl`: The :attr:`~f3dasm.design.ExperimentData.jobs` object
These files are used to load the :class:`~f3dasm.design.ExperimentData` object again using the :meth:`~f3dasm.design.ExperimentData.from_file` method.
These files are used to load the :class:`~f3dasm.ExperimentData` object again using the :meth:`~f3dasm.ExperimentData.from_file` method.
.. code-block:: python
Expand All @@ -394,6 +394,6 @@ Storing to other datatypes
Alternatively, you can convert the input- and outputdata of your data-driven process to other well-known datatypes:
* :class:`~numpy.ndarray` (:meth:`~f3dasm.design.ExperimentData.to_numpy`); creates a tuple of two :class:`~numpy.ndarray` objects containing the input- and outputdata.
* :class:`~xarray.Dataset` (:meth:`~f3dasm.design.ExperimentData.to_xarray`); creates a :class:`~xarray.Dataset` object containing the input- and outputdata.
* :class:`~pd.DataFrame` (:meth:`~f3dasm.design.ExperimentData.to_pandas`); creates a tuple of two :class:`~pd.DataFrame` object containing the input- and outputdata.
* :class:`~numpy.ndarray` (:meth:`~f3dasm.ExperimentData.to_numpy`); creates a tuple of two :class:`~numpy.ndarray` objects containing the input- and outputdata.
* :class:`~xarray.Dataset` (:meth:`~f3dasm.ExperimentData.to_xarray`); creates a :class:`~xarray.Dataset` object containing the input- and outputdata.
* :class:`~pd.DataFrame` (:meth:`~f3dasm.ExperimentData.to_pandas`); creates a tuple of two :class:`~pd.DataFrame` object containing the input- and outputdata.
Loading

0 comments on commit 7f96e7e

Please sign in to comment.