Merge pull request #142 from rosswhitfield/update_nersc_docs

Simplify managing conda environments on NERSC docs

Author: rosswhitfield

.github/workflows/workflows.yml

@@ -36,7 +36,7 @@ jobs:
       matrix:
         os: [ubuntu-latest, macos-latest]
         python-version: [3.6, 3.7, 3.8, 3.9]
-        dask-version: ['2021.07.2']
+        dask-version: ['2021.08.0']
         include:
           # same as NERSC Cori module python/3.7-anaconda-2019.10
           - os: ubuntu-latest
@@ -97,7 +97,7 @@ jobs:
               python3-pip
               python-is-python3
     - name: Install python testing dependencies
-      run: python -m pip install pytest-cov pytest-timeout psutil dask==2021.07.2 distributed==2021.07.2
+      run: python -m pip install pytest-cov pytest-timeout psutil dask==2021.08.0 distributed==2021.08.0
     - name: Install IPS (in develop mode)
       run: python -m pip install -e .
     - name: testing running IPS (--help)

doc/user_guides/nersc_conda.rst

@@ -2,39 +2,51 @@
 Installing IPS on NERSC
 =======================
 
-NERSC recommends the use of anaconda environments to mange python
+NERSC recommends the use of anaconda environments to manage python
 installs, see `Brief introduction to Python at NERSC
-<https://docs.nersc.gov/development/languages/python/overview/>`_.
+<https://docs.nersc.gov/development/languages/python>`_.
 
 There is a conda environment already constructed and maintained for
 the *atom* project created using the `shareable environment`_
-method. You can activate it by running ``source
-/global/common/software/atom/cori/ips-framework-new/bin/activate``.
+method. You can activate it and run IPS by:
+
+.. code-block:: bash
+
+   module load python
+   source activate /global/common/software/atom/cori/ips-framework-new
+   ips.py --config=simulation.config --platform=platform.conf
 
 Creating you own conda environment
 ----------------------------------
 
-This guide will go through creating a conda environment on NERSC
+This guide will go through creating a conda environment on NERSC and
 installing the IPS Framework using `Option 2: Module + source activate
 <https://docs.nersc.gov/development/languages/python/nersc-python/#option-2-module-source-activate>`_
 
 First, you need to load the python module, then create and activate a
 new conda environment. This will create the conda environment in your
-home directory
+home directory (``$HOME/.conda/envs``):
 
 .. code-block:: bash
 
   module load python
   conda create --name my_ips_env python=3.8 # or any version of python >=3.6
   source activate my_ips_env
 
-Next, get download the IPS Framework and install it into the conda
-environment
+If you would like the same packages and versions in your conda
+environment as found in the python modules on Cori, you can clone that
+environment. In this case using ``python/3.7-anaconda-2019.10``.
+
+.. code-block:: bash
+
+  module load python/3.7-anaconda-2019.10
+  conda create -n my_ips_env --clone base
+  source activate my_ips_env
+
+Next, install IPS-Framework into the conda environment
 
 .. code-block:: bash
 
-  git clone https://github.com/HPC-SimTools/IPS-framework.git
-  cd IPS-framework
   python -m pip install ipsframework
 
 To leave your environment
@@ -43,9 +55,9 @@ To leave your environment
 
   conda deactivate
 
-The example below show how to select the newly create conda
-environment to run use, see `Running Python in a batch job
-<https://docs.nersc.gov/development/languages/python/overview/#running-python-in-a-batch-job>`_
+The example below shows how to select the newly create conda
+environment in a batch script, see `Running Python in a batch job
+<https://docs.nersc.gov/development/languages/python/#running-python-in-a-batch-job>`_
 
 .. code-block:: bash
 
@@ -63,29 +75,25 @@ environment to run use, see `Running Python in a batch job
 Creating a shareable environment on /global/common/software
 -----------------------------------------------------------
 
-Creating an conda environment on /global/common/software is the
-recommend way to have one environment shared between many uses, this
-is covered by `Option 4a: Install your own Python without containers
-<https://docs.nersc.gov/development/languages/python/nersc-python/#option-4a-install-your-own-python-without-containers>`_.
-There may also be performance benefits to running from this location
-instead of your home directory.
+By default when you create a conda environment it will be created in
+``$HOME/.conda/envs``, to create one elsewhere that can be used by
+others you can use the ``--prefix`` option, see `Creating conda
+environments
+<https://docs.nersc.gov/development/languages/python/nersc-python/#creating-conda-environments>`_.
 
-Following the instruction we do
+In this example we are cloning the conda environment from the
+``python/3.7-anaconda-2019.10`` module and install ``ipsframework``.
 
 .. code-block:: bash
 
-  wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
-  bash Miniconda3-latest-Linux-x86_64.sh -b -p /global/common/software/myproject/env
-  source /global/common/software/myproject/env/bin/activate
-
-Then install IPS into the environment, from within the IPS-framework
-source directory::
-
+  module load python/3.7-anaconda-2019.10
+  conda create --prefix /global/common/software/myproject/env --clone base
+  source activate /global/common/software/myproject/env
   python -m pip install ipsframework
 
-The example below show how to select the newly create conda
-environment to run use, see `Running Python in a batch job
-<https://docs.nersc.gov/development/languages/python/overview/#running-python-in-a-batch-job>`_
+The example below shows how to select the newly create conda
+environment in you batch script, see `Running Python in a batch job
+<https://docs.nersc.gov/development/languages/python/#running-python-in-a-batch-job>`_
 
 .. code-block:: bash
 
@@ -94,10 +102,10 @@ environment to run use, see `Running Python in a batch job
   #SBATCH --nodes=1
   #SBATCH --time=5
 
-  source /global/common/software/myproject/env/bin/activate
+  module load python
+  source activate /global/common/software/myproject/env
   ips.py --config=simulation.config --platform=platform.conf
 
-
 Installing dependencies
 ~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -107,92 +115,27 @@ To see which packages are currently install in your environment run:
 
    conda list
 
-You can install just the dependencies you need by
-
-.. code-block:: bash
-
-   conda install matplotlib netcdf4 ...
-
-If you would like the same versions and dependencies in your conda
-environment as found in the python modules on Cori, you can export
-that environment and set your environment to be the same.
-
-Export ``python/3.7-anaconda-2019.10`` to yml file.
-
-.. code-block:: bash
-
-   module load python/3.7-anaconda-2019.10
-   conda env export --name base > environment.yml
-
-   # remove mpi4py and buildtest from environment.yml as these should be installed manually
-   sed -i '/mpi4py/d' environment.yml
-   sed -i '/buildtest/d' environment.yml
-
-Activate your conda environment and force it to match the
-``environment.yml`` file. ``mpi4py`` should be install separately
-`according to NERSC
-<https://docs.nersc.gov/development/languages/python/parallel-python/#mpi4py-in-your-custom-conda-environment>`_.
+You can install any other dependencies you need by
 
 .. code-block:: bash
 
-   source activate my_ips_env
-   conda env update -n my_ips_env --file environment.yml
-
-   # or
-
-   source /global/common/software/myproject/env/bin/activate # your environment
-   # setup base environment
-   conda env update -n base --file environment.yml
-
-Install ``mpi4py`` if needed
-
-.. code-block:: bash
-
-   MPICC="$(which cc) --shared" python -m pip install --no-binary mpi4py mpi4py
-
-Alternatively you can start with the same Anaconda environment which
-will have almost everything you need already and then just install the
-few missing dependencies. As an example, this use the same Anaconda
-distribution as in the ``python/3.7-anaconda-2019.10`` module.
-
-.. code-block:: bash
-
-   wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh
-   bash Anaconda3-2019.10-Linux-x86_64.sh -b -p /global/common/software/myproject/env
-   source /global/common/software/myproject/env
-   conda install netcdf4
-   MPICC="$(which cc) --shared" python -m pip install --no-binary mpi4py mpi4py
-   python -m pip install dask-mpi
+   conda install numpy matplotlib netcdf4 ...
 
 User development
 ~~~~~~~~~~~~~~~~
 
-Miniconda or Anaconda can can be installed to your home directory for easy development of component wrappers.
-
-.. code-block:: bash
-
-   wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh
-   bash Anaconda3-2019.10-Linux-x86_64.sh -b -p $HOME/anaconda3
-   source $HOME/anaconda3/bin/activate
-   conda install netcdf4
-   MPICC="$(which cc) --shared" python -m pip install --no-binary mpi4py mpi4py
-   python -m pip install dask-mpi
-   python -m pip install ipsframework
-
-
-After which you can switch between youy development and the production
-environment on atom project by
+You should keep your development environment separate from the
+production environment. If you do development in your ``my_ips_env``
+conda environment you can switch between that and the production
+environment on the atom project by
 
 .. code-block:: bash
 
    # switch to production environment
-   source /global/common/software/atom/cori/ips-framework-new/bin/activate
+   source activate /global/common/software/atom/cori/ips-framework-new
 
    # switch bask to user development environment
-   source $HOME/miniconda3/bin/activate
-
-To see which environment you are currently in you can run
-
-.. code-block:: bash
+   source activate my_ips_env
 
-   conda env list
+Your bash prompt should be updated to reflect which environment you
+have active.