Merge pull request #107 from jackapbutler/add-cluster-env-script

feat: add cluster conda environment script

Author: maw501

scripts/CLUSTER.md

@@ -0,0 +1,25 @@
+# Working with the Stability cluster
+
+We currently run our large scale experiments on the Stability AI HPC cluster.
+This subdirectory features a few helpful scripts that can help you get up and
+running on the cluster.
+
+1. [Install Miniconda](stability-cluster/miniconda_install.sh) -
+   installs miniconda for your cluster environment.
+
+2. [Create Environment](stability-cluster/env_creation.sh) -
+   creates a basic conda environment for experiments.
+
+   - Creates a conda environment at the prefix `CONDA_ENV_PATH` path.
+     > Using the positional argument passed into the script
+   - Clones `chemnlp` into your personal cluster `USER` directory.
+   - Installs the current revision of the `chemnlp` repository and
+     dependencies that are in your personal directory into the conda environment.
+
+   ```bash
+   # for creating a personal environment
+   source scripts/stability-cluster/env_creation.sh jack/ jack/
+
+   # for creating an experiment environment (usually called by another script)
+   source scripts/stability-cluster/env_creation.sh experiments/my-experiment jack/
+   ```

scripts/stability-cluster/env_creation.sh

@@ -0,0 +1,27 @@
+#! /bin/bash
+### This script creates a conda environment for chemnlp
+### The first arg ($1) is the prefix directory where the environment is saved
+### The second arg ($2) is the directory to use when building the environment
+
+## Must already have miniconda installed!
+export CONDA_ENV_PATH=/fsx/proj-chemnlp/$1/conda/env/chemnlp-standard
+export PYTHON_VER=3.8
+
+## ensure we can use activate syntax in slurm scripts
+CONDA_BASE=$(conda info --base)
+source $CONDA_BASE/etc/profile.d/conda.sh
+
+# Create Python environment through conda
+conda create --force --prefix ${CONDA_ENV_PATH} python=${PYTHON_VER} -y
+conda activate ${CONDA_ENV_PATH}
+
+# Python requirements
+## cd into your directory inside of proj-chemnlp
+cd /fsx/proj-chemnlp/$2
+
+## clone + submodules (ok if exists)
+[ ! -d 'chemnlp' ] && git clone --recurse-submodules --remote-submodules [email protected]:OpenBioML/chemnlp.git
+
+## install
+pip install chemnlp # our repo
+pip install -r chemnlp/gpt-neox/requirements/requirements.txt # base gpt-neox requirements

scripts/stability-cluster/miniconda_install.sh

@@ -0,0 +1,5 @@
+#! /bin/bash
+
+cd ~
+wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+bash Miniconda3-latest-Linux-x86_64.sh # Follow instructions, accept all conditions blindly