Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conda Best Practice - migration #803

Draft
wants to merge 42 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
bf4ee86
remove anaconda channel from build env
wwarriner Aug 14, 2024
2a1a129
disable warnings for no git revision date
wwarriner Aug 14, 2024
df16e06
change to using_conda.md
wwarriner Aug 14, 2024
47cdc8e
add conda migration faq draft
wwarriner Aug 14, 2024
44dcb84
remove anaconda from using_conda.md
wwarriner Aug 16, 2024
16b5f95
removed anaconda from getting_started page
wwarriner Aug 16, 2024
9968eac
remove anaconda from ood_jupyter
wwarriner Aug 16, 2024
9ca4f8e
remove anaconda from ood_matlab
wwarriner Aug 16, 2024
0e22837
update software/software
wwarriner Aug 16, 2024
4ee8da2
move section to software/software
wwarriner Aug 16, 2024
d90bd87
remove anaconda from ood_r
wwarriner Aug 16, 2024
83eeb51
fix typo
wwarriner Aug 16, 2024
796e5c2
remove anaconda from gpus
wwarriner Aug 16, 2024
6814a9e
remove anaconda from slurm_tutorial
wwarriner Aug 16, 2024
3fbcfd8
remove anaconda from submitting_jobs
wwarriner Aug 16, 2024
bdeec92
remove anaconda from common_software.csv
wwarriner Aug 16, 2024
c27bdd7
remove anaconda from cheaha/tutorial/index
wwarriner Aug 16, 2024
10b26df
update cheaha/tutorial/index
wwarriner Aug 16, 2024
accb89b
update cheaha/tutorial/index
wwarriner Aug 16, 2024
49c42ba
remove anaconda from pytorch_tensorflow tutorial
wwarriner Aug 16, 2024
fcea560
remove anaconda from contributor guide
wwarriner Aug 16, 2024
20acdc5
remove anaconda from storage.md
wwarriner Aug 16, 2024
e1c2a23
remove anaconda from data_management/lts/interfaces.md
wwarriner Aug 16, 2024
463386f
update uab-rc-facilities.txt
wwarriner Aug 16, 2024
f2c86eb
remove anaconda from support.md
wwarriner Aug 16, 2024
df651c4
remove anaconda from uab_cloud/installing_software.md
wwarriner Aug 16, 2024
2aeaa7e
remove anaconda from getting_containers.md
wwarriner Aug 16, 2024
9a3893b
remove anaconda from r_environments.md
wwarriner Aug 16, 2024
3144643
remove anaconda from shell.md
wwarriner Aug 16, 2024
ef60530
removed as much anaconda as I could from using_conda.md
wwarriner Aug 16, 2024
f184f9a
fixed unicode double quote character
wwarriner Aug 16, 2024
0aa4539
remove miniconda from uab_cloud/installing_software
wwarriner Aug 16, 2024
0aa7c9a
Made several changes, replaced images, edited the docker container tu…
iam4tune Aug 29, 2024
5590f54
Merge branch 'main' into feat-anaconda-to-miniforge
iam4tune Aug 29, 2024
b4ff9c9
Moved the section to a new page, moved images, and redirected links, …
iam4tune Aug 29, 2024
1c09f9a
made changes as a result of review
iam4tune Aug 29, 2024
4c44b0f
Corrected a link in the index page
iam4tune Sep 6, 2024
e60edd1
Merge branch 'main' into feat-anaconda-best-practice
iam4tune Sep 9, 2024
b22e14c
Made changes to mkdocs.yml
iam4tune Sep 11, 2024
951b496
Merge branch 'main' into feat-anaconda-best-practice
iam4tune Sep 11, 2024
03493ef
fixed error from review
iam4tune Sep 11, 2024
4279f72
Edited the conda links found in 'data_management/lts/tutorial/individ…
iam4tune Sep 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/cheaha/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,13 +164,13 @@ Slurm is our job queueing software used for submitting any number of job scripts

A large variety of software is available on Cheaha as modules. To view and use these modules see [the following documentation](./software/modules.md).

For new software installation, please try searching [Anaconda](../workflow_solutions/using_anaconda.md) for packages first. If you still need help, please [send a support ticket](../help/support.md)
For new software installation, please try searching for [`conda` packages](../cheaha/tutorial/conda_good_practice.md#step-by-step-guide-to-finding-conda-software-packages) first. If you still need help, please [send a support ticket](../help/support.md)

### Conda Packages

A significant amount of open-source software is distributed as Anaconda or Python libraries. These libraries can be installed by the user without permission from Research Computing using Anaconda environments. To read more about using Anaconda virtual environments see our [Anaconda page](./software/software.md#anaconda-on-cheaha).
A significant amount of open-source research software is distributed as `conda` packages or Python libraries. These libraries can be installed by the user using `conda` environments. To read more about using `conda` environments see our [`conda` page](./software/software.md#conda-on-cheaha).

If the software installation instructions tell you to use either `conda install` or `pip install` commands, the software and its dependencies can be installed using a virtual environment.
If the software installation instructions tell you to use either `conda install` or `pip install` commands, the software and its dependencies can be installed using a `conda` environment.

## How to Get Help

Expand Down
47 changes: 24 additions & 23 deletions docs/cheaha/open_ondemand/ood_jupyter.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Jupyter Apps

Jupyter Notebooks and Jupyter Lab are both available as standalone apps in OOD. Jupyter is commonly used with Anaconda environments. If you are unfamiliar with Anaconda environments please see the [Working with Anaconda Environments section](#working-with-anaconda-environments) below before continuing here.
Jupyter Notebooks and Jupyter Lab are both available as standalone apps in OOD. Jupyter is commonly used with `conda` environments. If you are unfamiliar with `conda` environments please see the [Working with `conda` Environments section](#working-with-conda-environments) below before continuing here.

To launch the Jupyter notebook, select the menus 'Interactive Apps -> Jupyter Notebook'. The job creation and submission form appears:

Expand All @@ -10,7 +10,7 @@ As with all interactive apps, you'll need to select the resources required using

## Environment Setup

To modify the environment that Anaconda and Jupyter will run in, please use the Environment Setup field to load modules and modify the environment `$PATH`. Be aware that any changes to the environment made in this window will be inherited by terminals as well as notebooks opened within Jupyter.
To modify the environment that `conda` and Jupyter will run in, please use the Environment Setup field to load modules and modify the environment `$PATH`. Be aware that any changes to the environment made in this window will be inherited by terminals as well as notebooks opened within Jupyter.

### CUDA

Expand All @@ -34,17 +34,17 @@ The `Extra Jupyter Arguments` field allows you to pass additional arguments to t

## Working with other programming languages within Jupyter Notebook

To work with other programming languages within Jupyter Notebook, you need to install the corresponding kernel for each language, similar to the process used for Python with the `ipykernel`. This can be done using package managers such as `pip` or `conda`, or by following language-specific instructions. For example, to install `R kernel` for the R language, we can run the `conda install -c r r-essentials` command. Please ensure that the kernel is installed in your Anaconda environment. Then, select the desired language environment from the kernel dropdown menu.
To work with other programming languages within Jupyter Notebook, you need to install the corresponding kernel for each language, similar to the process used for Python with the `ipykernel`. This can be done using package managers such as `pip` or `conda`, or by following language-specific instructions. For example, to install `R kernel` for the R language, we can run the `conda install -c r r-essentials` command. Please ensure that the kernel is installed in your `conda` environment. Then, select the desired language environment from the kernel dropdown menu.

Once the necessary kernels are installed, if you wish, you can write and run multiple code cells in different languages within a single notebook. Easily switch between kernels and select the preferred one for each language, and then proceed to run the code cells in their respective languages.

## Working with Anaconda Environments
## Working with `conda` Environments

By default, Jupyter notebooks will use the base environment that comes with the Anaconda3 module. This environment contains a large number of popular packages and may useful for something quick, dirty, and simple. However, for any analysis needing specific package versions or special packages, you will need to create your own environment and select it from the `Kernel` menu. For information on creating and managing Anaconda environments please see our [Using Anaconda page](../../workflow_solutions/using_anaconda.md). Then please review our [Cheaha-specific Anaconda page](../software/software.md#anaconda-on-cheaha) for important tips and how to avoid common pitfalls.
By default, Jupyter notebooks will use the base environment that comes with the `Miniforge3` module. This environment contains a large number of popular packages and may useful for something quick, dirty, and simple. However, for any analysis needing specific package versions or special packages, you will need to create your own environment and select it from the `Kernel` menu. For information on creating and managing `conda` environments please see our [Using `conda` page](../../workflow_solutions/using_conda.md). Then please review our [Cheaha-specific `conda` page](../software/software.md#conda-on-cheaha) for important tips and how to avoid common pitfalls.

To change the kernel, use the `Kernel` dropdown and select `Change Kernel`. From the list, choose the kernel corresponding to your desired Anaconda environment (see below for an example). If your environment isn't appearing, you may be missing the ipykernel package. To do so, use `conda install ipykernel` to get ipykernel packgae installed into your environment, so Jupyter can recognize your environment.
To change the kernel, use the `Kernel` dropdown and select `Change Kernel`. From the list, choose the kernel corresponding to your desired `conda` environment (see below for an example). If your environment isn't appearing, you may be missing the ipykernel package. To do so, use `conda install ipykernel` to get ipykernel packgae installed into your environment, so Jupyter can recognize your environment.

![! Select your Anaconda environment from the Kernel dropdown menu in Jupyter](images/jupyter_kernel.png)
![! Select your `conda` environment from the Kernel dropdown menu in Jupyter](images/jupyter_kernel.png)

### Creating an Environment for use with Jupyter Notebook

Expand All @@ -55,9 +55,9 @@ We can create a new environment, that houses all of the packages, modules, and l
- [OOD Terminal](./ood_layout.md#opening-a-terminal). Be sure to run the following steps in a job!
- [OOD HPC Desktop Job Terminal](./hpc_desktop.md). This method will ensure terminal commands are run in a job.

1. [Create](../../workflow_solutions/using_anaconda.md#create-an-environment) and [activate](../../workflow_solutions/using_anaconda.md#activate-an-environment) your new environment, following the linked steps.
1. [Create](../../workflow_solutions/using_conda.md#create-an-environment) and [activate](../../workflow_solutions/using_conda.md#activate-an-environment) your new environment, following the linked steps.

1. [Install your desired packages into your activated environment](../../workflow_solutions/using_anaconda.md#install-packages).
1. [Install your desired packages into your activated environment](../../workflow_solutions/using_conda.md#install-packages).

1. Remember to install 'ipykernel' in your activated environment, using `conda install ipykernel`.

Expand All @@ -81,9 +81,15 @@ We can create a new environment, that houses all of the packages, modules, and l

### Python Executable Issues

Jupyter Notebook by default loads `Anaconda3`. Hence do not load any versions of `Anaconda3` module in the `Environment Setup` field in the OOD Jupyter Notebook, as it causes Python mismatch, and the errors are hard to diagnose.
Jupyter Notebook by default loads `Miniforge3`. Hence do not load any versions of `Miniforge3` module in the `Environment Setup` field in the OOD Jupyter Notebook, as it causes Python mismatch, and the errors are hard to diagnose.

Having custom installs of Anaconda/Miniconda/ can cause the above similar issue. If you have installations of any of these software in your personal space, delete those directories and instead use the `Anaconda3` module.
Having self-installed `conda` software can cause the above issue. This includes self-installed Anaconda, Miniconda, Mambaforge, or Miniforge. If you have installations of any of these software in your personal space, delete those directories and instead use the `Miniforge3` module.

<!-- markdownlint-disable MD046 -->
!!! important

The Anaconda and Miniconda software are subject to the Anaconda Terms of Service and may not be used for UAB business.
<!-- markdownlint-enable MD046 -->

To identify a Python mismatch, use the commands `which python` and `python --version` to confirm the desired Python executable and version. Within the `conda` environment, `which python` prints the path of the Python executable (e.g. `~/.conda/envs/remora/bin/python`). If it doesn't match the expected version, an unexpected Python version may be in use.

Expand All @@ -105,22 +111,17 @@ While launching an OOD HPC Desktop Job or any OOD Applications, if the user gets

Using `conda init` causes a block of code automatically inserted into the `.bashrc` file in your `$HOME` directory. This code block may interfere with the proper functioning of various OOD applications, resulting in a VNC error. To address this issue, it is recommended to follow the instructions outlined in the [FAQ entry](https://ask.cyberinfrastructure.org/t/why-do-i-get-an-error-when-launching-an-open-ondemand-hpc-interactive-session/2496).

### Pip Installs Packages Outside of Environment
### Installing Pip Packages Outside of Your Environments

When installing packages within a `conda` environment using `pip`, it's crucial to ensure that you install `pip` within the same conda environment and use `pip` from that environment. If `pip` is used outside of Anaconda or within an environment without `pip` installed, the packages are installed to `~/.local`. This can lead to unexpected package conflicts, as Python loads packages from `~/.local` before loading from Anaconda environments, and shows the following error,

```bash
Requirement already satisfied: numpy in /home/$USER/.local/lib/python3.11/site-packages (1.26.3)
```
<!-- markdownlint-disable MD046 -->
!!! danger

For the above case, resolving errors involve deleting the `~/.local` directory.
Using `pip install` without loading Miniforge3 will cause hard-to-diagnose errors and broken workflows.

Here's an example of the correct procedure for installing `pip` packages within a `conda`:
Using `pip install` in the `base` environment will cause the same hard-to-diagnose errors and broken workflows.

1. Load the `Anaconda3` module using `module load Anaconda3`.
1. Create or activate the desired Anaconda environment. Please refer to the [Anaconda documentation](../../workflow_solutions/using_anaconda.md#create-an-environment)
1. Install `pip` within the `conda` environment using `conda install pip` or `conda install python`. `pip` and `python` are packaged together, installing one will always install the other.
1. Use `pip` when this `conda` environment is active to install packages. Please refer to [Installing packages with `pip`](../../workflow_solutions/using_anaconda.md#installing-packages-with-pip)
Read more about this issue, and how to resolve it, [here](../software/software.md#installing-pip-packages-outside-of-your-environments).
<!-- markdownlint-disable MD046 -->

### Tensorflow and PyTorch GPU issues

Expand Down
10 changes: 5 additions & 5 deletions docs/cheaha/open_ondemand/ood_matlab.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,17 @@ Matlab is available for use graphically in your browser via OOD. As with other s
Matlab tends to consume substantial memory at startup. You may experience difficulty with job errors below 20 GB of total memory.
<!-- markdownlint-enable MD046 -->

## Using Anaconda Python from within Matlab
## Using `conda` Python from within Matlab

Matlab has the ability to interoperate with Python from within Matlab. The official documentation for this featuer may be found at <https://www.mathworks.com/help/matlab/call-python-libraries.html>.

This section is dedicated to using this feature with Anaconda on Cheaha. To use Python contained in an Anaconda Environment within Matlab, please use the following steps.
This section is dedicated to using this feature with `conda` on Cheaha. To use Python contained in a `conda` Environment within Matlab, please use the following steps.

1. Create an [HPC Interactive Desktop Job](hpc_desktop.md).
1. Open a terminal in that job. The following steps should all be run in this terminal unless otherwise specified.
1. [Load the Anaconda Module](../software/software.md#loading-anaconda).
1. [Create an Environment](../../workflow_solutions/using_anaconda.md#create-an-environment) in Anaconda with the packages needed.
1. [Activate the Environment](../../workflow_solutions/using_anaconda.md#activate-an-environment),
1. [Load the `conda` Module](../software/software.md#loading-conda).
1. [Create an Environment](../../workflow_solutions/using_conda.md#create-an-environment) in `conda` with the packages needed.
1. [Activate the Environment](../../workflow_solutions/using_conda.md#activate-an-environment),
1. Load the Matlab [Module](../software/modules.md).
1. Start Matlab by entering the command `matlab`.
1. Verify success by entering `pyenv` at the Matlab prompt (not the terminal window). Multiple lines of text will be returned at the prompt. Among them you should see a line like the following, with your environment name in place of `<env_name>`.
Expand Down
20 changes: 10 additions & 10 deletions docs/cheaha/open_ondemand/ood_rstudio.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,16 @@ RStudio is available for use graphically in your browser via OOD. As with other

## RStudio and Python

If you have a workflow that uses both R and Python, it is strongly recommended to use the [reticulate](https://rstudio.github.io/reticulate/) package along with Anaconda environments. Reticulate allows researchers to load Python packages into a native R session as objects. For instance, if someone prefer some functionality of the `pandas` package but has other code already written in R, they can import `pandas` to R and use both simultaneously.
If you have a workflow that uses both R and Python, it is strongly recommended to use the [reticulate](https://rstudio.github.io/reticulate/) package along with `conda` environments. Reticulate allows researchers to load Python packages into a native R session as objects. For instance, if someone prefer some functionality of the `pandas` package but has other code already written in R, they can import `pandas` to R and use both simultaneously.

This also allows researchers to download precompiled command line binaries into an Anaconda environment and easliy use them in their R scripts.
This also allows researchers to download precompiled command line binaries into a `conda` environment and easily use them in their R scripts.

For setup, use the following steps:

1. In a terminal on a compute node, either in an HPC Desktop job or by clicking the blue Host button on any job card:

1. Load the `Anaconda3` module
1. Create an Anaconda environment. More information about how to create Anaconda environments can be found [in our documentation](../../workflow_solutions/using_anaconda.md).
1. Load the `Miniforge3` module
1. Create a `conda` environment. More information about how to create `conda` environments can be found [in our documentation](../../workflow_solutions/using_conda.md).
1. Activate your environment and install your requuired python packages using either `pip install` or `conda install` depending on the package source.

<!-- markdownlint-disable MD046 -->
Expand All @@ -32,16 +32,16 @@ For setup, use the following steps:

1. In RStudio:

1. Add the command `module load Anaconda3` to the Environment Setup window when requesting the RStudio job.
1. Add the command `module load Miniforge3` to the Environment Setup window when requesting the RStudio job.
1. If not already installed, install the `reticulate` package using either `install.packages` or the [renv](#rstudio-projects-and-renv) package.
1. Use `reticulate::use_condaenv('env_name')` to load your conda environment.
1. From here, you will be able to interact with all of the python packages and non-python precompiled binaries in your Anaconda environment using R and RStudio. Please read more about how to do that in [reticulate's documentation](https://rstudio.github.io/reticulate/#importing-python-modules).
1. Use `reticulate::use_condaenv('ENVIRONMENT')` to load your conda environment which has the name `ENVIRONMENT`.
1. From here, you will be able to interact with all of the python packages and non-python precompiled binaries in your `conda` environment using R and RStudio. Please read more about how to do that in [reticulate's documentation](https://rstudio.github.io/reticulate/#importing-python-modules).

For cases where your R code only needs access to precompiled binaries or libraries and does not need to import any Python libraries, you can instead create your Anaconda environment and add the following lines into the Environment Setup window:
For cases where your R code only needs access to precompiled binaries or libraries and does not need to import any Python libraries, you can instead create your `conda` environment and add the following lines into the Environment Setup window:

``` bash
module load Anaconda3
conda activate <env_name>
module load Miniforge3
conda activate ENVIRONMENT
```

This will add those binaries and libraries to your environment `$PATH` which RStudio will inherit.
Expand Down
Loading