Conda

See also: HPC and HTC tutorial | Run MPI On Grid'5000 | GPUs on Grid5000 | GNU Parallel | Modules | Conda | Guix | Singularity | Deep Learning Frameworks

	Note
	This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

Introduction

Conda is an open source package management system and environment management system for installing multiple versions of software packages and their dependencies and switching easily between them. It works on Linux, OS X and Windows, and was created for Python programs but can package and distribute any software.

To get started with Conda, have a look at this Conda cheat sheet and this Getting Started with Conda guide.

Conda, Miniconda, Anaconda ?

• conda is the package manager.
• miniconda is a minimal python distribution for conda that includes base packages
• anaconda is another python distribution for conda that includes 160+ additionnal packages to miniconda

On Grid'5000, we installed conda using the miniconda installer, but you are free to create an anaconda environment, using the anaconda meta-package.

More information about Miniconda vs Anaconda is available on the Conda website.

Conda or Mamba?

mamba is a reimplementation of the conda package manager in C++. Conda has a reputation for taking time when dealing with complex sets of dependencies. Mamba is much more efficient and is fully compatible with Conda packages and supports most of Conda's commands. It consists of:

• mamba: a Python-based CLI conceived as a drop-in replacement for conda, offering higher speed and more reliable environment solutions
• micromamba: a pure C++-based CLI, self-contained in a single-file executable
• libmamba: a C++ library exposing low-level and high-level APIs on top of which both mamba and micromamba are built

Mamba is available as a module on Grid'5000:

frontal:

module load mamba

Then, since its syntax is generally compatible with Conda, you can use the mamba command where you would use the conda command.

Conda on Grid'5000

Conda is already available in Grid'5000 as a module. You do not need to install Anaconda or Miniconda on Grid'5000 !

Load Conda module

• To make it available on a node or on a frontend, load the Conda module as follows (default version):

frontal:

module load conda

Optional: Conda initialization and activation in your shell

Conda initialization is the process of defining some shell functions that facilitate activating and deactivating Conda environments, as well as some optional features such as updating PS1 to show the active environment. It is not required to use Conda.

The conda shell function is mainly a forwarder function. It will delegate most of the commands to the real conda executable driven by the Python library.

There are two ways to initialize conda in standard installation:

• 1. occasionally : activate conda in your current shell (ex: bash)

$:

eval "$(conda shell.bash hook)"

• 2. always : activate conda in your login shell environment permanently (this command modifies your .bashrc by adding conda setup directives)

$:

conda init

In Grid'5000, the conda initialization is made transparently by loading the conda module.

The conda activate or conda deactivate commands relies on the conda shell initialization to load/unload the corresponding conda environment variables to the current shell session.

By default, you are located in the base Conda environment that corresponds to the base installation of Conda.

If you’d prefer that conda’s base environment not be activated on startup, set the auto_activate_base parameter to false:

$:

conda config --set auto_activate_base false

Verify your conda configuration with this command:

$:

conda config --show

Look at all available configuration options with:

$:

conda config --describe

Conda environments

Conda allows you to create separate environments containing files, packages, and their dependencies that will not interact with other environments.

When you begin using conda, you already have a default environment named base. You can create separate environments to keep your programs isolated from each other. Specifying the environment name confines conda commands to that environment.

	Warning
	The base environment is stored in a read-only directory as shown by conda info command That's why you need to systematically create your own conda environments to install the software you need.

• List all your environments

$:

conda info --envs

or

$:

conda env list

• Create a new environment

$:

conda create --name ENVNAME

• Activate this environment before installing package

$:

conda activate ENVNAME

For further information:

• https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
• Managing your data science project environments with Conda

Conda package installation

In its default configuration (the default Conda channel), Conda can install and manage the over 7,500 packages at https://repo.anaconda.com/pkgs/ that are built, reviewed, and maintained by Anaconda.

$:

conda install <package>

• Install specific version of package:

$:

conda install <package>=<version>

• Uninstall a package:

$:

conda uninstall <package>

For further information:

• https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/packages.html

Conda package installation from channels

Channels are the locations of the repositories where Conda looks for packages. Channels may point to a Cloud repository or a private location on a remote or local repository that you or your organization created. Useful channels are:

• conda-forge from https://conda-forge.org. It is free for all to use.
• nvidia from https://anaconda.org/nvidia. It provides Nvidia's software.

To install a package from a specific channel:

$:

conda install -c <channel_name> <package>

• List all packages installed with their source channels

$:

conda list --show-channel-urls

For further information:

• https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/channels.html

	Warning
	Installing Conda packages can be time and resources consuming. Preferably use a node (instead of a frontend) to perform such an operation. Note, using a node is mandatory if you need to access specific hardware resources like GPU.

Application examples

Create an environment

For example create environment <env_name> (specify a Python version; otherwise, it is the module default version)

fgrenoble:

conda create -y -n <env_name> python=x.y

Load this environment

fgrenoble:

conda activate <env_name>

Install a package into

fgrenoble:

conda install <package_name>

Exit from the loaded environment

fgrenoble:

conda deactivate

Remove unused Conda environments

	Warning
	Conda packages are installed in $HOME/.conda. You could, therefore, rapidly saturate your homedir quota (25GB by default). Do not forget to occasionally remove unused Conda environment to free up space.

• To delete an environment

fgrenoble:

conda deactivate conda env remove --name <env_name>

• To remove unused packages and the cache. Do not be concerned if this appears to try to delete the packages of the system environment (ie. non-local).

fgrenoble:

conda clean -a

Use a Conda environment in a job

As seen in the previous section, the Conda environment is stored by default in user's homedir (at ~/.conda). Once the environment is created and packages installed, it is usable on all nodes from the given site.

For interactive jobs

Load, init, and active you conda environment env_name in an interactive job

frontal:

oarsub -I

node:

module load conda conda activate env_name

For batch jobs

Load, initialize, and active you conda environment env_name in a batch job

First prepare your conda environment on the frontend:

• module load and conda initialization
• conda creation of an environment testconda containing gcc from conda-forge channel
• list installed packages with source info

fsiteA:

module load conda conda create --name testconda conda activate testconda conda install -c conda-forge gcc_linux-64 gxx_linux-64

• launch this commands and keep output result

fsiteA:

conda info conda list -n testconda --show-channel-urls

In this example, we launch a job that does the same tasks but in batch job.

• The important step is to source shell environment to execute module and activate conda

fsiteA:

oarsub 'bash -l -c "module load conda ; conda activate testconda ; conda info ; conda list -n testconda --show-channel-url"'

OAR_JOB_ID=1539228

• Is job finished ?

fsiteA:

oarsub -C 1539228

# Error: job 1539228 is not running. Its current state is Finishing.

• Compare output with the previous one : they should be identical

fsiteA:

cat OAR.1539228.std

Advanced Conda environment operations

Synchronize Conda environments between Grid'5000 sites

• To synchronize a Conda directory from a siteA to a siteB:

fsiteA:

rsync --dry-run --delete -avz ~/.conda siteB.grid5000.fr:~

To really do things, the --dry-run argument has to be removed and siteB has to be replaced by a real site name.

Share Conda environments between multiple users

You can use two different approaches to share Conda environments with other users.

Export an environment as a yaml file

• Export it as follows:

fgrenoble:

conda env export > environment.yml

• Share it by putting the yaml file in your public folder

fgrenoble:

cp environment.yml ~/public/

• Other users can create the environment from the environment.yml file

fgrenoble:

conda env create -f ~/<login>/public/environment.yml

• Advantage : it prevents other users from damaging the environment if they add packages that could conflict with other packages and/or even delete packages that another user might need.
• Inconvenient : it's not a true shared environment. The environment is duplicated on other users' home directory. Any modification on one Conda environment will not be automatically replicated on others.

Use a group storage

Group Storage gives you the possibility to share a storage between multiple users. You can take advantage of a group storage to share a single Conda environment among multiple users.

• Create a shared Conda environment with --prefix to specify the path to use to store the conda environment

flyon:

conda create --prefix /srv/storage/storage_name@server_hostname_(fqdn)/ENVNAME

• Activate the shared environment (share this command with the targeted users)

flyon:

conda activate /srv/storage/storage_name@server_hostname_(fqdn)/ENVNAME

• Advantage : It avoids storing duplicate packages and makes any modification accessible to all users
• Inconvenients :
  • Users could potentially harm the environment by installing or removing packages.
  • When installing additional packages, conda still stores them in the package cache located in your home directory. Use conda clean as described above to clean those files.

• Create your environments by defaut in a group storage location

You can modify you ~/.condarc file to specify this location for conda environment and package installation as follow (change the location to suit your group and your convenience). Add this lines:

pkgs_dirs:
  - /srv/storage/storage_name@server_hostname_(fqdn)/conda_shared_envs/pkgs/
envs_dirs:
  - /srv/storage/storage_name@server_hostname_(fqdn)/conda_shared_envs/envs/

Build your HPC-IA framework with conda

Here are some pointers to help you set up your software environment for HPC or AI with conda

• HPC_and_HTC_tutorial
• Running MPI applications on Grid'5000
• Deep Learning Frameworks documentation