	Note
	This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

Note

Note

The purpose of this document is:

to explain how to use conda
gives examples to install and configure software with conda to create environments for running HPC and AI jobs on Grid'5000

It was written by consolidating the following different information resources:

Grid'5000 documentation Environment modules, HPC and HTC tutorial, Deep Learning Frameworks
An in-depth tutorial contributed by a Grid'5000 user, Ismael Bada
User:Bjonglez/Debian11/Deep Learning Frameworks

Introduction

Conda is an open source package management system and environment management system for installing multiple versions of software packages and their dependencies and switching easily between them. It works on Linux, OS X and Windows, and was created for Python programs but can package and distribute any software.

Reference:

[conda website https://docs.conda.io/projects/conda/en/latest/index.html]
[Conda cheat sheet https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html]

Installation

The conda package and environment manager is included in all versions of Anaconda®, Miniconda, and Anaconda Repository. Conda is also included in Anaconda Enterprise, which provides on-site enterprise package and environment management for Python, R, Node.js, Java, and other application stacks. Conda is also available on conda-forge, a community channel.

Anaconda or Miniconda?

See: https://docs.conda.io/projects/conda/en/latest/user-guide/install/download.html#anaconda-or-miniconda

Conda avaibility in Grid'5000

Conda is already available in Grid'5000 as a [module https://www.grid5000.fr/w/Software_using_modules]. Just execute `module load miniconda3` from a node or a frontend to make it available.

If you want to install conda on your desktop follow the [installation guide](https://conda.io/projects/conda/en/stable/user-guide/install/index.html#) in the conda Website.

You have 3 conda download options:

Download Anaconda---free.
Download Miniconda---free.
Purchase Anaconda Enterprise.

Miniconda installation on your desktop

Commands to install miniconda somewhere in your home directory

``` wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh eval "$(conda shell.bash hook)" # two load conda shell environment ```

NB: your ` .bashrc` is modified by the installation program

Conda usage

Getting started

The command `conda init bash` load the conda added bash environment

The command `conda info` display information about current conda install

Usage ``` $ conda --help usage: conda [-h] [-V] command ...

conda is a tool for managing and deploying applications, environments and packages.

Options:

positional arguments:

 command
   clean             Remove unused packages and caches.
   compare           Compare packages between conda environments.
   config            Modify configuration values in .condarc. This is modeled
                     after the git config command. Writes to the user
                     .condarc file (/user/lmirtain/home/.condarc) by default.
                     Use the --show-sources flag to display all identified
                     configuration locations on your computer.
   create            Create a new conda environment from a list of specified
                     packages.
   info              Display information about current conda install.
   init              Initialize conda for shell interaction.
   install           Installs a list of packages into a specified conda
                     environment.
   list              List installed packages in a conda environment.
   package           Low-level conda package utility. (EXPERIMENTAL)
   remove (uninstall)
                     Remove a list of packages from a specified conda
                     environment.
   rename            Renames an existing environment.
   run               Run an executable in a conda environment.
   search            Search for packages and display associated
                     information.The input is a MatchSpec, a query language
                     for conda packages. See examples below.
   update (upgrade)  Updates conda packages to the latest compatible version.
   notices           Retrieves latest channel notifications.

options:

 -h, --help          Show this help message and exit.
 -V, --version       Show the conda version number and exit.

```

See:

[Conda cheat sheet https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html] for curent commands
[Getting Started with Conda https://towardsdatascience.com/managing-project-specific-environments-with-conda-b8b50aa8be0e]

Conda channels and packages

Channels are the locations of the repositories where conda looks for packages. Channels may point to a Cloud repository or a private location on a remote or local repository that you or your organization created. In its default configuration, conda can install and manage the over 7,500 packages at https://repo.anaconda.com/pkgs/ that are built, reviewed, and maintained by Anaconda. This is the default conda channel which may require a paid license, as described in the repository terms of service a commercial license.

Other usefull channels:

The conda-forge channel https://conda-forge.org/ is free for all to use.
The nvidia channel https://anaconda.org/nvidia provides Nvidia's software
The IBM PowerAI https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/#/ Watson Machine Learning Community Edition

list all packages + source channels

inside:

conda list --show-channel-urls

Install a package from specific channel

inside:

conda install -c CHANNELNAME PKG1 PKG2

Visit for details:

[1]
[2]

Conda environments

Conda allows you to create separate environments containing files, packages, and their dependencies that will not interact with other environments.

When you begin using conda, you already have a default environment named "base". You can create separate environments to keep your programs isolated from each other. Specifying the environment name confines conda commands to that environment.

List all your environments

inside:

conda info --envs

or

inside:

conda env list

Typical sequence

create a new environment

inside:

conda create --name ENVNAME

activate this environment before installing package

{Term|location=inside|cmd=conda activate ENVNAME}

Recommandation: Packages are installed in $HOME/.conda You could, therefore, rapidly saturate the disk quota of your $HOME

See more:

[3]
[Managing your data science project environments with Conda https://towardsdatascience.com/managing-project-specific-environments-with-conda-406365a539ab]

Use conda on Grid'5000

Remind, you don't need to install Anaconda or Miniconda to create your own environment on Grid'5000! You just load conda by using the `module load miniconda3` command.

Log in a grid'5000 frontal

inside:

ssh login@access.grid5000.fr

go to your favorite grid5000 site

inside:

ssh grenoble

Load conda and source it

inside:

module load miniconda3

inside:

eval "$(conda shell.bash hook)"

Current commands are:

    conda init <SHELL_NAME> to initialize your shell 
    conda deactivate to exit from the environment loaded with module load
    conda create -y -n <name> python=x.y to create an environment (specify a Python version; otherwise, it is the module default version)
    conda activate <name>
    conda env remove --name <name> to correctly delete the environment
    conda clean -a to remove unused packages and the cache. Do not be concerned if this appears to try to delete the packages of the system environment (ie. non-local).

Tip: Creating an environment can be time and resource consuming. Preferably reserve and connect to a node via oarsub command (mandatory if you need to access to specific hardware ressource like GPU).

Create a specific environment for PowerPC arch

Because the version of pytorch in PowerAI is too old for py38 or py39, we must use python 3.7,

inside:

conda create create --name pytorch-ppc64-py37 python=3.7

Use this specific environment

inside:

conda activate pytorch-ppc64-py37

Install pytorch in this environment

inside:

conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

inside:

conda install pytorch -c

Use NIVDIA tools

NVIDIA libraries are available via Conda and you can manage project specific versions of the NVIDIA CUDA Toolkit, NCCL, and cuDNN using Conda.

NVIDIA actually maintains their own Conda channel and the versions of CUDA Toolkit available from the default channels are the same as those you will find on the NVIDIA channel.

To compare build numbers version from default and nvidia channel

inside:

conda search --channel nvidia cudatoolkit

Compare also limitation of versions from the diffent channels: ie. NVIDIA

See:

[4]
[Managing CUDA dependencies with Conda https://towardsdatascience.com/managing-cuda-dependencies-with-conda-89c5d817e7e1]

conda create --name NvidiaTools conda activate NvidiaTools conda install cudatoolkit -c nvidia

Use pytorch

Installation pytorch from nvidia channel {{Term|location=inside|cmd=conda install pytorch -c nvidia`

Hello world pytorch: - See: https://towardsai.net/p/l/how-to-set-up-and-run-cuda-operations-in-pytorch

Tensorflow

`conda install tensorflow-gpu -c conda-forge`
ne pas oublier de faire activate après l'installation
```gemini-1@lyon:~/dload nniclausse# python
Python 3.7.11 (default, Jul 27 2021, 14:32:16)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
> import tensorflow as tf
> hello = tf.constant("hello TensorFlow!")
> sess=tf.Session()
```

CUDA

pour afficher la version installée:
`nvcc --version`
utilisation des channels conda-forge ou nvidia pour installer cuda

GCC

installer les dernières version de gcc via conda-forge (acutellement gcc 12)
`conda install -c conda-forge gcc_linux-64 gxx_linux-64`

OpenMPI

ucx = pour utiliser le réseau rapide (infiniband)
```

  module load miniconda3/4.10.3_gcc-10.2.0 
  conda activate mamba
  conda activate --stack openmpi
  mamba install -c conda-forge gcc_linux-64 openmpi ucx

```
Installation de NetPIPE pour tester la latence et le débit réseau
```

  wget https://src.fedoraproject.org/lookaside/pkgs/NetPIPE/NetPIPE-3.7.1.tar.gz/5f720541387be065afdefc81d438b712/NetPIPE-3.7.1.tar.gz
  tar zvxf NetPIPE-3.7.1.tar.gz 
  cd NetPIPE-3.7.1/
  make mpi

```
réserver 2 procs sur 2 noeuds différents:
`oarsub -I -l /nodes=2/core=1`
version sans réseau rapide:

   `mpirun -np 2 --machinefile $OAR_NODEFILE --prefix $CONDA_PREFIX --mca plm_rsh_agent oarsh  NPmpi `

version avec ucx:

   `mpirun -np 2 --machinefile $OAR_NODEFILE --prefix $CONDA_PREFIX --mca plm_rsh_agent oarsh --mca pml ucx  --mca osc ucx`

Keras

Keras is a high-level neural networks API, written in python, which is used as a wrapper of theano, tensorflow or CNTK.
Keras allows to create deep learning experiments much more easily than using directly theano or tensorflow,
it's the recommended tool for beginners and even advanced users who don't want to deal and spend too much time with the complexity of low-level libraries as theano and tensorflow. 
`conda install keras`

Conda

Contents

Introduction

Installation

Anaconda or Miniconda?

Conda avaibility in Grid'5000

Miniconda installation on your desktop

Conda usage

Getting started

Conda channels and packages

Conda environments

Use conda on Grid'5000

Create a specific environment for PowerPC arch

Use NIVDIA tools

Use pytorch

CUDA

GCC

OpenMPI

Keras

Navigation menu