Production:Getting Started: Difference between revisions

From Grid5000
Jump to navigation Jump to search
No edit summary
(Replaced content with "NB: does not concern users from center clusters migrated to Abaca, their accounts will be initialized by the technical team (they will receive an e-mail notification).")
Tag: Replaced
Line 1: Line 1:
{{Portal|User}}
NB: does not concern users from center clusters migrated to Abaca, their accounts will be initialized by the technical team (they will receive an e-mail notification).
{{TutorialHeader}}
 
== Getting support ==
The '''[[Support]] page''' describes how to get help during your Production (Abaca) usage (common with Grid'5000 usage)
 
There's also an '''[[Production:FAQ]] page''' with the most common question related to Production usage.
 
== Connecting for the first time ==
 
The primary way to move around Abaca and Grid'5000 is using SSH. A [[SSH|reference page for SSH]] is also maintained with advanced configuration options that frequent users will find useful.
 
As described in the figure below, when using Abaca or Grid'5000, you will typically:
# connect, using SSH, to an access machine
# connect from this access machine to a site frontend
# on this site frontend, reserve resources (nodes), and connect to those nodes
 
[[Image:Grid5000_SSH_access.png|950px|center|Grid5000 Access]]
 
=== Connect to a Abaca access machine ===
To enter the Abaca network (common with Grid'50000) from Internet, one must use an access machine: <code class="host">access.grid5000.fr</code> (Note that <code class="host">access.grid5000.fr</code> is a round robin alias to either: <code class="host">access-north</code> which is currently hosted in Lille, or <code class="host">access-south</code> currently hosted in Sophia Antipolis).
 
For all connections, you must use the <code class="replace">login</code> that was provided to you when you created your Abaca/Grid'5000 account.
 
{{Term|location=outside|cmd=<code class="command">ssh</code> <code class="replace">login</code><code class="command">@</code><code class="host">access.grid5000.fr</code>}}
You will get authenticated using the SSH public key you provided in the account creation form. Password authentication is disabled.
 
=== Connecting to a Grid'5000 site ===
Grid'5000 is structured in '''<code class="host">sites</code>''' (<code class="host">Grenoble</code>, <code class="host">Rennes</code>, <code class="host">Nancy</code>, ...). Each site hosts one or more clusters (homogeneous sets of machines, usually bought at the same time).
 
To connect to a particular site, do the following (blue and red arrow labeled SSH in the figure above).
 
{{Term|location=access|cmd=<code class="command">ssh</code> <code class="replace">site</code>}}
 
; Home directories
You have a '''different home directory on each Grid'5000 site''', so you will usually use [[Rsync]] or <code class="command">scp</code> to move data around.
On <code class="host">access</code> machines, you have direct access to each of those home directories, through NFS mounts (but using that feature to transfer very large volumes of data is inefficient). Typically, to copy a file to your home directory on the Nancy site, you can use:
{{Term|location=outside|cmd=<code class="command">scp</code> <code class="replace">myfile.c</code> <code class="replace">login</code>@<code class=host>access.grid5000.fr</code>:<code class=file>nancy/targetdirectory/mytargetfile.c</code>}}
 
'''Grid'5000 does NOT have a BACKUP service for users' home directories''': it is '''your responsibility''' to save important data in someplace outside Grid'5000 (or at least to copy data to several Grid'5000 sites in order to increase redundancy).
 
Quotas are applied on home directories -- by default, you get 25 GB per Grid'5000 site. If your usage of Grid'5000 requires more disk space, it is possible to request quota extensions in the account management interface, or to use other storage solutions (see [[Storage]]).
 
=== Alternative Connections  ===
 
==== SSH connection through a web interface ====
 
If you want an out-of-the-box solution which does not require you to setup SSH, you can connect through a web interface.
The interface is available at https://intranet.grid5000.fr/shell/SITE/. For example, to access nancy's site, use: https://intranet.grid5000.fr/shell/nancy/
To connect you will have to type in your credentials twice (first for the HTTP proxy, then for the SSH connection).
 
This solution is probably suitable to follow this tutorial, but is unlikely to be suitable for real Grid'5000 usage. So you should probably read the next sections about how to setup and use SSH at some point.
 
==== VPN ====
A VPN service is also available, allowing to connect directly to any Grid'5000 machines (bypassing the access machines). See [[VPN|the VPN page]] for more information.
 
==== HTTP reverse proxies ====
If you only require HTTP/HTTPS access to a node, a reverse HTTP proxy is also available, see the [[HTTP/HTTPs_access]] page.
 
=== Recommended tips and tricks for an efficient use of Grid'5000===
 
==== SSH configuration ====
 
* Configure [[SSH#Using_SSH_ProxyCommand_feature_to_ease_the_access_to_hosts_inside_Grid.275000|SSH aliases using the ProxyCommand option]]. Using this, you can avoid the two-hops connection (access machine, then frontend) but establish connections directly to frontends. This requires using OpenSSH, which is the SSH software available on all GNU/Linux systems, MacOS, and also recent versions of Microsoft Windows.
 
{{Term|location=outside|cmd=editor <code class=file>~/.ssh/config</code>}}
Host <code class=host>g5k</code>
  User <code class=replace>login</code>
  Hostname access.grid5000.fr
  ForwardAgent no
Host <code class=host>*.g5k</code>
  User <code class=replace>login</code>
  ProxyCommand ssh g5k -W "$(basename %h .g5k):%p"
  ForwardAgent no
'''Reminder:''' <code class=replace>login</code> is your Grid'5000 username
 
Once done, you can establish connections to any machine (first of all: frontends) inside Grid'5000 directly, by suffixing <code class=host>.g5k</code> to its hostname (instead of first having to connect to an access machine). E.g.:
{{Term|location=outside|cmd=<code class=command>ssh</code> <code class=replace>rennes</code>.<code class=host>g5k</code>}}
{{Term|location=outside|cmd=<code class=command>scp</code> <code class=file>a_file</code> <code class=replace>lille</code>.<code class=host>g5k</code>:}}
==== Bash prompt ====
It is possible to modify your bash prompt to display useful informations related to your current job, such as its jobid, the reserved nodes and the remaining time.
 
{{Term|location=fnancy|cmd=<code class="command">jdoe@fnancy:~$ oarsub -C 3241912</code>}}
{{Term|location=grisou-15|cmd=<pre class="command">
Connect to OAR job 3241912 via the node grisou-15.nancy.grid5000.fr
[OAR] OAR_JOB_ID=3241912
[OAR] Your nodes are:
      grisou-15.nancy.grid5000.fr*16
 
[jdoe@grisou-15 ~](3241912-->57mn)$ sleep 1m
[jdoe@grisou-15 ~](3241912-->55mn)$
</pre>}}
 
You will find [https://oar.imag.fr/wiki:use_cases_and_user_tips#oar_aware_shell_prompt_for_interactive_jobs here] all the information you need to setup such a prompt if you are interested.
 
==== Miscellaneous ====
 
* Use <code class="command">rsync</code> instead of <code class="command">scp</code> for better performance with multiple files.
* For a '''better bandwidth or latency''', you may also be able to connect directly via the '''local access machine of one of the Grid'5000 sites'''. Local accesses use <code class="host">access.</code><code class=replace>site</code><code class=host>.grid5000.fr</code> instead of <code class="host">access.grid5000.fr</code>. However, mind that '''per-site access restrictions are applied''': see [[External access]] for details about local access machines.
* Access your data from your laptop using [[SSH#Mounting_remote_filesystem_.28sshfs.29|SSHFS]]
* Edit files over SSH with your favorite text editor, with e.g.:
{{Term|location=outside|cmd=<code class="command">vim</code> <code class=file>scp://nancy.g5k/my_file.c</code>}}
 
== Platform usage ==
 
=== Submitting jobs with OAR  ===
 
; Interactive usage
 
To reserve a single host (one node) for one hour, in interactive mode, do:
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I</code>}}
 
As soon as the resource becomes available, you will be directly connected to the reserved resource with an interactive shell, as indicated by the shell prompt, and you can run commands on the node:
 
{{Term|location=grisou-1|cmd=<code class="command">lscpu</code>}}
 
; Reserving only part of a node
 
To reserve only one CPU core in interactive mode, run:
{{Term|location=fnancy|cmd=<code class="command">oarsub -l core=1</code> <code>-I</code>}}
 
{{Note|text= When reserving only a share of the node's cores, you will have a share of the memory with the same ratio as the cores. If you take the whole node, you will have all the memory of the node. If you take half the cores, you will have half the memory, and so on... You cannot reserve a memory size explicitly.}}
 
; Non-interactive usage (scripts)
 
You can also simply launch your experiment along with your reservation:
{{Term|location=fnancy|cmd=<code class="command">oarsub -l host=1/core=1</code> <code>"my_mono_threaded_script.py --in $HOME/data --out $HOME/results"</code>}}
 
Your program will be executed as soon as the requested resources are available. As this type of job is not interactive, you will have to check for its termination using the <code class="command">oarstat</code> command.
 
{{Template:OARscript}}
 
=== Submission queues ===
 
{{Note|text=OAR is the resources and jobs management system (a.k.a batch manager) used in Grid'5000, just like in traditional HPC centers. '''However, settings and rules of OAR that are configured in Grid'5000 slightly differ from traditional batch manager setups in HPC centers, in order to match the requirements for an experimentation testbed'''. Please remember to read again '''Grid'5000 [[Grid5000:UsagePolicy#Resources_reservation|Usage Policy]]''' to understand the expected usage.}}
 
It exists different submission queues:
* ''default'' used if none are specified
* ''exotic'' to use resources with specific hardware (e.g., PMEM, arm, power, ...)
* ''production'' only available at Nancy for computation usage
* ''besteffort'' to start best-effort jobs
 
 
==== Production queue in Nancy ====
 
* Job submission
 
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code>-I</code> <code>-q production</code>}}
 
* Resources restriction per walltime
 
To make sure that someone requesting only a few nodes, for a small amount of time will be able to get soon enough, the nodes are split into categories. This depends on each cluster and is visible in the Gantt chart or at the [[Nancy:Hardware#graffiti|Nancy Hardware page]]. An example of split is:
* 30% -- 24h (1 day)
* 30% -- 48h (2 days)
* 40% -- 168h (one week)
 
Note that ''best-effort'' jobs are excluded from those limitations.
 
Another OAR feature that could impact the scheduling of your jobs is the OAR scheduling with fair-sharing, which is based on the notion of ''karma'': this feature assigns a dynamic priority to submissions based on the history of submissions by a specific user. With that feature, the jobs from users that rarely submit jobs will be generally scheduled earlier than jobs from heavy users.
 
=== Discovering and visualizing resources ===
 
There are several ways to learn about the site's resources and their status.
 
; Site's resources
 
* The site's MOTD (message of the day) lists all clusters and their features. Additionally, it gives the list of current or future downtimes due to maintenance.
 
* [[Hardware|Hardware pages]] contain a detailed description of the site's hardware
* Site pages on the wiki (e.g. [[Nancy:Home]]) contain a detailed description of the site's hardware and network
 
 
; Site's status
 
* Current or future downtimes due to maintenance are available from [https://www.grid5000.fr/status/ https://www.grid5000.fr/status/].
* The [[Status]] page links to the resource status on each site, with two different visualizations available:
** Monika, that provides the current status of nodes (see [https://intranet.grid5000.fr/oar/Nancy/monika.cgi Nancy's current status])
** Gantt, that provides current and planned resources reservations (see [https://intranet.grid5000.fr/oar/Nancy/drawgantt-svg/ Nancy's current status]; example in the figure below).
 
[[Image:nancy-gantt.png|600px|center|Example of Drawgantt in Nancy site]]
 
=== Change default job specifications ===
 
In Grid'5000 the smallest unit of resource managed by OAR is the core (cpu core), but by default a OAR job reserves a host (physical computer including all its cpus and cores, and possibly gpus). Hence, what OAR calls ''nodes'' are hosts (physical machines). In the <code class="command">oarsub</code> resource request (<code class="command">-l</code> arguments), ''nodes'' is an alias for ''host'', so both are equivalent. But prefer using ''host'' for consistency with other argumnents and other tools that expose ''host'' not ''nodes''.
 
; Other types of resources
 
To reserve only one GPU (with the associated CPU cores and share of memory) in interactive mode, run:
{{Term|location=flille|cmd=<code class="command">oarsub -l gpu=1</code> <code>-I</code>}}
 
{{Note|text=Even if the node has several GPUs, this reservation will only be able to access a single one. It's a good practice if you only need one GPU: other users will be able to run jobs on the same node to access the other GPUs. Of course, if you need all GPUs of a node, you have the option to reserve the entire node which includes all its GPUs.}}
 
To reserve several GPUs and ensure they are located in a single node, make sure to specify <code class="command">host=1</code>:
 
{{Term|location=flille|cmd=<code class="command">oarsub -l host=1/gpu=2</code> <code>-I</code>}}
 
; Choosing the job duration
 
Of course, you might want to run a job for a different duration than one hour. The <code>-l</code> option allows you to pass a comma-separated list of parameters specifying the needed resources for the job, and <code>walltime</code> is a special resource defining the duration of your job:
 
{{Term|location=fnancy|cmd=<code class="command">oarsub -l host=1/core=2,walltime=0:30</code> <code>-I</code>}}
The walltime is the expected duration you envision to complete your work. Its format is <code>[hour:min:sec|hour:min|hour]</code>. For instance:
 
* <code>walltime=5</code> => 5 hours
* <code>walltime=1:22</code> => 1 hour and 22 minutes
* <code>walltime=0:03:30</code> => 3 minutes and 30 seconds
 
; Working with more than one node
 
You will probably want to use more than one node on a given site.
 
To reserve two hosts (two nodes), in interactive mode, do:
{{Term|location=fnancy|cmd=<code class="command">oarsub -l host=2</code> <code>-I</code>}}
or equivalently (''nodes'' is an alias for ''host''):
{{Term|location=fnancy|cmd=<code class="command">oarsub -l nodes=2</code> <code>-I</code>}}
 
You will obtain a shell '''on the first node of the reservation'''. It is up to you to connect to the other nodes and distribute work among them.
By default, you can only connect to nodes that are part of your reservation. If you completely own the nodes within one job (or with one job per '''complete''' node), you will be able to connect those by using <code class="command">ssh</code>. In the case of nodes that are not completely owned within a job (if you have reserved only a part of the nodes or by having multiple jobs on nodes) you will have to use <code class="command">oarsh</code> connector to go from one node to the other. The connector supports the same options as the classical <code class="command">ssh</code> command, so it can be used as a replacement for software expecting ssh. 
{{Term|location=gros-49|cmd=<br>
<code class="command">uniq</code> <code class="env">$OAR_NODEFILE</code> <code># list of resources of your reservation</code><br>
<code class="command">ssh</code> <code class="replace">gros-1</code><code>    # try to connect a node not in the file (should work)</code><br>
<code class="command">oarsh</code> <code class="replace">gros-54</code><code> # connect to the other node of your reservation (should work)</code><br>
<code class="command">ssh</code> <code class="replace">gros-54</code><code> # connect to the other node of your reservation (should work)</code><br>}}
 
{{Note|text=To take advantage of several nodes and distribute work between them, a good option is [[GNU_Parallel]].}}
 
<code class="command">oarsh</code> is a wrapper around <code class="command">ssh</code> that enables the tracking of user jobs inside compute nodes (for example, to enforce the correct sharing of resources when two different jobs share a compute node). If your application does not support choosing a different connector, be sure to reserve nodes entirely (which is the default with <code class="command">oarsub</code>) to be able to use <code class="command">ssh</code>.
 
; Selecting nodes from a specific cluster or cluster type
 
* Reserve nodes from a specific cluster
{{Term|location=fgrenoble|cmd=<code class="command">oarsub</code> <code class="replace">-p dahu</code> <code class="command">-l host=2,walltime=2</code> <code>-I</code>}}
* Reserve nodes in the [[Grid5000:UsagePolicy#Rules_for_the_production_queue|'''production''' queue]]
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code class="replace">-q production</code> <code class="replace">-p grappe</code> <code class="command">-l host=2,walltime=2</code> <code>-I</code>}}
* Reserve nodes from an '''exotic''' cluster type
{{Term|location=flyon|cmd=<code class="command">oarsub</code> <code class="replace">-t exotic</code> <code class="replace">-p pyxis</code> <code class="command">-l host=2,walltime=2</code> <code>-I</code>}}
 
Clusters with the '''exotic''' type either have a non-x86 architecture, or are specific enough to warrant this type. Resources with an exotic type are never selected by default by OAR. Using <code class="command">-t exotic</code> is required to obtain such resources.
 
The type of a cluster can be identified on the [[Hardware]] pages, see for instance [[Lyon:Hardware]].
 
{{Warning|text=When using the <code class="command">-t exotic</code> option, you can still obtain non-exotic resources! You should filter on the cluster name or other properties if you want exclusively exotic resources.}}
 
 
; Selecting specific nodes
 
If you know the exact node you want to reserve, you can specify the hostname of the node you require:
 
{{Term|location=fgrenoble|cmd=<code class="command">oarsub</code> <code class="replace">-p dahu-12</code> <code class="command">-l host=1,walltime=2</code> <code>-I</code>}}
 
If you want several specific nodes, you can use a list:
{{Term|location=fgrenoble|cmd=<code class="command">oarsub</code> <code class="replace">-p "host IN (dahu-5, dahu-12)"</code> <code class="command">-l host=2,walltime=2</code> <code>-I</code>}}
 
 
; Using OAR properties
 
The OAR nodes database contains a set of properties for each node, and the <code class="command">-p</code> option actually filters based on these properties:
* Nodes with Infiniband FDR interfaces:
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code class="replace">-p "ib=FDR"</code> <code class="command">-l host=5,walltime=2 -I</code>}}
* Nodes with power sensors and GPUs:
{{Term|location=flyon|cmd=<code class="command">oarsub</code> <code class="replace">-p "wattmeter=YES AND gpu_count > 0"</code> <code class="command">-l host=2,walltime=2</code> <code>-I</code>}}
* Nodes with 2 GPUs:
{{Term|location=flille|cmd=<code class="command">oarsub</code> <code class="replace">-p "gpu_count = 2"</code> <code class="command">-l host=3,walltime=2</code> <code>-I</code>}}
* Nodes with a specific CPU model:
{{Term|location=flille|cmd=<code class="command">oarsub</code> <code class="replace">-p "cputype = 'Intel Xeon E5-2630 v4'"</code> <code class="command">-l host=3,walltime=2</code> <code>-I</code>}}
* Since <code class="command">-p</code> accepts SQL, you can write advanced queries:
{{Term|location=fnancy|cmd=<code class="command">oarsub</code> <code class="replace">-p "wattmeter=YES AND host NOT IN (graffiti-41, graffiti-42)"</code> <code class="command">-l host=5,walltime=2</code> <code>-I</code>}}
{{Term|location=flille|cmd=<code class="command">oarsub</code> <code class="replace">-p "cputype LIKE 'AMD%'"</code> <code class="command">-l host=3,walltime=2</code> <code>-I</code>}}
 
The OAR properties available on each site are listed on the Monika pages linked from [[Status]] ([https://intranet.grid5000.fr/oar/Nancy/monika.cgi example page for Nancy]). The full list of OAR properties is available on [[OAR_Properties|this page]].
 
{{Note|text=Since this is using a SQL syntax, quoting is important! Use double quotes to enclose the whole query, and single quotes to write strings within the query.}}
 
 
 
=== Advanced job management topics (specific to Grid'5000) ===
 
; Reservations in advance
 
By default, <code class="command">oarsub</code> will give you resources as soon as possible: once submitted, your request enters a queue. This is good for non-interactive work (when you do not care when exactly it will be scheduled), or when you know that the resources are available immediately.
 
You can also reserve resources at a specific time in the future, typically to perform large reservations over nights and week-ends, with the <code class="command">-r</code> parameter:
{{Term|location=fnancy|cmd=<code class="command">oarsub</code><code> -l host=3,walltime=3 </code><code class="command">'''-r'''</code><code> '2020-12-23 16:30:00'</code>}}
 
{{Note|text=Remember that '''all your resource reservations must comply with the [[Grid5000:UsagePolicy#Resources_reservation|Usage Policy]]'''. You can verify your reservations' compliance with the Policy with <code>usagepolicycheck -t</code>.}}
 
; Extending the duration of a reservation
Provided that the resources are still available after your job, you can extend its duration (walltime) using e.g.:
{{Term|location=fnancy|cmd=<code class="command">oarwalltime</code> <code class="replace">12345</code> <code class="replace">+1:30</code>}}
This will request to add one hour and a half to job 12345.
 
For more details, see the oarwalltime section of the [[Advanced_OAR#Changing_the_walltime_of_a_running_job_.28oarwalltime.29|Advanced OAR]] tutorial.
 
== Using nodes in the default environment ==
 
When you run <code class="command">oarsub</code>, you gain access to physical nodes with a default (''standard'') software environment. This is a Debian-based system that is regularly updated by the [[Support|technical team]]. It contains many pre-installed software.
 
=== Environment module  ===
 
* `module avail` : list all available modules
* `module load module_name` : load `module_name` and its dependencies
* `module list` : list all loaded modules
* `module purge` : reset all loaded module
 
fgrenoble:~$ oarsub -I
node:~$ python --version
node:~$ module load python/3.8.12_gcc-10.2.0
node:~$ python --version
 
See https://www.grid5000.fr/w/Modules
 
=== Guix ===
 
Guix is available on frontends and nodes (using standard or deployed ''-nfs'' and ''-big'' environments) directly through the <code>guix</code> command.
For example:
 
$ guix install hello
 
See https://www.grid5000.fr/w/Guix
 
=== Docker ===
 
* To use docker:
 
fgrenoble:~$ oarsub -I
node:~$ g5k-setup-docker
node:~$ docker run hello-world
 
* To use GPU from Docker container using NVIDIA Container Toolkit, use `g5k-setup-nvidia-docker`  instead of `g5k-setup-dock`.
 
=== Singularity ===
 
* To use Singularity:
fgrenoble:~$ oarsub -I
node:~$ singularity run docker://hello-world
 
* To use GPU with Singularity:
fnancy:~$ oarsub -q production -I -l gpu=1
node:~$ singularity run --nv docker://tensorflow/tensorflow:latest-gpu
 
=== Conda ===
 
 
* Load conda module and activate bash completion
 
{{Term|location=fgrenoble|cmd=<code class="command">module load coda</code>}}
{{Term|location=fgrenoble|cmd=<code class="command">eval "$(conda shell.bash hook)"</code>}}
 
* Create an environment
{{Term|location=fgrenoble|cmd=<code class="command">conda create -y -n tensorflow</code>}}
 
* Load this environment
{{Term|location=fgrenoble|cmd=<code class="command">conda activate tensorflow</code>}}
 
* Install tensorflow package
{{Term|location=fgrenoble|cmd=<code class="command">conda install -c conda-forge tensorflow-gpu</code>}}
 
See https://www.grid5000.fr/w/Conda
 
=== Becoming root with sudo-g5k ===
 
On HPC clusters, users typically don't have root access.  However, Grid'5000 allows more flexibility: if you need to install additional system packages or to customize the system, it is possible to become root.  The tool to do this is called [[sudo-g5k]].
 
{{Note|text=Using [[sudo-g5k]] has a cost for the platform: at the end of your job, the node needs to be completely reinstalled so that it is clean for the next user. It is best to avoid running [[sudo-g5k]] in very short jobs.}}
 
 
=== Additional storage ===
 
{| class="wikitable"
|-
! Storage Resource !! Backups? !! Protocol used !! Persistence period !! Provisioning mechanism !! Network connectivity
|-
| [[#.2Fhome|/home]] || No || NFS || long-term || Quota + User Acct mgmt || Variable (1Gb/s - 10 Gb/s)
|-
| [[#Group Storage|Group Storage]] || No || NFS || long-term || Manual || Variable (1Gb/s - 10 Gb/s)
|-
| [[#On_node_local_disks_reservation|On node local disks reservation]] || No || - || medium-term || OAR || -
|-
| /tmp || No || - || short-term (job) || OAR job || -
|-
| [[#Local_disks|Local disks]] || No || - || short-term (job) || OAR job || -
|}
 
More information on storage is available [[Storage|here]].
 
==== Home directory ====
 
The home directory is a network filesystem (NFS): data in your home directory is not actually stored on the node itself, it is stored on a storage server managed by the Grid'5000 team. In particular, it means that all reserved nodes share the same home directory, and it is also shared with the site frontend. For example, you can compile or install software in your home (possibly using pip, virtualenv), and it will be usable on all your nodes.
 
{{Note|text=The home directory is only shared within a site. Two nodes from different sites will not have access to the same home.}}
 
==== Group storage ====
 
This service provides large storage spaces, possibly shared between multiple Grid'5000 users. Those storage spaces are accessible on Grid'5000 over NFS.
 
For instance:
{{Term|location=fnancy|cmd=<code class="command">cd /srv/storage/igrida@storage1.rennes.grid5000.fr</code>}}
 
; Production queue in Nancy
 
The data needed for experiments of the production teams is stored on:
* talc-data (3 volumes - talc, talc2 and talc3 - respectively providing 58T + 58T + 71T = 187T of storage space) is a storage server dedicated to the multispeech research team, but compatible with the Group Storage mechanisms.
* talc-data2 (213T of storage space) is a regular [[Group Storage]] server, and talc-data
 
Please remember that those data are hosted on a NFS server that is not recommended for compute usage.
 
==== Local storage ====
 
Some nodes have additional local disks, see [[Hardware#Storage]] for a list of available disks for each cluster.
 
There are two ways to access these local disks:
 
# On some clusters, '''local disks need to be reserved''' to be accessible. See [[Disk_reservation|Disk reservation]] for a list of these clusters and for documentation on the reservation process.
# On other clusters, '''local disks can be used directly'''. In this case, jump directly to [[Disk_reservation#Using_local_disks_once_connected_on_the_nodes|Using local disks]].
 
In both cases, the disks are simply provided as raw devices, and it is the responsibility of the user to partition them and create a filesystem. Note that there may still be partitions and filesystems present from a previous job.
 
==== /tmp ====
 
The <code class="file">/tmp/</code> directory is stored on a local disk of the node.  Use this directory if you need to access data locally.
The size of /tmp is different from to node to node. It is equal to the total size of the (first) local disk minus 75 GB (which is reserved for the operating system).
 
== Deploying your nodes to get root access and create your own experimental environment ==
 
{{Note|text=There is a tool, called <code class="command">sudo-g5k</code> (see the [[sudo-g5k]] page for details), that provides root access on the ''standard'' environment. It does not allow deep reconfiguration as Kadeploy does, but could be enough if you just need to install additional software, with e.g. <code class="command">sudo-g5k</code><code> apt-get install your-favorite-editor</code>. The node will be transparently reinstalled using Kadeploy after your reservation. Usage of <code class="command">sudo-g5k</code> is logged.}}
 
=== Deploying a system on nodes with Kadeploy ===
 
Reserve one node (the <code class="replace">deploy</code> job type is required to allow deployment with Kadeploy):
{{Term|location=fnancy|cmd=<code class="command">oarsub</code><code> -I -l host=1,walltime=1:45 -t </code><code class="replace">deploy</code>}}
 
Start a deployment of the <code class="env">debian11-base</code> environment on that node (this takes 5 to 10 minutes):
 
{{Term|location=fnancy|cmd=<code class="command">kadeploy3</code> <code class="env">debian11-base</code>}}
By default, all the nodes of the reservation are deployed. Alternatively, you can use <code class="command">-m</code> to specify a node (such as <code class="command">-m</code> <code class="host">gros-42.nancy.grid5000.fr</code>).
 
Kadeploy copy your SSH key from <code class="command">~/.ssh/authorized_keys</code> to the node's root account after deployment, so that you can connect without password. You may want to use another SSH key with <code class="command">-k</code> (such as <code class="command">-k</code> <code class="host">~/custom_authorized_keys</code>).
 
Finally, connect to the node as root:
{{Term|location=fnancy|cmd=<code class="command">ssh</code> <code>root@</code><code class="replace">gros-42</code>}}
 
==== On Grid'5000 reference environments ====
Grid'5000 reference environments are named accordingly to the following scheme: <code class="replace">OS</code><code class="replace">version</code><code class="command">-</code><code class="replace">architecture</code><code class="command">-</code><code class="replace">variant</code>.
* <code class="replace">OS</code><code class="replace">version</code> is the OS distribution name and version, for instance '''<code class="file">debian11</code>''' (Debian 11 "Bullseye", released on 08/2021), '''<code class="file">ubuntu2004</code>''' (Ubuntu 2004 "Focal", released on 04/2020), or '''<code class="file">centos8</code>''' (Centos 8, clone of RHEL 8, released on 09/2019).
*  <code class="replace">variant</code> defines the set of features included in the environment, as follows:
{|class="wikitable"
|-
!style=height:1em  rowspan="2"|Variant
!rowspan="2"|OS available
! rowspan="2"|Grid'5000-specific tuning<br>for performance<br>(e.g., TCP buffers for 10 GbE)
!colspan="5"|Installed tools
! rowspan="2"| Network storage<br>accessible
! rowspan="2"|Hypervisor
|-
! | Stantard system<br>utilities*
! | Common<br>utilities**
! | [[Modules| Scientific software<br/>available via ''module'']]
! | [[Guix| Packages available<br>via ''Guix'']]
! | Advanced<br>packages***
|-
! style=height:4em  rowspan="2" |min
|style="text-align: center; font-weight:bold; background-color:#ffffff;" |Debian
|style="text-align: center; font-weight:bold; background-color:#ffffff;"  rowspan="2" | [[Image:NoStarted.png]]
|style="text-align: center; background-color:#ffffff;"  rowspan="2" |  [[Image:Check.png]]
|style="text-align: center;  background-color:#ffffff;"  rowspan="2" | [[Image:NoStarted.png]]
|style="text-align: center;  background-color:#ffffff;"  rowspan="2" | [[Image:NoStarted.png]]
|style="text-align: center;  background-color:#ffffff;"  rowspan="2" | [[Image:NoStarted.png]]
|style="text-align: center;  background-color:#ffffff;"  rowspan="2" | [[Image:NoStarted.png]]
|style="text-align: center;  background-color:#ffffff;"  rowspan="2" | [[Image:NoStarted.png]]
|style="text-align: center;  background-color:#ffffff;"  rowspan="2" | [[Image:NoStarted.png]]
|-
!style="text-align: center; font-weight:bold; background-color:#ffffff;" | Ubuntu, CentOS, etc.
|-
!style=height:4em|base
|style="text-align: center; font-weight:bold; background-color:#ffffff;" |Debian
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center;  background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center;  background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center;  background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center;  background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center;  background-color:#ffffff;" | [[Image:NoStarted.png]]
|-
!style=height:4em  rowspan="2" |nfs
!style="text-align: center; font-weight:bold; background-color:#ffffff;" |Debian
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" rowspan="2"  | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" rowspan="2" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" rowspan="2"  | [[Image:NoStarted.png]]
|style="text-align: center; font-weight:bold; background-color:#cce7ff;" rowspan="3"| Support for:
- mounting your home, group<br>storage and Ceph.
 
- using Grid'5000 user account<br>on deployed nodes.
|style="text-align: center;  background-color:#ffffff;"  rowspan="2"  | [[Image:NoStarted.png]]
|-
|style="text-align: center; font-weight:bold; background-color:#ffffff;" |Ubuntu, CentOS, etc.
|style="text-align: center; background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:NoStarted.png]]
|-
!style=height:4em|big
|style="text-align: center; font-weight:bold; background-color:#ffffff;" |Debian
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center;  background-color:#ffffff;"  | [[Image:NoStarted.png]]
|-
|-
!style=height:4em|xen
|style="text-align: center; font-weight:bold; background-color:#ffffff;" |Debian
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:Check.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center; background-color:#ffffff;" | [[Image:NoStarted.png]]
|style="text-align: center; font-weight:bold; background-color:#cce7ff;" |Xen hypervisor Dom0
+ minimal DomU
|}
 
<small>'''<nowiki>*</nowiki>''' ''Including SSH server and network drivers.''<br>
'''<nowiki>**</nowiki>''' ''Including among others: Python, Ruby, curl, git, vim, etc.''<br>
'''<nowiki>***</nowiki>''' ''Packages for development, system tools, editors and shells.''</small>
 
The list of all supported environments is available by running <code class="command>kaenv3</code> on any frontend. Note that environments are versioned: old versions can be listed using the <code class="command>kaenv3 -l -s</code> command and a former version retrieved and used by adding the <code class="command">--env-version</code><code class="replace">YYYYMMDDHH</code> option to the <code class="command">kaenv3</code> or <code class="command">kadeploy3</code> commands (also see the <code class="command">man</code> pages). This can be useful to reproduce experiments months or years later,  using a previous version of an environment. On some sites, environments exist on different architectures (<code class="env">x86_64</code>, <code class="env">ppc64le</code> and <code class="env">aarch64</code>). The full list can be found in the [[Advanced_Kadeploy#Search_an_environment|Advanced Kadeploy]] page.
 
The Grid'5000 reference environments are build from recipes using the '''<code class="command">kameleon</code>''' tool from recipes detailing the whole construction process, and updated on a regular basis (see versions). See the [[Environment creation]] page for details.
 
== Using efficiently Grid'5000 ==
 
Until now you have been logging, and submitting jobs manually to Grid'5000. This way of doing is convenient for learning, prototyping, and exploring ideas. But it may quickly become tedious when it comes to performing a set of experiments on a daily basis. In order to be more efficient and user-friendly, Grid'5000 also support more convenient ways of submitting jobs, such as [[API|API requests]] and [[Notebooks|computational notebooks]].
 
=== Notebooks ===
 
Grid'5000 also supports Jupyter notebooks and Jupyter lab servers.
Jupyter lab servers provide you with a simple web interface to submit jobs on Grid'5000 and run python Notebooks.
Using notebooks will allow you to track your experiment evolution during your exploratory phase while scripting part of your process.
 
You can find more information about Jupyter Lab and python notebooks on the '''[[Notebooks]]''' page.
 
=== Scripting libraries ===
 
Several scripting libraries built on top of the Grid'5000 API is available:
 
* '''[http://execo.gforge.inria.fr/doc/latest-stable/index.html Execo]''' offers a Python API for asynchronous control of local or remote, standalone or parallel, unix processes. It is especially well suited for quickly and easily scripting workflows of parallel/distributed operations on local or remote hosts: automate a scientific workflow, conduct computer science experiments, perform automated tests, etc. The core python package is execo. The execo_g5k package provides a set of tools and extensions for the Grid5000 testbed. The execo_engine package provides tools to ease the development of computer sciences experiments.
* '''[https://github.com/ruby-cute/ruby-cute Ruby-Cute]''' is a set of Commonly Used Tools for Experiments, or Critically Useful Tools for Experiments, depending on who you ask. It is a library aggregating various Ruby snippets useful in the context of (but not limited to) development of experiment software on distributed systems testbeds such as Grid'5000. Ruby-Cute is structured in different modules. G5K module allows you to communicate with Grid'5000. Net::SSH::Multi module allows the parallel execution of commands in several remote machines using the SSH protocol. TakTuk module is a wrapper of taktuk parallel command executor.
* '''[[Funk]]''' helps you to find resources for your experiments, by:
** giving you the number of nodes available at a date and for walltime
** finding the slots for a combination of resources and a walltime
** finding the slot with the maximum number of nodes for a period and a walltime
** managing the reservation of the resources
* '''[https://discovery.gitlabpages.inria.fr/enoslib/ EnOSlib]''' is a python library that mutualizes common experiment practices especially when dealing with distributed applications deployments. EnOSlib uses different providers to get resources from an infrastructure. For instance, on Grid'5000 one can easily get a physical environment (non-deploy job/deploy job with or without multiple NICs configured) or a virtualized environment (e.g based on kvm virtual machines). Resources are configured using safe parallel actions (based on Ansible Modules) or using on-the-shelf packaged applications (e.g a monitoring stack, a distributed network packets sniffer). 
* '''[https://jobqueue.dask.org/en/latest/index.html Dask-jobqueue]''' is a Python library which makes it easy to deploy Dask on common job queuing systems typically found in high performance supercomputers, academic research institutions, and other clusters. It can be used to facilite the passage between different resource managers, based on OAR and Slurm for example.
 
== Going further ==
In this tutorial, you learned the basics of Grid'5000:
* The general structure of Grid'5000, and how to move between sites
* How to manage you data (one NFS server per site; remember: it is not backed up)
* How to find and reserve resources using OAR and the <code class="command">oarsub</code> command
* How to get root access on nodes using Kadeploy and the <code class="command">kadeploy3</code> command
 
You should now be ready to use Grid'5000.
 
=== Additional tutorials ===
There are '''many more tutorials available on the [[Users Home]] page'''. Please have a look at the page to continue learning how to use Grid'5000.

Revision as of 18:43, 5 February 2025

NB: does not concern users from center clusters migrated to Abaca, their accounts will be initialized by the technical team (they will receive an e-mail notification).

Retrieved from "https://www.grid5000.fr/mediawiki/index.php?title=Production:Getting_Started&oldid=104916"