CLCAR 2009 OAR Tutorial
This practice is about running jobs on a cluster. You will learn how to access a Grid'5000 cluster, how to install your data and how to run your jobs and visualize them.
Access cluster
Enter site
On Grid'5000, you can only submit your jobs when you are inside a site. So you must connect to one of the sites of the grid.
Once you know where you want to run your job, the command to connect to the site is:
(do not forget to replace login with your Grid'5000 login and site by the chosen site, for example nancy)
|
outside:
|
ssh login@access.site.grid5000.fr |
Then you have to connect you to the frontend machine to make OAR reservations
|
|
accessMachine:
|
ssh frontend |
Note that in some sites, access machine and frontend are the same server.
Prepare the experiment environment
An experiment is often composed of running codes and their configuration. The experiments of this practice are based on the HPLinpack benchmark. This benchmark quantifies the number of floating operations per second during the solving of a random dense linear system.
The programs to run and their configuration are available outside of Grid'5000 infrastructures. Thus, we need to retrieve those external data an install them on the cluster where we are currently connected.
Finally we prepare the execution environment of the experiment.
Setup data on the cluster
We retrieve our data on the frontend.
The experiment data are stored on a specific directory on grid5000 frontend clusters. Data retrieval can be performed on the frontal using the following command:
|
|
frontend:
|
cp /home/grenoble/ygeorgiou/runhpl.tgz .
|
Once done, we unpack our experiment data:
|
|
frontend:
|
tar xvzf ~/runhpl.tgz
|
A directory called runhpl is now visible in our home directory. This directory will be available on every compute node of this site, thanks to a NFS-mount of user home directories.
|
Warning |
|---|---|
During the execution of programs that generate of lot of disk writing, it is a good idea to do these writing on each node local disk and not on your networkly-shared home directory. This way, you will avoid a lot of NFS troubles, like lags or breakdowns. As the | |
Interactive run
OAR2, the Grid'5000 batch scheduler, has an interactive mode. This mode connects the user to the first of his allocated nodes.
Submission
Submit an interactive job:
|
|
frontend:
|
oarsub -I |
OAR2 returns a number. This number identify our submission on the various visualization interfaces:
OAR_JOB_ID=8670
You were automatically connected on the first node that is allocated to your job. OAR2 initialize several environment variables set to reflect current submission. These variables can be used by your script to adapt to current submission properties:
|
|
node:
|
env |
Especially the list of your dedicated nodes can be viewed:
|
|
node:
|
cat $OAR_NODEFILE |
|
Note |
|---|---|
Depending on the site where you made the submission, the same node name may appear more than one time inside the | |
It is time to run our script:
|
|
node:
|
~/runhpl/runhpl |
Results is going to be printed on the standard output. Remember that we quantifies the number of floating operation per second.
Visualization
Submission is visible on the Monika web interface of the site where it was submitted:
Cluster status cannot be obtained in command-line from the node where you were connected by OAR. You need another terminal connected to the oar machine of the site or cluster (do not forget to replace OAR_JOB_ID by your current submission id):
|
|
frontend:
|
oarstat -f -j OAR_JOB_ID |
Ending
With interactive submission, submission ending is not related to the end of your script run. It depends on the connection to the submission main node that OAR automatically made for you. Thus you can run as many scripts as you want before job deadline.
|
|
Warning |
|---|---|
Your current submission uses a default walltime of 1 hours. If you are still connected to the submission nodes after 1 hours, OAR2 will automatically disconnect you. | |
By quitting the shell opened for you by OAR upon the oarsub command (e.g. shell on the first node of your job), your will terminate the job. Thus, as we are finished with our job here, you can end it by typing type:
|
|
node:
|
exit |
Web interfaces and command-line tools reflect submission ending:
Visualize cluster
Experiment data are ready, before running the job, we are now going to analyze cluster state, which can be visualized in many ways.
oarstat
oarstat is a command-line tool to view current or planned job submission.
View each submissions:
|
|
frontend:
|
oarstat |
View each submission details:
|
|
frontend:
|
oarstat -f |
View a specific submission details
|
|
frontend:
|
oarstat -f -j OAR_JOB_ID |
View each submissions from a given user: View the status of a specified job:
|
|
frontend:
|
oarstat -s -j OAR_JOB_ID |
View each submissions from a given user:
|
|
frontend:
|
oarstat -u login |
oarnodes
oarnodes is also a command-line tool. It shows cluster node properties:
|
|
frontend:
|
oarnodes |
Among returned information there is current node state. This state is generally Alive or Absent. When nodes are sick, their state is Suspected or Dead.
oarprint
oarprint is a tool providing a pretty print of a job resources.
The command prints a sorted output of the resources of a job with regard to a key property, with a customisable format.
On a job connection node (where $OAR_RESOURCE_PROPERTIES_FILE is defined):
|
|
node:
|
oarprint host -P host,cpu,core -F "host: % cpu: % core: %" -C+ |
On the submission frontend:
|
|
frontend:
|
oarstat -j OAR_JOB_ID -p | oarprint core -P host,cpuset,memcore -F "%[%] (%)" -f - |
Monika
Monika is a web interface which in a way synthesizes information given by oarstat and oarnodes. Current node state is displayed and the current and planned submission list is printed at the bottom of the web page.
Starting from the site list web page, click on the site or the cluster where your are currently connected to view its current state:
DrawOARGantt
Monika displays current node status and current and planned job submissions. DrawOARGantt is also a web interface, but it prints past, current and planned node states on a temporal diagram.
Starting from the site list web page, click on the site or the cluster where your are currently connected to briefly manipulate the web interface options:
Ganglia
Node load, memory usage, cpu usage and so on are available with the Ganglia web interface.
Starting from this interface home page, click on the site or the cluster where your are currently connected to quickly analyze available metrics and diagrams:
|
|
Note |
|---|---|
| |
By default Ganglia displays current metrics values but you can view an abstract of these values over the past year.
Passive run
OAR2, Grid'5000 batch scheduler, has a passive mode. In this mode, a script specified at submission time is run on the main dedicated node. This script must know about dedicated nodes to split its work between them.
Submission
Submit a passive job with our script:
|
|
frontend:
|
oarsub ~/runhpl/runhpl |
|
|
Note |
|---|---|
Environment variables, described during our interactive submission, are also initialized by OAR for passive jobs. | |
Visualization
Submission is visible on the Monika web interface of this site where it was submitted:
You can also view current cluster status with command-line tool (do not forget to replace OAR_JOB_ID by your current submission id):
|
|
frontend:
|
oarstat -f -j OAR_JOB_ID |
Sometime it is useful to know when a passive job starts. It is possible to do that by using the "--state" or "-s" option of oarstat. This option is optimized to get the status of a specified job faster than with "-f" option. It is especially designed for a scripting usage.
|
|
frontend:
|
oarstat -s -j OAR_JOB_ID |
The following script launchs in a passive job the runhpl program and waits until the job starts.
launchJob.sh :
#!/bin/bash
my_script="~/runhpl/runhpl"
oar_job_id=`oarsub $my_script | grep "OAR_JOB_ID" | cut -d '=' -f2`
oar_stdout_file="OAR.$oar_job_id.stdout"
until oarstat -s -j $oar_job_id | grep Running ; do
echo "Waiting for the job to start..."
sleep 1
done
echo "Job $oar_job_id is started !"
To analyze standard output stdout and standard error output stderr of the main node, where our script is run, OAR puts theses output in files named OAR.OAR_JOB_ID.stdname. Depending on your submission id, you should see in your home directory:
OAR.OAR_JOB_ID.stdoutOAR.OAR_JOB_ID.stderr
Thus you can follow your job's run in live:
|
|
frontend:
|
tail -f OAR.OAR_JOB_ID.stdout |
To quit job monitoring, please press Ctrl + c.
You can specify the files that will store the standard output and error stream of the job by using the -O and -E option in the oarsub command. For example it's possible to redirect the oarsub output in /dev/null or in other file
|
|
frontend:
|
oarsub ~/runhpl/runhpl -O /dev/null |
|
|
frontend:
|
oarsub ~/runhpl/runhpl -O ~/runhpl.log |
Ending
OAR submission automatically ends when script ends. You can see this on the Monika web interface of the site where the script was submitted. Job's dedicated nodes returned in their available state:
Submission left its print on the DrawGantt web interface of its host site:
Submission has also gone from command-line status:
|
|
frontend:
|
oarstat |
Our job's results are available at the end of standard output file:
|
|
frontend:
|
tail OAR.OAR_JOB_ID.stdout |
Node number specification
Until now our submissions used the default node number: 1 node.
We are going now to submit an interactive job on 2 nodes:
|
|
frontend:
|
oarsub -I -l nodes=2 |
We are automatically connected to one of these 2 nodes due to interactive submission. We can learn about our dedicated resources:
|
|
node:
|
cat $OAR_NODEFILE |
As you can read, the benchmark script detects available node number and can adapt to 1, 2 and 4 nodes:
|
|
node:
|
~/runhpl/runhpl |
We can verify that the run occurs on the other node with another terminal (do not forget to replace OAR_JOB_ID by the current job identifier and hostname by the allocated node name):
|
|
frontend:
|
export
OAR_JOB_ID=OAR_JOB_ID |
|
|
frontend:
|
oarsh hostname ps -C xhpl.x86_64 |
Planning
Until now our submissions used the default start time and the default duration: immediate start and 1 hour duration. We are going to submit jobs with a specific duration and a delayed start.
Let us run the script on December 7th, 2008 at 3:30pm for a 10-minute duration:
|
|
frontend:
|
oarsub -r '2008-12-07 15:30:00' -l nodes=2,walltime=0:10:00 ~/runhpl/runhpl |
|
|
Note |
|---|---|
You can submit your job without specifying script when you do delayed submission. In this case, allocated nodes are waiting for job's owner activity. | |
The delayed submission appears as Scheduled on the Monika web interface of the site where it was submitted or on the command-line (do not forget to replace OAR_JOB_ID by your submission id):
|
|
frontend:
|
oarstat -f -j OAR_JOB_ID |
When it is time for submission to begin, you can connect to its main node to interactively run the script or monitor its run:
|
|
frontend:
|
oarsub -C OAR_JOB_ID -I |
Submission ending does not occur when you disconnect from its main node, even if you omit specifying a script to run. Ending occurs when specified script end or when duration deadline expire. If you do not specify a script to run and you finished, it is a good idea to release allocated nodes earlier:
|
|
frontend:
|
oardel OAR_JOB_ID |
Pay attention to the fact that your job will now be in the Error state, in this case.
Quick Glossary
Please refer to the following definitions to better understand the different OAR mecanisms:
OAR Grid
OAR Grid usage is presented in the next tutorial : Grid experiment