Funk: Difference between revisions

From Grid5000
Jump to navigation Jump to search
 
(62 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
{{Portal|User}}
{| border="0" cellpadding="5"  
{| border="0" cellpadding="5"  
|-
|-
Line 5: Line 5:
|valign="top"|  
|valign="top"|  


'''(F)ind yo(U)r (N)odes on g5(K)''' is a tool that help you to plan your multi-site experiments on Grid5000. It is a Python script based on [http://execo.gforge.inria.fr/doc/ Execo] that:
'''(F)ind yo(U)r (N)odes on g5(K)''' is a tool that help you to plan your experiments on Grid'5000. It is a Python script based on [https://gitlab.inria.fr/mimbert/execo Execo] that:
* give you the number of nodes available at a date and for walltime
* give you the number of nodes available at a date and for walltime
* find the slots for a combination of resources and a walltime
* find the slots for a combination of resources and a walltime
* find the slot with the maximum number of nodes for a period and a walltime.
* find the slot with the maximum number of nodes for a period and a walltime.
Some basic examples can be found [[#examples|here]].
|valign="top"| [[File:Funk_Slots.png|300px]]
|}


Projet source, issue and feature requests can be found [https://github.com/lpouillo/Funk here].
Projet source, issue and feature requests can be found [https://github.com/lpouillo/Funk here].


Some basic usage can be found [[#usage|here]].
{{Warning|text=This tool is currenlty in development}}
|valign="top"| [[File:Funk_Slots.png|300px]]
|}


==Running the tool==


==Running the tool==
===Inside Grid'5000===
{{Term|location=frontend|cmd=$ <code class="command">funk</code> -h}}
See below for options descriptions.


===No installation===
===Outside Grid'5000===
A bash wrapper is installed on '''lyon''' frontend in '''/home/lpouilloux/bin/''' and allow you to use funk without any configuration and installation.
You need to install [https://mimbert.gitlabpages.inria.fr/execo/userguide.html#installation execo] and [https://mimbert.gitlabpages.inria.fr/execo/userguide.html#more-advanced-usages configure it for Grid'5000].
You must log into the Lyon frontend, move to the directory and run ./funk with the options you need (see [[#options|below]]).
    ssh lyon
    /home/lpouilloux/bin/funk -h


===With installation===
You also need [https://pypi.python.org/pypi/httplib2 python-httplib2] to make use of the request on the Grid'5000 [[API]].  
You need to install the [http://execo.gforge.inria.fr/doc/tutorial.html#installation git version of execo].
If you have no admin rights, install it with :
  easy_install --user httplib2
or
  pip install --user httplib2
Once it is done, you can clone the repository with:
Once it is done, you can clone the repository with:
  git clone https://github.com/lpouillo/Funk.git
  git clone https://github.com/lpouillo/Funk.git
And run the script with  
and run the script.
    ./funk.py
 
==<span id="examples">Usage</span>==
I want to find how many nodes are currently available on sites lyon,nancy,rennes for a walltime of 3 hours
{{Term|location=frontend|cmd=<code class="command">funk</code> -m <code class="replace">date</code> -r <code class="replace">lyon,nancy,rennes</code> -w <code class="replace">3:00:00</code>}}
I have a deadline in one week and I absolutely need to run a large
scale experiment on the whole platform (avoiding periods under usage policy
restrictions)
{{Term|location=frontend|cmd=<code class="command">funk</code> -m <code class="replace">max</code> -r <code class="replace">grid5000</code> -w <code class="replace">4:00:00</code> -e <code class="replace">"2014-02-17 16:00:00"</code> -c}}
I need specific resources combination for my experiments and I want to know when a slot will be available, with deploy job type, and reserve it automatically (-y option) :
{{Term|location=frontend|cmd=<code class="command">funk</code> -m <code class="replace">free</code> -r <code class="replace">taurus:2,nancy:100</code> -w <code class="replace">10:00:00</code> -o <code class="replace">"-t deploy"</code> -y}}
 
Please be aware of the [[Grid5000:UsagePolicy|Grid'5000 Usage Policy]] or use <code class="replace">-c</code> to avoid usage policy periods.
A walltime for a night reservation is 11:58:00 or is for a weekend 62:58:00.


==<span id="options">Options</span>==
==<span id="options">Options</span>==
===running===
===General options===
* (-m): three modes available
Define mode and controls I/O.
** ''now''  = give you the number of nodes available at the time the script is run
** ''free'' = find the slots for a combination of resources
** ''max''  = find the maximum number of nodes for the period specified.
* (-y): perform the reservation automatically
* (-v): verbose mode (a lot of debugging informations)
* (-q): quiet mode (no informations displayed)
* (-p): draw gantt diagramm and slots (require matplotlib 1.2, not available on Grid'5000 frontends)


===resources===
  -h, --help            show this help message and exit
* (-r): define the elements you need by giving a comma separated list of elements (grid5000, site_uid, cluster_uid)
  --version            show program's version number and exit
** element1,element2,element3 for mode ''now'' and ''max''
  -m MODE, --mode MODE  Setup the mode: date, free or max
** element1:n_nodes1,element2:n_nodes2,element3:n_nodes3 for mode ''free''
                        default = date
* (-k): ask for a global KaVLAN
  -y, --yes            Perform the reservation automatically
* (-o): define your oargridsub options (default = '-t deploy')
  --quiet              Run without printing anything
  -v, --verbose        Run in verbose mode
  -p PROGRAM, --program PROGRAM
                        The program to be run when the reservation start
  --plots              Draw a Gantt plot and the slots
  --json                Output the computed data to standard output
  --json-file JSON_FILE
                        Output the computed data to a json file


===time===
===Reservation===
* (-w): walltime of the reservation (1:00:00)
{{Warning|text=[[OARGrid2]] has been discontinuited and is not maintained anymore.<br>
* (-s): starting date for the planning computation (2013-07-12 16:12:35)
If you use an old version of Funk, please adapt your usage by using "--no-oargrid" option.}}
* (-e): ending date for the planning computation (2013-07-16 16:12:35)
Customize your Grid'5000 reservation.
  -r RESOURCES, --resources RESOURCES
                        Comma separated list of Grid'5000 elements  (grid5000, site or cluster)
                        -r element1,element2 for date and max modes
                        -r element1:n_nodes1,element2:n_nodes2 for free mode
                        default = grid5000
  -b BLACKLIST, --blacklist BLACKLIST
                        Remove clusters from planning computation
  -R RATIO, --ratio RATIO
                        Apply a given ratio to the host resources
  -o SUBMISSION_OPTS, --submission_opts SUBMISSION_OPTS
                        Extra options to pass to the oarsub/oargridsub command line
  -k, --kavlan          Ask for a KaVLAN
  -n SUBNET, --subnet SUBNET
                        Ask for subnets. slash_22=1 will retrieve a /22 subnet on every site of your requests,
                        but you can specify site1:slash_22=2,site2:slash_19=1
  -j JOB_NAME, --job_name JOB_NAME
                        The job name passed to the OAR subjobs
                        default = FUNK
  -q QUEUE, --queue QUEUE
                        The OAR queue to use
                        default = default
  --virtual            Use only clusters with hardware virtualization
  --infiniband          Use only clusters with infiniband or myrinet interfaces
  --green              Use only clusters that have energetical power measurements
  --no-oargrid          Don't use oargridsub and perform parallel oar reservations
  --force-oargrid      Force the usage of oargrisub, even if only one site is involved


==<span id="usage">Usage</span>==
===Time===
Finding the number of available nodes from now to now + walltime
Define options related to date and time.
  funk.py -w 1:00:00 -m now -r grid5000
  -w WALLTIME, --walltime WALLTIME
Finding the first free slots for a resource combination
                        Reservation walltime in OAR format
  funk.py -w 2:00:00 -m free -r lille:10,lyon:10,sophia:10
                        default = 1:00:00
Finding the maximum number of nodes available for the resource and with a
  -s STARTDATE, --startdate STARTDATE
KaVLAN
                        Starting date in OAR format
  funk.py -w 10:00:00 -m max -r lyon,sophia,edel -k
                        default = 1443470554
  -e ENDDATE, --enddate ENDDATE
                        End date in OAR format
                        default = 1445284954
  -c, --charter        Avoid charter (usage policy) periods

Latest revision as of 11:15, 12 June 2024

(F)ind yo(U)r (N)odes on g5(K) is a tool that help you to plan your experiments on Grid'5000. It is a Python script based on Execo that:

  • give you the number of nodes available at a date and for walltime
  • find the slots for a combination of resources and a walltime
  • find the slot with the maximum number of nodes for a period and a walltime.

Some basic examples can be found here.

Funk Slots.png

Projet source, issue and feature requests can be found here.


Running the tool

Inside Grid'5000

Terminal.png frontend:
$ funk -h

See below for options descriptions.

Outside Grid'5000

You need to install execo and configure it for Grid'5000.

You also need python-httplib2 to make use of the request on the Grid'5000 API. If you have no admin rights, install it with : 

 easy_install --user httplib2

or 

 pip install --user httplib2

Once it is done, you can clone the repository with: 

 git clone https://github.com/lpouillo/Funk.git

and run the script.

Usage

I want to find how many nodes are currently available on sites lyon,nancy,rennes for a walltime of 3 hours

Terminal.png frontend:
funk -m date -r lyon,nancy,rennes -w 3:00:00

I have a deadline in one week and I absolutely need to run a large scale experiment on the whole platform (avoiding periods under usage policy restrictions)

Terminal.png frontend:
funk -m max -r grid5000 -w 4:00:00 -e "2014-02-17 16:00:00" -c

I need specific resources combination for my experiments and I want to know when a slot will be available, with deploy job type, and reserve it automatically (-y option) :

Terminal.png frontend:
funk -m free -r taurus:2,nancy:100 -w 10:00:00 -o "-t deploy" -y

Please be aware of the Grid'5000 Usage Policy or use -c to avoid usage policy periods. A walltime for a night reservation is 11:58:00 or is for a weekend 62:58:00.

Options

General options

Define mode and controls I/O. 

 -h, --help            show this help message and exit
 --version             show program's version number and exit
 -m MODE, --mode MODE  Setup the mode: date, free or max 
                       default = date
 -y, --yes             Perform the reservation automatically
 --quiet               Run without printing anything
 -v, --verbose         Run in verbose mode
 -p PROGRAM, --program PROGRAM
                       The program to be run when the reservation start
 --plots               Draw a Gantt plot and the slots
 --json                Output the computed data to standard output
 --json-file JSON_FILE
                       Output the computed data to a json file

Reservation

Warning.png Warning

OARGrid2 has been discontinuited and is not maintained anymore.
If you use an old version of Funk, please adapt your usage by using "--no-oargrid" option.

Customize your Grid'5000 reservation. 

 -r RESOURCES, --resources RESOURCES
                       Comma separated list of Grid'5000 elements  (grid5000, site or cluster)
                       -r element1,element2 for date and max modes
                       -r element1:n_nodes1,element2:n_nodes2 for free mode
                       default = grid5000
 -b BLACKLIST, --blacklist BLACKLIST
                       Remove clusters from planning computation
 -R RATIO, --ratio RATIO
                       Apply a given ratio to the host resources
 -o SUBMISSION_OPTS, --submission_opts SUBMISSION_OPTS
                       Extra options to pass to the oarsub/oargridsub command line
 -k, --kavlan          Ask for a KaVLAN
 -n SUBNET, --subnet SUBNET
                       Ask for subnets. slash_22=1 will retrieve a /22 subnet on every site of your requests, 
                       but you can specify site1:slash_22=2,site2:slash_19=1
 -j JOB_NAME, --job_name JOB_NAME
                       The job name passed to the OAR subjobs
                       default = FUNK
 -q QUEUE, --queue QUEUE
                       The OAR queue to use
                       default = default
 --virtual             Use only clusters with hardware virtualization
 --infiniband          Use only clusters with infiniband or myrinet interfaces
 --green               Use only clusters that have energetical power measurements
 --no-oargrid          Don't use oargridsub and perform parallel oar reservations
 --force-oargrid       Force the usage of oargrisub, even if only one site is involved

Time

Define options related to date and time. 

 -w WALLTIME, --walltime WALLTIME
                       Reservation walltime in OAR format
                       default = 1:00:00
 -s STARTDATE, --startdate STARTDATE
                       Starting date in OAR format
                       default = 1443470554
 -e ENDDATE, --enddate ENDDATE
                       End date in OAR format
                       default = 1445284954
 -c, --charter         Avoid charter (usage policy) periods
Retrieved from "https://www.grid5000.fr/mediawiki/index.php?title=Funk&oldid=103049"