TakTuk
During the course of an experiment on G5k, one often wants to run programs or scripts on all the reserved nodes simultaneously. For instance, to send data to remote workers. This could be achieved sequentially with a loop construct in a script, but this method does not scale well. That's why a launcher for parallel commands named TakTuk has been installed on each G5k site.
Short Explanation about TakTuk mechanics
To speed up the remote execution of the same command on a large number of nodes, TakTuk both
- uses a deployment tree to initiate several connections to remote peers at the same time
- initiates locally a few connections in parallel (the deployment window)
The balance beetween the two mechanism is ensured by an adaptive work-stealing algorithm combined with an on-line evaluation of the local machine load (see A guide to TakTuk use by Guillaume Huard for more details).
Therefore, for taktuk to operate correctly, peers should be able to connect to each other. By default, TakTuk uses ssh connections. So, to check if Taktuk execution can work, please follow instructions on this page: Test key authentication. There is also a practical example for setting up taktuk in the Advanced OAR tutorial.
Using TakTuk with the base environment
Prerequisites
You own a reservation for some machines:
|
users@frontale.site.grid5000.fr:
|
oarsub -I -l nodes=4 |
You are logged on one of the reserved nodes:
|
|
users@node-15.site.grid5000.fr:
|
|
Connections can be made without password from this node and between nodes using your account.
Using Taktuk to launch scripts
To execute scripts simultaneously on all the nodes:
|
|
users@node-15.site.grid5000.fr:
|
taktuk -f $OAR_FILE_NODES broadcast exec [ /home/site/user/testScript.sh ] |
Note: do not forget to type the characters [ and ] and separate them from their content (using spaces for instance).
By default, TakTuk displays a lot of informations and part of it might not be necessary. Basically, you can choose exactly what is displayed by TakTuk (type of information and format of the displayed messages) using the output-template option. For instance, the following options suppress the connector and status streams and format the output of commands to use the host name followed by the content of the output:
-o connector -o status -o output='"$host: $line\n"'
Example:
To check if time clocks are synchronized on all the nodes (Result formated correctly)
|
|
users@node-15.site.grid5000.fr:
|
taktuk -c "oarsh" -o connector -o status -o output='"$host: $line\n"' -f $OAR_FILE_NODES broadcast exec [ date ] |
Output
node-101.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007 node-23.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007 node-24.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007 node-67.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007 node-97.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
To find CPUs installed on nodes:
|
|
user@node-15.site.grid5000.fr:
|
taktuk -c "oarsh" -o connector -o status -o output='"$host: $line\n"' -f $OAR_FILE_NODES broadcast exec [ "cat /proc/cpuinfo | grep 'model name'" ] | uniq |
Output
node-101.bordeaux.grid5000.fr: model name : Intel(R) Xeon(TM) CPU 3.00GHz node-23.bordeaux.grid5000.fr: model name : AMD Opteron(tm) Processor 248 node-24.bordeaux.grid5000.fr: model name : AMD Opteron(tm) Processor 248 node-67.bordeaux.grid5000.fr: model name : Intel(R) Xeon(TM) CPU 3.00GHz node-97.bordeaux.grid5000.fr: model name : Intel(R) Xeon(TM) CPU 3.00GHz
Note: instead of using the option -c "oarsh" in each command, we could have type the following command :
|
|
users@node-15.site.grid5000.fr:
|
export TAKTUK_CONNECTOR=oarsh |
and then taktuk would know that it need to use oarsh as connector.
Using TakTuk with your own environment but without TakTuk installation
Prerequisites
Start by making a reservation:
|
|
users@frontale.site.grid5000.fr:
|
oarsub -I -l nodes=4 -t deploy |
Deploy your environment:
|
|
users@frontale.site.grid5000.fr:
|
kadeploy3 -f $OAR_FILE_NODES -e wheezy-x64-nfs -k |
Using Taktuk to launch programs
Using the -s option, you can use taktuk to deploy on remote hosts that don't have taktuk installed.
|
|
users@frontale.site.grid5000.fr:
|
taktuk -s -o connector -o status -o output='"$host: $line\n"' -f $OAR_FILE_NODES broadcast exec [ "apt-get update" ] |
To execute an apt-get as root for system update, you have to use the following option: "-l root" before any machine specification.
|
|
users@frontale.site.grid5000.fr:
|
taktuk -l root -s -o connector -o status -o output='"$host: $line\n"' -f $OAR_FILE_NODES broadcast exec [ "apt-get update" ] |
Examples
- To execute a simple command like
dateon 1 node:
taktuk -m node-5.site.grid5000.fr broadcast exec [ date ]
- Same execution but on more than 1 node:
taktuk -m node-5.site.grid5000.fr -m node-10.site.grid5000.fr broadcast exec [ date ]
- Execute command on nodes given by OAR after reservation
taktuk -f $OAR_FILE_NODES broadcast exec [ date ]
- Self propagation of TakTuk if it is not available everywhere (works only if you can login from nodes to nodes without typing your password):
taktuk -s -f $OAR_FILE_NODES broadcast exec [ date ]
- Connect to nodes with a different login,
rootfor instance:
taktuk -l root -f $OAR_FILE_NODES broadcast exec [ date ]
- In order to test latencies between nodes using taktuk:
As a user from the frontend, run:
taktuk -c "ssh -p 6667" -s -o 'state=(($line == 4)?($init_date - $start_date)." s de connection de $host a $peer (dont ".($init_date-$reply_date)." du a TakTuk)\n":"")' -f allnodes.txt b e [ true ] | tee taktuk.log
where taktuk version is at least 3.3 and allnodes.txt contains the list of the nodes.
Then doing a grep connection taktuk.log | sort -n gives:
0.220579862594604 s de connection de azur-25.sophia.grid5000.fr a sol-12.sophia.grid5000.fr (dont 0.047921895980835 du a TakTuk) 0.220623016357422 s de connection de azur-17.sophia.grid5000.fr a sol-36.sophia.grid5000.fr (dont 0.0476739406585693 du a TakTuk) 0.222663879394531 s de connection de azur-1.sophia.grid5000.fr a sol-24.sophia.grid5000.fr (dont 0.0479059219360352 du a TakTuk) 0.225576162338257 s de connection de azur-30.sophia.grid5000.fr a sol-27.sophia.grid5000.fr (dont 0.0478250980377197 du a TakTuk) 0.226027965545654 s de connection de azur-16.sophia.grid5000.fr a sol-31.sophia.grid5000.fr (dont 0.0470089912414551 du a TakTuk) ...