TakTuk

During the course of an experiment on G5k, one often wants to run programs or scripts on all the reserved nodes simultaneously. For instance, to send data to remote workers. This could be achieved sequentially with a loop construct in a script, but this method does not scale well. That's why a launcher for parallel commands named TakTuk has been installed on each G5k site.

What is TakTuk

TakTuk is a parallele launcher.

To speed up the remote execution of the same command on a large number of nodes, TakTuk both

• uses a deployment tree to initiate several connections to remote peers at the same time
• initiates locally a few connections in parallel (the deployment window)

The balance beetween the two mechanism is ensured by an adaptive work-stealing algorithm combined with an on-line evaluation of the local machine load (see A guide to TakTuk use by Guillaume Huard for more details).

Therefore, for taktuk to operate correctly, peers should be able to connect to each other. By default, TakTuk uses ssh connections. So, to check if Taktuk execution can work, please follow instructions on this page: Test key authentication. There is also a practical example for setting up taktuk in the Advanced OAR tutorial.

External Links

• TakTuk Homepage
• TakTuk Documentation

Using TakTuk in Grid'5000

Using TakTuk with Grid'5000's default environment

Prerequisites

You own a reservation for some machines:

users@frontale.site.grid5000.fr:

oarsub -I -l nodes=4

You are logged on one of the reserved nodes:

users@node-15.site.grid5000.fr:

Connections can be made without password from this node and between nodes using your account.

Using Taktuk to launch scripts

To execute scripts simultaneously on all the nodes:

users@node-15.site.grid5000.fr:

taktuk -c "oarsh" -f $OAR_FILE_NODES broadcast exec [ /home/site/user/testScript.sh ]

Note: do not forget to type the characters [ and ] and separate them from their content (using spaces for instance).

Note: taktuk uses classic ssh by default. It's possible to do without the -c "oarsh" but in this case you'll have to type oarsub -I -l nodes=4 -t allow_classic_ssh (see : https://www.grid5000.fr/mediawiki/index.php/Getting_Started#Reserving_resources_with_OAR:_the_basics )

By default, TakTuk displays a lot of informations and part of it might not be necessary. Basically, you can choose exactly what is displayed by TakTuk (type of information and format of the displayed messages) using the output-template option. For instance, the following options suppress the connector and status streams and format the output of commands to use the host name followed by the content of the output:

-o connector -o status -o output='"$host: $line\n"'

Example

To check if time clocks are synchronized on all the nodes (Result formated correctly)

users@node-15.site.grid5000.fr:

taktuk -c "oarsh" -o connector -o status -o output='"$host: $line\n"' -f <( uniq $OAR_FILE_NODES ) broadcast exec [ date ]

Output

node-101.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
node-23.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
node-24.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
node-67.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007
node-97.bordeaux.grid5000.fr: Fri Mar 23 09:38:46 CET 2007

To find CPUs installed on nodes

user@node-15.site.grid5000.fr:

taktuk -c "oarsh" -o connector -o status -o output='"$host: $line\n"' -f <( uniq $OAR_FILE_NODES ) broadcast exec [ "cat /proc/cpuinfo | grep 'model name'" ] | uniq

Output

node-101.bordeaux.grid5000.fr: model name       :                   Intel(R) Xeon(TM) CPU 3.00GHz
node-23.bordeaux.grid5000.fr: model name        : AMD Opteron(tm) Processor 248
node-24.bordeaux.grid5000.fr: model name        : AMD Opteron(tm) Processor 248
node-67.bordeaux.grid5000.fr: model name        :                   Intel(R) Xeon(TM) CPU 3.00GHz
node-97.bordeaux.grid5000.fr: model name        :                   Intel(R) Xeon(TM) CPU 3.00GHz

Note: instead of using the option -c "oarsh" in each command, we could type the following command :

users@node-15.site.grid5000.fr:

export TAKTUK_CONNECTOR=oarsh

and then taktuk would know that it needs to use oarsh as a connector.

Using TakTuk with your own environment but without TakTuk installation

Prerequisites

Start by making a reservation:

users@frontale.site.grid5000.fr:

oarsub -I -l nodes=4 -t deploy

Deploy your environment:

users@frontale.site.grid5000.fr:

kadeploy3 -f $OAR_FILE_NODES -e debian9-x64-base -k

Using Taktuk to launch programs

Using the -s option, you can use taktuk to deploy on remote hosts that don't have taktuk installed.

users@frontale.site.grid5000.fr:

taktuk -s -o connector -o status -o output='"$host: $line\n"' -f <( uniq $OAR_FILE_NODES ) broadcast exec [ "apt-get update" ]

To execute an apt-get as root for system update, you have to use the following option: "-l root" before any machine specification.

users@frontale.site.grid5000.fr:

taktuk -l root -s -o connector -o status -o output='"$host: $line\n"' -f <( uniq $OAR_FILE_NODES ) broadcast exec [ "apt-get update" ]

Some extra examples

• To execute a simple command like date on 1 node:

taktuk -m node-5.site.grid5000.fr broadcast exec [ date ]

• Same execution but on more than 1 node:

taktuk -m node-5.site.grid5000.fr -m node-10.site.grid5000.fr broadcast exec [ date ]

• Execute command on nodes given by OAR after reservation

taktuk -f $OAR_FILE_NODES broadcast exec [ date ]

• Self propagation of TakTuk if it is not available everywhere (works only if you can login from nodes to nodes without typing your password):

taktuk -s -f $OAR_FILE_NODES broadcast exec [ date ]

• Connect to nodes with a different login, root for instance:

taktuk -l root -f $OAR_FILE_NODES broadcast exec [ date ]

• In order to test latencies between nodes using taktuk:

As a user from the frontend, run:

taktuk -c "ssh -p 6667" -s -o 'state=(($line == 4)?($init_date - $start_date)." s de connection de $host a $peer (dont ".($init_date-$reply_date)." du a TakTuk)\n":"")' -f allnodes.txt b e [ true ] | tee taktuk.log

where taktuk version is at least 3.3 and allnodes.txt contains the list of the nodes.

Then doing a grep connection taktuk.log | sort -n gives:

0.220579862594604 s de connection de azur-25.sophia.grid5000.fr a sol-12.sophia.grid5000.fr (dont 0.047921895980835 du a TakTuk)
0.220623016357422 s de connection de azur-17.sophia.grid5000.fr a sol-36.sophia.grid5000.fr (dont 0.0476739406585693 du a TakTuk)
0.222663879394531 s de connection de azur-1.sophia.grid5000.fr a sol-24.sophia.grid5000.fr (dont 0.0479059219360352 du a TakTuk)
0.225576162338257 s de connection de azur-30.sophia.grid5000.fr a sol-27.sophia.grid5000.fr (dont 0.0478250980377197 du a TakTuk)
0.226027965545654 s de connection de azur-16.sophia.grid5000.fr a sol-31.sophia.grid5000.fr (dont 0.0470089912414551 du a TakTuk)
...