Hemera:Tools

Here we present the tools which have been developped by Hemera people to ease large scale experiments on Grid'5000.

Experiment Engines

• EXPO: experiment engine for distributed platforms. It aims at simplifying the experimental process on such platforms. More informations on [1].
• XPFlow: a new approach to description and execution of experiments involving large-scale computer installations. More informations on [2].
• execo: a Python library that allows you to finely manage unix processes on thousands of remote hosts. More informations on [3].

Experiment planification

A new module has been added to Execo to directly access to the OAR database and retrieve the planning of the various ressources available on the Grid'5000 sites. This module can be used in a Python experimental script (see Execo planning example).

For non-Python user, a command line tool has been developped to ease a multi-site experiment planification.and is installed on any Grid'5000 frontend.

Please have a look at Funk documentation.

Virtualization

A Python module, vm5k, has been created to help users managing virtual machines on Grid'5000. It allow to easily manage up to 10 000 virtual machines.

It is composed of:

• a command line tool that automatically configure the hosts and deploy virtual machines on them,
• a execo_engine that allow reproducible experiments involving virtual machines
• a lib to deploy debian with KVM and libvirt and to manage virtual machines on these hosts

Documentation can be found here.

Monitoring

The Hemera initiative take a leading part in the development and deployment of a new monitoring system. It is based on the Kwapi framework, originally developped for OpenStack, and has been adapted and extended to provide fine-grain live monitoring of energy consumption and network traffic.

More information on page Monitoring.

Storage

• Hadoop_On_Execo: The Apache Hadoop project provides an open-source framework for reliable, scalable, distributed computing. As such, it can be deployed and used in the Grid 5000 platform. However, its configuration and management may be sometimes difficult, specially under the dynamic nature of clusters within Grid 5000 reservations. In turn, Execo offers a Python API to manage processes execution. It is well suited for quick and easy creation of reproducible experiments on distributed hosts. hadoop_g5k provides a layer built on top of Execo that allows to manage Hadoop clusters and prepare reproducible experiments in Hadoop. It offers a set of scripts to be used in command-line interfaces and a Python API.

Network topology

topo5k is a tool based on Execo that help you in discovering the network topology. Here is a compact view of the Grid'5000 infrastructure.