Hemera:WG:Methodology

Completing challenging experiments on Grid’5000 (Methodology)

Leaders: Lucas Nussbaum (ALGORILLE), Olivier Richard (MESCAL)

Experimental platforms like Grid’5000 or PlanetLab provide an invaluable help to the scientific community, by making it possible to run very large-scale experiments in con- trolled environment. However, while performing relatively simple experiments is generally easy, it has been shown that the complexity of completing more challenging experiments (involving a large number of nodes, changes to the environment to introduce heterogene- ity or faults, or instrumentation of the platform to extract data during the experiment) is often underestimated.

This working group will explore different complementary approaches, that are the basic building blocks for building the next level of experimentations on large scale exper- imental platforms. This encompasses several aspects.

First, designing a complex experiment raises strong methodological issues: the exper- imental scenario, as well as the experimental conditions and metrics, need to be carefully chosen. At this stage, it is also important to consider experimentation at real-scale as one element in a set of different evaluation methods, and the possibility to combine it with other complementary approaches such as modelling and simulation.

After the various parameters of the experiment have been chosen, the experiment need to be translated into a serie of actions on the platform. Reserving and controlling thousands of resources requires the use of specialized tools, such as scalable parallel launchers, data distribution solutions, and domain-specific languages able to express the numerous steps that compose experiments in an efficient way.

While Grid’5000 already provides some heterogeneity, it is often necessary to intro- duce more heterogeneity and/or changing experimental conditions during the experiment. For example, validating the ability of an application to cope with degraded network con- ditions, requires the ability to control the experimental environment in ways are that not available for standard Grid’5000 experiments. Techniques such as emulation or fault injection provides some answer to that problem, but there is still some work to be done before it they are easily usable on Grid’5000.

Finally, the extraction of results from experiments in Grid’5000 is a problem in itself: it requires tools that are both scalable (possibly collecting data from thousands of nodes), accurate, and non-intrusive (to limit the observer effect). While monitoring systems like Ganglia already provide some functionality, they have numerous limitations that prevent them from being usable in the context of Grid’5000 experiments.

Grid'5000 Gotchas wiki page

The Grid5000:Gotchas is a result of this WG.

Notes from the methodology BOF at Grid'5000 Spring School 2011

Objectives:

Discuss methods, tools, good practices and tips facilitating making experiments on Grid'5000. Try to outline:

• actions that the community should take to improve the situation
• actions that the technical team should consider taking (improvement of the platform, development or improvement of existing tools)

List of ideas to discuss:

• Purpose of using G5K
• Connecting and moving around, manually and automatically
• Managing data between sites
• Editing source code
• Reserving resources
• Deploying
• Grid'5000 API
  • reference
  • reservation / deployment
  • metrology
• automating experiments
• collecting data during your experiment (monitoring)
• analyzing data
• things that annoy you about Grid'5000

How we worked:

Each attendee to the BOF described how he used Grid'5000, and what are his problems. We discussed possible improvements to their workflow, and solutions to their problems.

Swann Perarnau (PhD Grenoble):

experiments mostly on one machine, using different architectures
PB: description of machines (wrt architecture)
* e.g size of cache, different cache levels often not completely documented => import hwloc output into API?
PB: need to change some bios parameters
* how to restore parameters at end of job?
Possible implementations:
* do it using kaconsole + expect
* do it after boot, using machine-specific tools (pb: need RHEL env, etc.)
* do it by flashing the bios (PXE + FreeDOS, N. Turro + nvram?)
* CoreBoot?
PB: users can change the bios config, and nobody will notice.
PB: ligforge not whitelisted by http proxy
PB: cohérence nommage dans l'API (majuscules/minuscules, etc...) + structure.
=> faire une passe sur les catégories + le nommage. Intégration avec hwloc pour faciliter la tache ?
Utilise scripts shell pour automatiser les expés (awk grep sed)
Pour analyser les données, R + tikz/pgfplot

Pierre Neyron (IR CNRS Grenoble):

Aide les autres à faire des expés
Besoin T. Gautier: machines très récentes dans G5K, en un seul exemplaire (tests préliminaires)
Besoin utilisateurs GPU: avoir accès à "leurs" machines dès qu'ils en ont besoin
Besoin utilisateurs GPU: dernière version des outils NVIDIA (CUDA 4)

Pierre Riteau (PhD Rennes):

XP liées à la virtualisation et au cloud (hyperviseur modifé pour live migration, modifs dans stack de cloud pour propager des images plus rapidement)
SSH + ProxyCommand (simsim pour configurer .ssh/config automatiquement)
Gérer les données: dépots Git (dépot sur chaque frontale)
Envs NFS pour avoir le home monté sur les noeuds
PB: embêtant de devoir synchroniser ses données entre les sites
Solution: sshfs? mais pas installé sur les frontales
Solution: monter le home NFS d'un site sur les noeuds de tous les sites (mais serveurs NFS configurés de manière trop restrictive => vérifier qu'on a "*" dans  /etc/exports sur tous les serveurs -- Pierre dit qu'il le fera)
(Grid)?Gantt+disco pour découvrir les ressources, + oarsub (pas l'API -- il faut un humain dans la boucle pour savoir comment "configurer" le job)
Aimerait réécrire disco avec l'API pour gagner en perf
Déploiement avec kadeploy en interactif pour bidouiller, utilisation de la lib ruby gosen (qui utilise l'API) pour scripter les déploiements
Utilisation de l'API de réf dans ses scripts Nimbus pour découvrir les caractéristiques des noeuds et configurer son déploiement Nimbus
N'utilise pas l'API métrologie
Automatisation:
script bash, qui se transforme en script ruby (net-ssh, net-ssh-multi)
la plupart du temps un gros script sans vraie organisation, factorisation quand des patterns apparaissent
le script récupère les résultats et les écrit ds un fichier
fichiers récupérés sur machine locale, d'autres scripts pour aggréger
gnuplot pour tracer les graphes
réfléchit à passer à R
PBs:
* machines "défectueuses" non détectées qui pourrissent les résultats d'expériences
=> tester le débit disque avec g5k-checks?
=> pas de g5k-checks dans l'env de référence. il faudrait un g5k-checks lançable par l'utilisateur dans son env déployé, qui vérifie différentes valeurs de perf sur le noeud (ex: perf CPU au cas où "qqun" modifie des valeurs dans le bios - see above). Outil distribué pour les tests réseaux?
=> machines avec virtualisation HW désactivée

Mathieu Djamaï (PhD DOLPHIN, Lille):

déploie des réseaux P2P pour de l'optim combinatoire
data: un exécutable diffusé sur tous les sites
oargridsub pour réserver les ressources
PB: découverte de la topologie physique
Intéressé par la documentation de la topo physique dans l'API
PB: génère beaucoup de fichiers (150000 noeuds, 30 fichiers chacun) sur NFS
PB: découpe son XP en tranches, pb de l'état à la terminaison d'un job
Solutions:
* calculer la date de fin du job, faire son checkpoint "un peu avant" (OAR est plus en retard qu'en avance pour killer les jobs)
* utiliser le signal envoyé par OAR pour démarrer le checkpoint
automatisation en script shell
post-processing avec script shell + gnuplot

Ali ASIM (Post-doc DOLPHIN, Lille):

During PhD, XP on distributed simulation
Could not deploy env created on one site in other sites
PB: used sid-x64-base (still not removed!!)

Actions that should be considered, either by the tech team or by the community:

• consider the ability to change the BIOS configuration during future Kadeploy developments
• improve the description of machines in the Grid'5000 API
  • use hwloc to improve the description of the CPUs
  • do a pass on the description of all sites to refresh/correct the data and improve the structure
• whitelist ligforge and other popular forges on the HTTP proxies
• install sshfs on frontends (and allow users to use it)
• check that it's possible to mount every NFS server from every site
• rewrite disco using the Grid'5000 API
• improve g5k-checks
  • in order to enable users to run tests on deployed nodes
  • with more tests (disk bandwidth, CPU performance in case someone changed some CPU settings)
• make sure the sid-x64-* environments are removed everywhere