G5k-checks

Description

g5k-checks

• g5k-checks is expected to be integrated into the production environment of the Grid'5000 computational nodes. It gathers a collection of programs which check that a node meets several basic requirements before it declares itself as available to the OAR server.
• This lets the admins enable some checkers which may be very specific to the hardware of a cluster.

g5k-checks executes at boot time in two phases:

• Phase 1
  • An init script, /etc/init.d/g5k-checks, runs all checkers that must run early enough in the boot process.
  • They are listed in the variable CHECKS_FOR_INIT in the configuration file.
  • Then it enables all checkers listed in the variable CHECKS_FOR_OAR for Phase 2.
• Phase 2
  • This phase strongly relies on the check mechanism provided by OAR and the oar-node configuration file (/etc/default/oar-node for deb distros, /etc/sysconfig/oar-node for rpm ones).
  • The oar-node flavour of OAR installation embeds an hourly cron job, oarnodecheckrun, which runs all executable files stored in /etc/oar/check.d/. Then the server periodically invokes remotely oarnodecheckquery which will return with status 0 if and if only there are some log files in a given OAR directory. So if a checker in /etc/oar/check.d/ finds something wrong, it simply has to create a log file in that directory.
  • The version of /etc/(default|sysconfig)/oar-node that g5k-checks installs runs both oarnodecheckrun and oarnodecheckquery scripts. If the latter fails, then the node is not ready to start, and it loops on running those scripts until either oarnodecheckquery returns 0 or a timeout is reached. If the timeout is reached, then it does not attempt to declare the node as "Alive".
  • During Phase 1, the enabling of a checker simply turns out to adding a symbolic link in /etc/oar/check.d to its "oar-node driver". We name this a short script file which interfaces the core checker to the OAR check mechanism.

At any moment when the node is running g5k-checks may be called either to disable or to enable checks. This is expected to be used by OAR prologue and epilogue:

• /etc/init.d/g5k-checks stop: disable OAR checks
• /etc/init.d/g5k-checks start: enable OAR checks for oarnodecheckrun
• /etc/init.d/g5k-checks startrun: enable OAR checks for oarnodecheckrun and run once the couple oarnodecheckrun/oarnodecheckquery, without waiting for one hour to be passed.

Basically, the OAR prologue should call "stop", while the epilogue should call "startrun".

• At installation time, g5k-checks configures the local syslog daemon: it first looks for a free <n> such as local<n>.alert and local<n>.warning selectors are not used, and then defines them with the action "@syslog". If there is no "syslog" host on the local network, then it defaults to writing messages in the local syslog file.

• The checkers use the local<n> facility to report only important messages. They use a local log file for debugging messages. Please see section 3.2 for further details.

g5k-parts

g5k-parts is designed to run at both phases of g5k-checks (see above).

• In Phase 1, g5k-parts validates the partitioning of a Grid'5000 computational node against the G5K Node Storage convention: all partitions but /tmp are primary, and /tmp is a logical partition inside the only extended partition.
• It first compares /etc/fstab with its backup generated at deployment time. When errors are found at this level, /etc/fstab is reset and the machine reboots.
• Then for every partition given on the command line, it first matches its geometry on the hard drive with the partition layout saved at deployment time. It may perform several other checks (e.g. the formatting of the partition) depending on the partition role. It attempts to fix errors, which it reports to the syslog system. Sometimes, g5k-parts cannot fix the error (e.g. hard drive errors); it can only prevent the node from declaring itself as alive to the OAR server, with a simple stamp file (not executable !) whose existence is tested by the oar-node driver of g5k-parts in Phase 2:

/etc/oar/check.d/g5k-parts-init-failed

• There is a special processing for NFS shares which the node must mount at boot time. Sometimes such mounts fail and this is not due to the node itself but to the NFS server(s) or the network connection. To avoid blocking the boot process (and have kadeploy3 fail because of a timeout when it should not), g5k-parts tries every mount only once. Subsequent tries are done in Phase 2.
• In Phase 2, the oar-node driver of g5k-parts calls the script with a single argument, "nfs", in order to limit the checks to the NFS shares.

g5k-proc

• g5k-proc aims at verifying if the caracteristics of the processor model and the processor frequency are those expected.
• To perform this test, g5k-proc is based on g5k-api-lib.rb which parses the reference API to retrieve information about the processor model and the processor frequency.
• Then, it compares the retrieved information to the contents of /proc/cpuinfo.

g5k-ethernet-link

• g5k-ethernet-link checks whereas the ethernet link has been correctly negociated.
• Like g5k-proc, it parses the reference API and retrieves information about the Ethernet devices such as the rate.
• Then, it makes a comparison between the retrieved value and the output of ethtool.
• This checker could be enhanced to manage the InfiniBand, Myrinet or still the second ethernet interface if this one is configured on the production environment.

Installation

Production environment

Dependencies:

• deb distributions
  • ethtool
  • rubygems + run "gem install rest-client"
• rpm distributions
  • ethtool
  • rubygem-mime-types
  • rubygem-rest-client

Check that the variables defined in Makefile.defs do match your needs.

make DISTRIB=(rpm|deb) install

Check that the syslog configuration which was generated is correct.

Postinstalls of the Production Environment

Please read carefully the instructions printed out by the make command above.

Adding a new Check

In this section we assume that you want to add a check named "new-check". First create the new-check/ directory at the root of the source tree.

Simple Case

In the "simple" case, there are only three files involved:

• bin/new-check, the core checker
• check.d/new-check, the oar-node driver
• Makefile:

topdir = ..
CHECK = new-check
include $(topdir)/Makefile.check

The core checker is the core part of the check. It can be a bash or a ruby script, but if it needs information from the G5K API, then you had better develop it in ruby, for the ruby library g5k-api-lib.rb provides all facilities to retrieve them.

The core checker MUST use the logging functions provided by the bash and ruby libraries with the conventions listed in section 3.2.

The oar-node driver interfaces the call to the check with the oar-node check mechanism. It is called without any argument but with an environment variable set: CHECKLOGFILE. It is the path of a file in the directory that oarnodecheckquery will search into. The oar-node driver should create it if the check fails. Just look at some examples in the check.d/ directories of existing checks.

Logging

The core checker MUST use the logging functions provided by g5k-api-lib.rb and libg5kchecks.bash, with the following convention:

• log A, "msg": send msg as an alert to the syslog system
• log W; "msg": send msg as a warning to the syslog system
• log I; "msg": send msg to a local log file, /var/log/g5k-checks.log

$LOG may be used to redirect the stdout and/or stderr of some commands.

• The alert level must be used for errors which prevent the node from being qualified for OAR reservation. An admin must have a look.
• The warning level must be used when the error is severe enough for admins to be interested in having a look at it, but which do not disqualify the node for OAR reservation: typically, when the checker can fix it.
• Alert and Warning level messages should not be longer than one-line, as far as

it is possible.

• The information level is intended to give a complement of information to the admin who would have a look: be free to log any kind of information there. At this level, you can use any of the following methods:
  • through the function: log I "msg"
  • directly into $LOG: e.g. >> ${LOG} 2>&1

BEWARE always to APPEND text, and NOT ERASE preceding information !!!

Finally, note that your checker may be called every hour by the oar-node check mechanism, so try to make it as silent as possible when it is run from the oar-node driver...

Complex Case

Some checks may need run earlier in the boot sequence than oar-node. Just like for the oar-node driver of the simple case, you must develop an "init" driver and move it to new-check/init.d/new-check. /etc/init.d/g5k-checks will call this driver with no argument before it enables the oar-node checks (see 1.1) For instance, g5k-parts need both an init and an oar-node driver.

Some checks may need more data than those set in the global g5k-checks configuration file. Those data shall be stored in $G5KCHECKS_HOME/data/ after installation. Their installation (and their generation if needed) shall be managed by new-check/Makefile. The Makefile of the simple case will not fit, of course. The Makefile you will develop shall meet the only requirement that the following targets must be defined (even if no command is associated):

         install:
         install-msg:
         clean:
         uninstall:
         uninstall-msg:

Once again, g5k-parts is a good example: its Makefile is based on the simple case Makefile, but add requisites to the compulsory targets... Note that it makes a full usage of the variables defined in Makefile.defs too.

Librairies

If you find some code duplicated in several checkers, then feel free to move it to the lib/ directory, either in the existing files if it makes sense, or in a new library. Only make sure the checker will load it...