Environment creation

From Grid5000
Revision as of 18:55, 7 December 2021 by Pneyron (talk | contribs) (Created page with "<!-- TODO * update https://www.grid5000.fr/w/Getting_Started#Deploying_nodes_with_Kadeploy to use Template:Reference environments * use Template:Reference environments in http...")

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Grid'5000 provides bare-metal as a service for experimenting on distributed computers, thanks to the kadeploy service. While kadeploy handles the efficient deployment of a user's customized system (also named environment in the Grid'5000 terminology) on many nodes, companion tools allow building such custom systems environments. Hence, this page describes the Grid'5000 environment creation processes, with the several methods for doing it, and helpful tools.

Introduction: the several ways for preparing a custom system environment

There are different ways to prepare a system environment for experiments:

  • (1) A first way consists in deploying a provided environment, e.g. Grid'5000 supported debian10-x64-big, and adding software and custom configurations to it after the initial deployment. That every time a new experiment is run (new deploy job). While it can be relevant, it has an obvious bias: the post-deployment setup is not factorized and must be redone every time and on all nodes.
  • (2) A second way consists in building a master system environment with all the wanted customizations, then deploying that pre-built environment on the experiment nodes. This way, the environment preparation is only done once for all times and all nodes: it is factorized.

Once again, building such a customized master system environment can be achieved in different ways:

  • (2-a) A first way consists in deploying an already provided environment (such as one of the Grid'5000 supported reference environments) on one node, doing some customizations on that node, then finally saving the operating system of the node as a master environment image, to later allow deploying it on all the nodes of an experiment. This usually involves the tgz-g5k command to create the master environment image (tarball).
  • (2-b) A second way consists in building the master environment to deploy on all the nodes of an experiment from a recipe which describes the whole system environment construction process (possibly from scratch). This obviously allows for the reconstructibility and sharing of the environment, hence it helps the reproducibility of the experiment. This involves the same build process that is used to produce the Grid'5000 reference environments, using the kameleon tool (Grid'5000 environment recipes are written in the kameleon recipe and Puppet languages).
Note.png Note

Even though it is somehow out of context in this page, we can also mention the use of the sudo-g5k command, which allows a user to gain right away the root privileges whenever needed in the production environment (available on machines by default), hence without requiring a deploy job and to actually deploy an environment with kadeploy beforehand. In the context of the creation of custom system environment, using sudo-g5k:

  • can simplify (1), because it avoids requiring an initial deployment. The standard environment (currently debian10 std) is just used.
  • can also simplify (2-a): after using sudo-g5k to modify the std environment as root, tgz-g5k can be used to export an environment image from the modified standard environment.
In both cases, one must understand that this however has the some drawbacks: it limits to using the debian10 std environment as base system on the nodes of the experiment, which may include some unnecessary complexity or limitations

In the remaining of this page, (1) will not be detailed: it is let to the user to choose a tool to deploy software and configurations on running systems, such as clush, taktuk, ansible, etc.

About Grid'5000 supported environments

While building a system environment from scratch (only taking a generic OS installation media as a base) may be doable, it is technically extremely difficult (see the last section of this page). Most Grid'5000 users should rather create customized environments on top of existing works, already done for Grid'5000.

The Grid'5000 technical team provides several reference environments, which a user's customized environment can be built on top of.

Of course, a user can also build a new customized environment on top of another user's customized environment.

The next sections give detailed technical howtos for customizing existing Grid'5000 environments, first following way (2-a), then way (2-b).

Creating an environment images using tgz-g5k

In this section, following the (2-a) way described above, we explain how to extend an existing Grid'5000 environment by first deploying it on a machine with kadeploy3, then bringing customization to that machine, and finally archiving the operating system of the machine with tgz-g5k to create a new environment image.

Note that this method allows building environment in the format to be deployed with kadeploy3, not VM image to use for instance with qemu (but qcow2 VM disk image can be modified in place). Also note that the next section shows how to build a VM disk image from a recipe using kameleon.

1: deploy the existing environment on a machine

First, we have to create the deploy job, to reserve a machine on which we will deploy the existing environment of our choice, which our customized environment will be based on.

Note.png Note

At this stage, it is wise choosing a Grid'5000 site and cluster that is not too loaded, furthermore using rather old hardware is of special interest because newer hardware usually has significantly longer boot time → see the Hardware page.

We do an interactive job (-I), of the deploy type (-t deploy), on only one machine (-l host=1). We will give ourselves 3 hours with -l walltime=3.

Terminal.png frontend:
oarsub -I -t deploy -l host=1,walltime=3

The interactive job opens a new shell on the frontend (careful: the job ends when exiting that shell).

Since the hostname of the reserved machine is stored in the $OAR_FILE_NODES file, we can deploy the reference environment of our choice (or another user's environment that we would like to extend) with kadeploy3:

Terminal.png frontend:
kadeploy3 -k -e debian10-x64-base -f $OAR_FILE_NODES

(if the chosen environment is not registered in kaenv3, see the -a option of kadeploy3 to point to a environment description file).

2: customize the environment

Once the deployment has run successfully, we can connect to the machine using ssh as root without password, and do any customization using shell commands.

Terminal.png frontend:
ssh root@hostname

You can therefore update your environment (to add any missing library you need, remove any package that you don't need in order to sizes down the image and possibly speeds up the deployment process, etc.)

Note: When you are done with the customization, mind clearing temporary files or caches to save disk space.

3: archive the environment image

We can now archive the customized environment, using tgz-g5k to create a Grid'5000 environment image from the filesystem of the machine. The environment image is a tarball of the filesystem of the OS with some adaptations.

Terminal.png frontend:
tgz-g5k -m hostname -f ~/environment_image.tgz

This will create a file named environment_image.tgz in your home directory on the frontend.

Note.png Note

About tgz-g5k:

  • If you want to create an image of a machine that run the Grid'5000 default environment (i.e. not in a deploy job) and that you modified after gaining the root privileges with using sudo-g5k, the -o option of tgz-g5k must be used so that the connection to the machine is done using oarsh/oarcp instead of ssh/scp.
  • If you want tgz-g5k to access the machine with a custom user id, you can use -u username (default is root).
  • More information on tgz-g5k in tgz-g5k -h or man tgz-g5k.

4: create the environment description file

The new environment image cannot be deployed directly: the image is only one part of an environment. An environment is described by a YAML document. To use the new image, it must be referred by an environment description, so that deploying that environment uses the new image. Note that the environment includes also other information such as the postinstall script and the kernel command line for instance, which can be changed independently from changing the environment image.

The easiest way to create a description for your new environment is to modify the description of the environment it is based on.

Since we used the debian10-x64-base reference environment, we can retrieve its description using the kaenv3 command and save it to a file. Then we'll use it as a base for the description of our customized environment.

Terminal.png frontend:
kaenv3 -p debian10-x64-base -u deploy > my-custom-environment.yaml

We now edit the file to change the environment name, version, description, author, and so on. The image file entry must of course be changed to point to our new environment image tarball file. Since it is stored locally in our home directory, the path can be a simple absolute path (remove the server:// prefix). Finally, the visibility line should be removed or its value changed to shared or private.

 1 ---
 2 name: my-debian
 3 version: 1
 4 description: my customized environment based on debian 10 (buster) - base
 5 author: john@doe.org
 6 visibility: shared
 7 destructive: false
 8 os: linux
 9 image:
10   file: /home/jdoe/environment_image.tgz
11   kind: tar
12   compression: gzip
13 postinstalls:
14 - archive: server:///grid5000/postinstalls/g5k-postinstall.tgz
15   compression: gzip
16   script: g5k-postinstall --net debian
17 boot:
18   kernel: "/vmlinuz"
19   initrd: "/initrd.img"
20   kernel_params:
21 filesystem: ext4
22 partition_type: 131
23 multipart: false

Once this is done, our customized environment is ready to be deployed (in a deploy job) using:

Terminal.png frontend:
kadeploy3 -k -f $OAR_NODEFILE -a my-custom-environment.yaml

(This kind of deployment is called anonymous deployment because the description is not yet in the Kadeploy3 environment registry. It is particularly useful when working by iteration on the environment, thus having to recreate the environment image several times. Otherwise, since registered environments are checksummed, changing the image file requires updating the registration every time with kaenv3)

Once your customized environment is ready, it's optionally the time to add it to the Kadeploy3 environment registry:

Terminal.png frontend:
kaenv3 -a my-custom-environment.yaml

Assuming you set the name field in the environment description to "my-debian", it will then be deployable using the following command:

Terminal.png frontend:
kadeploy3 -k -f $OAR_NODEFILE -e my-debian

If the visibility is set to shared, your environment will show up in the list of available registered environment for any user, using kaenv3 -l -u your_username.

Warning.png Warning

The registration of the environment does not make a copy of the environment image and postinstall files! Do not remove them or the environment will be broken. Also, environments registered by users are not automatically replicated on all sites (there is one kadeploy registry per site).

Creating an environment from a recipe using kameleon and puppet

In this section, following the (2-b) way described above, we explain how to build environments from recipes describing the whole creation process, rather than doing interactive modifications in command-line then use tgz-g5k to export the system to an image. With the method, all steps requested to build the environment are described, which helps traceability, reconstructability, and sharing.

Recipes are processed by a tool named kameleon to build the environments. Furthermore, for the recipes for Debian stable, kameleon let most of the process of the setup of the environment operating system to puppet.

About kameleon

Kameleon is a powerful utility to generate operating system environments (also known as appliances or images in other contexts) from recipes.

A kameleon recipe is composed of a main YAML file: the recipe. That recipe possibly depends on other YAML files: recipes it extends and the macro steps. It also depends on various other files: data, helper scripts,....

The recipe defines 3 sections:

  • bootstrap: describes typically how to install the base system from the installer, run it in a VM (qemu).
  • setup: describes modifications on the base system (done in the bootstrapped VM) in order to prepare the environment with all the desired features.
  • export: creates the system environment (e.g. the kadeploy environment, a VM image, ...) by exporting the content of the bootstrapped and modified (setup) VM.

In order to help share recipes, kameleon provides a template mechanism (with a template repository) that allows creating a recipe by extending another recipe. Typically, only the setup section should be modified when extending an existing recipe.

Kameleon has a lot of other features like build context isolation, interactive breakpoints, and so on. Context isolation means that kameleon can run a build process without altering the operating system from where the tool is called itself (kameleon typically uses qemu VMs for the build). Kameleon does not need to run as root.

See the Kameleon website for more information, such as explanations of the recipe syntax used in the YAML files. But mind that the most relevant information on building Grid'5000 environments is provided here.

About the use of kameleon in Grid'5000

Grid'5000 reference environments (environments that are provided/supported by the Technical Team) are built with kameleon, using recipes located in the environments-recipes repository.

The goal of providing this repository is threefold:

  1. users may look at how Grid'5000 reference environments a build
  2. users may modify Grid'5000 reference environments recipes (discouraged, prefer using them as templates that you extend, which is next point)
  3. users can use that repository as a template repository, to build their own recipe by extending a Grid'5000 reference environment recipe taken as a template.

This repository contains different kinds of recipes of interest for Grid'5000 users:

recipes to build the Grid'5000 Debian stable environments
those recipes have a setup section implemented using Puppet. As explaned earlier, different variants are offered for every version of Debian stable.
recipes to build Grid'5000 environments for some other Linux distributions (Debian testing, Ubuntu, Centos, ...)
Only a min (minimalistic) variant of the OS is offered. Those recipes do not use Puppet.
a recipe to build a new environment by starting from the image of another Grid'5000 environment (not from scratch)
See below for more details in the dedicated section.
other recipes that are not directly of interest for Grid'5000 (but may be used as dependencies of the Grid'5000 recipes)

Preparing the workspace to use kameleon

As explained above, we will use the Grid'5000 environment recipes as templates to build an environment dedicated to our experiment.

To do so, we first have to prepare our workspace to use kameleon.

Warning.png Warning

In the remaining of this document, kameleon commands are always run on a Grid'5000 machine in a regular job (not of type deploy), not on the personal workstation and never on Grid'5000 frontends. The root privilege is not required to build an environment with kameleon (except on PPC64 machines/the drac cluster where sudo-g5k ppc64_cpu --smt=off must be run to deactivate hyperthreading before running kameleon, because qemu on PPC64 does not support hyperthreading).

First we create an interactive job to reserve a node where to run the kameleon commands:

Terminal.png frontend:
oarsub -I

Kameleon is preinstalled on the nodes.

We then install install the repository of recipes for the Grid'5000 environments:

Terminal.png node:

In case you already installed the repository previously, you may want to update it. To do so, run the following command:

Terminal.png node:
kameleon repository update grid5000

You can then list the available recipe templates with:

Terminal.png node:
kameleon template list

As explained above, we see here the templates for Debian stable (9, 10, and 11) with their different variants. We also see the templates for other distributions with only the min variant. Finally, we see the template to build from an existing Grid'5000 environment. We'll detail the use of each of those templates below.

Finally we will create a new directory to store our work

Terminal.png node:
mkdir ~/my_recipes && cd ~/my_recipes

Create a new recipe

Creating an environment by extending a Grid'5000 recipe using Puppet

We present here how to extend a Grid'5000 recipe and use Puppet to bring some customizations in a traceable way.

As a reminder, Puppet is only used in the Debian stable environment recipes of Grid'5000. We will extend one of those.

The names of the Debian stable (Debian 9, 10 and 11) recipe templates end by a word after a dash: that's the variant name. Variants are min, base, nfs, xen, big, std (see above in this page for more details). Puppet is in charge of configuring the environment operating system with regard to the chosen variant. All variants are defined as Puppet classes that include each others in the following order:

min ⊂ base ⊂ nfs ⊂ big ⊂ std

Additionnaly, the xen class just includes the base class, so that the xen environment is just the base environment with Xen hypervisor and tools added.

This means that for changes made in the min class will affect all other variants. Changes made in the big class will affect the builds of both the big and std variants.

In this example we will extend the min environment recipe of Debian 11. To do so, we use the kameleon new command as follows:

Terminal.png node ~/my_recipes:
kameleon new debian11_custom grid5000/debian11-x64-min.yaml

This creates the ~/my_recipes/debian11_custom.yaml file which is our new recipe. Besides, kameleon took care of importing in the directory all the files for the new recipe depends on.

You can list the recipes which are present in your workspace using the list command:

Terminal.png node ~/my_recipes:
kameleon list

You can see your new debian11_custom recipe along with the recipes that it extends directly or indirectly.

You can look at the steps involved in the build of the recipes using the kameleon dryrun command:

Terminal.png node ~/my_recipes:
kameleon dryrun debian11_custom

We see that the setup section has setup and run steps for an orchestrator: what's the part that prepares everything to use Puppet in the recipe and run it.

Since we want to write our customization with the Puppet language, we do not have to modify the debian11_custom.yaml recipe file much. We may just change the environment description by editing the recipe file ~/my_recipes/debian11_custom.yaml, and changing the description field starting at line 4, for instance:

# DESCRIPTION: My Grid'5000 Debian Bullseye

Once done, we can close the file and look at the Puppet code.

Puppet is a software configuration management tool that includes its own declarative language to describe system configuration. It is a model-driven solution that requires limited programming knowledge to use.

The puppets modules used by the Grid'5000 reference environments are located in ~/my_recipes/grid5000/steps/data/setup/puppet/modules/env/manifests/.

For our use case, we can look at the commonpackages.pp file. This is a really simple file that requests packages to be installed.

We can install ffmpeg like this :

class env::commonpackages{
class env::commonpackages::ffmpeg{
  package{ 'ffmpeg':
    ensure => installed;

In this quite simple use case, but if you have a package like postfix which requires more configuration, it could be more complex! Puppet covers a lot of needs that we cannot describe in this documentation. To know more, please refer to the Puppet documentation.

Creating an environment by extending a Grid'5000 recipe without using Puppet

In this section, we will describe the customization of a Grid'5000 environment template using only the kameleon language, not Puppet. Thus the environment template that we extend is not limited to Debian stable.

We have two possibilities here for the method, using different recipe templates:

  • Method A: either use the grid5000/from_grid5000_environment/base.yaml recipe template, which builds by starting from an Grid'5000 environment (to specified in the recipe),
  • Method B: or extend any of the recipes of the Grid'5000 environments.
Note.png Note

Comparison of the two methods:

With A, by extending from_grid5000_environment/base.yaml
the new environment is built by first importing in a VM an existing environment image (tarball) and run the system (bootstrap section), so that you can do your customizations (described in the setup section of the recipe). Finally, the customized system (in the VM) is exported to become your new environment (export section). This is somehow similar to what one does when using tgz-g5k, but in a traceable way with everything written in a recipe.
  • Pros:
    • New method
    • Simpler recipes
    • Quicker build (not building from scratch using a distribution installer, not using Puppet even for Debian stable)
  • Cons:
    • May hide too much complexity
    • Understanding the overall environment setup requires reading both the recipe of the modified environment and of the new environment
    • Does not generate qemu qcow2 VM images
With B, by extending the recipe of a Grid'5000 environment
the new environment is built from scratch.
  • Pros:
    • Enable to use Puppet with Debian stable recipes (see the previous section)
    • Build both the environment for use with kadeploy and the qemu qcows image for use in VMs
    • Build from scratch
  • Cons:
    • Legacy method
    • Build is longer typically because installing from scratch, running the distribution installer
    • Build is a lot longer if Puppet is involved (when extending a Debian stable recipes template)
    • more complex (exposed all steps to build from scratch)

We details in the next 2 paragraphs how to work for both methods.

Extending grid5000/from_grid5000_environment/base.yaml

Let's assume we choose to follow method A and extend grid5000/from_grid5000_environment/base.yaml: We run the following command:

Terminal.png node ~/my_recipes:
kameleon new my_custom_environment grid5000/from_grid5000_environment/base.yaml

With method A, the environment to base work on is not given by the name of the recipe template we extend: the grid5000/from_grid5000_environment/base.yaml template works with any environment as base. Thus the Grid'5000 environment to import the image tarbal from (and build on top of) has to be specified within the recipe. We edit the new recipe file: my_custom_environment.yaml, and look at the global section. The required information to provide here is the grid5000_environment_import_name: it must be set to one of the environment that show when running kaenv3 -l on a frontend. For instance we may choose to use debiantesting-x64-min. We do not specify user and version since the default ones are just fine by default.

## Environment to build from
grid5000_environment_import_name: "debiantesting-x64-min"
#grid5000_environment_import_user: "deploy"
#grid5000_environment_import_version: ""

Other lines of the global section can be modified as suggested by the comments in the recipe file itself. This is self-explanatory so we do not detail more here.

Once done, the important part is to bring your customization steps for the setup of your environment in the setup section. See below.

Extending a the recipe of a Grid'5000 environment

If we prefer building from scratch (following method B) rather that starting from the image (tarball) of an existing environment, we can extend directly the recipe of one of the Grid'5000 reference environment. Let's say we want to extend grid5000/debiantesting-x64-min. We run the following command:

Terminal.png node ~/my_recipes:
kameleon new my_custom_environment grid5000/debiantesting-x64-min

(compared to method A, we just change the template used).

This creates the ~/my_recipes/my_custom_environment.yaml file, which this time directly describes that it extends grid5000/debiantesting-x64-min.

We can now edit the recipe file and look at comments inside it, that are self-explanatory.

Customize the environment in the setup section

Whatever template we choose to extend (following method A or B), our customization of the environment operating system is to be written in the setup section (bootstrap and export should not require any changes as long as we work on customizing the environment for Grid'5000).

For example, just like in the previous section with Puppet, we describe below how to install the ffmpeg package (but in the kameleon language this time).

We add a new step to the recipe, which is just a sequence of actions to execute. This basically gives a setup section in our recipe as follows:

 - "@base"
 - install_more_packages:
    - install_ffmpeg:
      - exec_in : apt-get update && apt-get install -y ffmpeg
  • "@base" means that the steps from the environment we extend should be executed before our new steps (a bit like when using super() in a constructor of a class in Java or Ruby to call the constructor of the inherited class).
  • exec_in means that the command will be executed with bash in the VM of the build process. See the kameleon documation for other commands.
  • install_more_packages is a macrostep, install_ffmpeg is a microstep: It is mandatory to define the 2 level of steps and respect the format of a correct YAML document.

Additionnaly, a macrostep and its microsteps can also be defined in a separate file. For instance, we can create the steps/setup/ directory hierarchy and the steps/setup/install_more_packages.yaml file inside, with the following content:

- install_ffmpeg:
    - exec_in : apt-get update && apt-get install -y ffmpeg

Then use it in the recipe with just:

 - "@base"
 - install_more_packages

(no : after install_more_packages here).

You can find more information about the kameleon language in the kameleon documentation

To inpect our environment before launching the build:

  • We can look at the information about the environment with kameleon info:
Terminal.png node ~/my_recipes:
kameleon info my_custom_environment
  • We can look at what the build of the environment involves by running kameleon dryrun:
Terminal.png node ~/my_recipes:
kameleon dryrun my_custom_environment

If any error is raised with those commands, it probably come from a bad syntax in the recipe (e.g. bad YAML formating).

Build and test

Once the recipe is written, we can launch the build. To do so, we just have to run following command:

Terminal.png node ~/my_recipes:
kameleon build my_custom_environment
Warning.png Warning

Depending on different factors (e.g. the size of the image you are about to create (what variant), the hardware used (SSD or HDD)), the build process can last from a few minutes to a lot longer.

We end up with a build directory that contains the freshly build files we are interested in:

File build/my_custom_environment/my_custom_environment.dsc
  • This is the description file of the new environment (this is a YAML file, the file extension does not really matter, be it .dsc, .env or .yaml)
  • It can be used either directly with kadeploy to run the deployment without registering the environment:

After creating a new job of type deploy, we run the following kadeploy command from the frontend:

Terminal.png frontend:
kadeploy -a ~/my_recipe/build/my_custom_environment/my_custom_environment.dsc
  • Or to register the environment with kaenv3, for later use with kadeploy.
Terminal.png frontend:
kaenv3 -a ~/my_recipe/build/my_custom_environment/my_custom_environment.dsc
File build/my_custom_environment/my_custom_environment.tar.zst
  • This is the tarball of our new environment, referred to in the environment description (previously debian11_custom.tgz because the compression was formerly gzip)
  • It can also be used directly, for instance with docker import

After installing docker on a reserved node with g5k-setup-docker, run:

Terminal.png node:
zstdcat ~/my_recipe/build/my_custom_environment/my_custom_environment.tar.zst | docker import - debian11-x64-min

Then run the docker container, for instance:

Terminal.png node:
docker run -ti debian11-g5k bash
File build/my_custom_environment/my_custom_environment.qcow2
  • It is a qcow2 version of the environment for use with qemu (as seen earlier, it is not built when extending grid5000/from_grid5000_environment/base).

Just run Qemu on the image:

Terminal.png node:
qemu-system-x86_64 -enable-kvm -m 2048 -cpu host ~/my_recipe/build/my_custom_environment/my_custom_environment.qcow2

Of course, writing recipes, building the environment, and testing it may be a trial and error process requiring to loop over the different stages.

Please note that kameleon provides an interactive debugging of the recipe in case of errors or when breakpoints are inserted in the recipe. See comments in the recipes which give the syntax to add a breakpoint.

Creating an environment for an unsupported Operating System

Note.png Note

This requires strong system administrator skills and a very good understanding of how the Grid'5000 bare metal deployment functions.

If an Operating System is not provided as a Grid'5000 environment already, please mind that kameleon recipe allows installation from scratch of any Linux systems:

  • boot the OS installer in a VM and do an unattended installation
  • run some additional setup
  • finally export the built system ready to be consumed by kadeploy3.

Therefore please mind using kameleon. Please get in touch with the Grid'50000 technical team to explain your motivation and ideas and possibly to get some help.