Environments creation using Kameleon and Puppet

From Grid5000
Jump to: navigation, search
Note.png Note

This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

This page summarize what you need to know to create environments used on Grid'5000 using Kameleon and Puppet.

Rebuild a standard Grid'5000 environment

Overview

This section describe the step to reproduce a vanilla Grid'5000 environment using Kameleon. Kameleon is software appliance builder: It means that it is made to create a complete operating system image from a recipe. Recipes are a set of YAML files that contains a list of steps that are executed to bootstrap, setup and export the image it creates. In order to help users to share it provide a template mechanism that you can extends with you own steps. Kameleon has a lot of other features like: Context isolation, caching of the recipe artefacts, interactive breakpoints and so on.

For more details about Kameleon see the official documentation: http://kameleon.imag.fr/

Two major tools are used to prepare environments used in Grid'5000 Kameleon and Puppet:

  • Kameleon to create a virtual machine (VM) for the build, install the minimal OS from the installation ISO image, then call Puppet inside it, and finally export its content as a kadeploy environment and qcow2 VM image (other format are possible).
  • Puppet to configure environments (install packages, configuration files, services...).
Note.png Note

As puppet is used internally you don't have to use it for your own recipe if you do not want to. Both methods are proposed here with or without puppet

Install Kameleon and other dependencies

To install Kameleon, follow instructions from the Kameleon website.

Note.png Note

Kameleon needs a recent version of libguestfish and ruby. This tutorial is known to work since Debian 9 Stretch.

To build Grid'5000 environments, we need to install a few additional packages and then the kameleon-builder gem:

Terminal.png localhost:
apt-get update && apt-get install --no-install-recommends git linux-headers-amd64 socat qemu-utils ruby-dev ruby-childprocess polipo pigz netcat eatmydata libguestfs-tools dirmngr python-future gnupg gnupg-agent
Terminal.png localhost:
gem install --no-ri --no-rdoc kameleon-builder
Warning.png Warning

The mdadm tool, installed as a libguestfs-tools dependency, may ask you which RAID array it should manage. If you don't know what to answer, leave the field blank.

All remaining commands of this tutorial may be run as a standard, non-root, user.

Prepare the build environment with a script

On a Grid'5000 deployed node with lot of RAM (>32Go) run :

#/bin/bash
set -e
set -x
echo "libguestfs0     libguestfs/update-appliance     boolean false" | debconf-set-selections
echo "mdadm   mdadm/initrdstart       string  none" | debconf-set-selections
KERNEL=linux-image-$(uname -r)
VERSION=$(apt-cache policy $KERNEL | grep Installed: | awk '{print $2}')
apt-get update
apt-get install -y systemtap linux-image-$(uname -r)-dbg=$VERSION linux-headers-$(uname -r)=$VERSION
/tmp/environments-recipes/tools/nofsync.stp </dev/null >/dev/null 2>&1 &
apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends git linux-headers-amd64 socat qemu-utils ruby-dev ruby-childprocess polipo pigz netcat eatmydata libguestfs-tools dirmngr python-future gnupg gnupg-agent
gem install --no-ri --no-rdoc kameleon-builder
mv /bin/gzip /bin/gzip.OLD
ln -s /usr/bin/pigz /bin/gzip
cd /tmp

It installs all dependencies and replaces gzip by pigz which is multithreaded implementation of gzip, so the tar.gz compression is decreases proportionally to the number of core of the node.

Get environments recipes

  • Get recipes from Github:
Terminal.png localhost:

You can then list the available templates with:

Terminal.png localhost:
kameleon template list

Build an image

We will create a brand new workspace directory to store builds' data:

Terminal.png localhost:
mkdir ~/my_recipes && cd ~/my_recipes


Pick the right recipe

Kameleon provides some recipes, and we added some others specialized for Grid'5000, their name end by -{variant_listed_below}. We will focus on Grid'5000 recipes here.

Grid'5000 images exist in different variants. As explained earlier, puppet is in charge of configuring images. Choosing a variant is in fact equivalent to picking a set of puppet recipes. The managed variants are: Min, Base, Nfs, Xen, Big, Std All recipes are included in each others, in the following order:

Min ⊂ Base ⊂ Nfs ⊂ Big ⊂ Std

This means that changes made in min recipes will affect all other recipes. Changes made in big recipes will affect big and std recipes.

'Xen' image is a special case: It is a Base image with xen kernel added.

See Getting_Started#Deploying_nodes_with_Kadeploy for more information on differences between these variants. Please note that Kameleon's build logs of Grid'5000 images are available under /grid5000/descriptions/log on frontends.

In this example we will create a Min image.

Instantiate a template

We instantiate a template in the current directory. Kameleon will take care of copying all files required to build the template for you.

Terminal.png localhost ~/my_recipes:
kameleon new debian10_custom grid5000/debian10-x64-min.yaml

To create images, Kameleon may use several back-ends (from_upstream, from_scratch,...). In Grid'5000, it uses recipes based on from_scratch.

You can list the recipes present in your workspace using the list command:

Terminal.png localhost ~/my_recipes:
kameleon list

You can see your new debian10_custom recipe along with the templates that it extends. But the description field is <MY RECIPE DESCRIPTION>. You can edit your recipe (~/my_recipes/debian10_custom.yaml) and change the description in the commented header, for example:

#==============================================================================
#
# DESCRIPTION: My Grid'5000 Debian Buster 
#
#==============================================================================

Customise your image

Maybe you have an experiment which requires always the same packages, and these packages aren't in min, are in std but std environnement is too big for you ? Kameleon allows to add some packages or whatever you want in your image really simply.

There are two ways to add a package to your image.

Add a new step

Note.png Note

This part is quick explanation to add packages to your image, you can find more details below in Environments_creation_using_Kameleon_and_Puppet#Modifying_an_image part.

The first way is add a new step to Kameleon recipe. A step is just a sequence of actions to execute described in a yaml file.

For example we will install ffmpeg.

1. Create a new file steps/setup/my_install.yaml

2. Fill this file with :

- install_ffmpeg:
    - exec_in : apt-get update && apt-get install -y ffmpeg

exec_in means the command will be executed with bash in the VM.

You can also execute multi-line script :

- install_ffmpeg: 
    - exec_in : |
         apt-get update
         apt-get install -y ffmpeg

3. Edit your recipe (~/my_recipes/debian10_custom.yaml)

...
- setup : 
   - "@base"
   - my_install

You can find more information on the official documentation

Edit puppet recipe

Grid'5000 images use puppet, a tool for configuration management to configure and install packages. Be careful, don't get the Kameleon recipe and puppet recipe mixed-up. Kameleon recipe is used by kameleon on your computer, puppet recipe is used by puppet in the VM which was created by kameleon.

So, the puppets recipes are in ~/my_recipes/grid5000/steps/data/setup/puppet/modules/env/manifests/. In our case, the file commonpackages.pp is interesting. If you open it, you can see a really simple file which ask the packages declared to be installed.

We can install ffmpeg like this :

class env::commonpackages{
}
...
class env::commonpackages::ffmpeg{
  package{ 'ffmpeg':
    ensure => installed;
  }
}
...

In this simple case Puppet or new Step are equivalent, but if you have a package like postfix which requires more configuration, puppet should be better. Puppet cover a lot of needs and we can't see all that puppet can do here. If you want to know more about it refer to the documentation.

Launch the build

Run following command:

Terminal.png localhost ~/my_recipes:
kameleon build debian10_custom.yaml --enable-cache
Note.png Note

If you have enough RAM (depending of the image but >=16Go is generally fine) think about mounting the "build" directory as tmpfs. This should decrease the build time by 2 or more

Warning.png Warning

Depending of different factors, like the size of the image you are about to create (variant), the network quality, using or not a cache, hardware used (ssd?) ... Build can last a few minutes (6 min on my really expensive laptop :) ) or a few hours.

Warning.png Warning

If the build failed during the execution of echo -e "run\nzerofree /dev/sda1" I guestfish' [...], your version of libguestfs is too old.

Warning.png Warning

Running Grid'5000 VPN may be required to build some recipes which need access to Grid'5000 APT repository (g5k_* deb metapackage).

You'll end up with a build folder that contains files your are interested in:

  • build/debian10_custom/debian10_custom.tgz and build/debian10_custom/debian10_custom.dsc, to deploy your image on Grid'5000 using Kadeploy
  • build/debian10_custom/debian10_custom.qcow2 is a qcow2 version of the environment
  • build/debian10_custom/debian10_custom-cache.tar.gz is a cache that contains all files download by Kameleon during the build. It allows a quicker rebuild of the environment, even without network access. Rebuilding using cache should be possible no matter when you try to rebuild the image (and even if remote repositories / package servers changed).

Run your image

Now that you have built your first image let's run it.

In Grid'5000 with Kadeploy

Warning.png Warning

build/debian10_custom/debian10_custom.dsc must be adapted, in particular the path to the tgz file. It can be specified manually in the file, or with the Kameleon option "--global=g5k_tgz_path:<path>", or inside the recipe (see Customize variables example)

You have to put your recipe and your description file in your Grid'5000 home on the right place. For example:

Terminal.png localhost:
ssh username@access.grid5000.fr "mkdir ~/site/my_g5k_images"
Terminal.png localhost:
rsync -aAXP --bwlimit=5000 build/debian10_custom/debian10_custom.tgz build/debian10_custom/debian10_custom.dsc username@access.grid5000.fr:~/site/my_g5k_images/
Note.png Note

If you configured your SSH client as described in the SSH tutorial for Grid'5000, you should be able to connect to access.grid5000.fr by writing only "g5k" instead of "username@access.grid5000.fr"

Note.png Note

If the rsync command takes too much time and if you are currently on one of the Grid'5000 sites, you can directly rsync on the frontend of this site, which should be much quicker:

Terminal.png localhost:
rsync -aAXP --bwlimit=5000 build/debian10_custom/debian10_custom.tgz build/debian10_custom/debian10_custom.dsc username@access.site.grid5000.fr:~/my_g5k_images/


If you didn't do it before, customize your image description (see Advanced Kadeploy for details)

Terminal.png frontend:
vim my_g5k_images/debian10_custom.dsc

Then register the description file with kaenv on the frontend:

Terminal.png frontend:
kaenv3 -a my_g5k_images/debian10_custom.dsc

And ask OAR for some nodes:

Terminal.png frontend:
oarsub -I -t deploy -l nodes=2,walltime=1

Finally launch it with kadeploy:

Terminal.png frontend:
kadeploy3 -f $OAR_NODEFILE -e debian10_custom

With Qemu

Note.png Note

This image is not contextualized by the kadeploy post install script so it can be a little miss-configured.

Just run Qemu on the image:

Terminal.png localhost:
qemu-system-x86_64 -enable-kvm -m 2048 -cpu host build/debian10_custom/debian10_custom.qcow2

With Docker

First you have to import your image in Docker. Simply use import with a tag to name your recipe:

Terminal.png localhost:
docker import build/debian10_custom/debian10_custom.tgz debian10-g5k

Then run the image with bash for example:

Terminal.png localhost:
docker run -ti debian10-g5k bash

Modifying a environment

Now, we will consider you have a few personal customization to make to your environment. We use kameleon to script them.

To get more information on the kameleon recipe writing, please have a look at the Kameleon web page:

Extend a Grid'5000 environment using Kameleon

This is the quickest method because it consists in just importing an existing environment provided by the Grid'5000 tech team, and adding some additional software on top of it before repacking it.

It does not involve rebuilding an environment from scratch as described in the next paragraphs, which notably requires for the Debian10 environments (-min, -base, -nfs, -std, -big, -xen) to run the whole puppet machinery.

The steps to build a custom environment are the following:

Get a job on a Grid'5000 node
Terminal.png frontend:
oarsub -I

Then on the node...

Prepare kameleon
  • Make sure to have kameleon installed:
Terminal.png node:
gem install --user kameleon-builder
  • Make sure to have ruby in your PATH (add to ~/.bashrc):
export PATH=$HOME/.gem/ruby/2.5.0/bin:$PATH
  • Make sure to have the Grid'5000 environment recipes repository available:
Terminal.png node:
kameleon repository list
grid5000 <- https://gitlab.inria.fr/grid5000/environments-recipes.git (master)
  • If not, we add it:
Terminal.png node:
Create the recipe

First we create our workspace

Terminal.png node:
mkdir kameleon-workspace && cd kameleon-workspace

And set the kameleon build directory in the tmp partition so that is it on a local drive, and make sure there is enough space (images are big).

Terminal.png node:
mkdir $XDG_RUNTIME_DIR/kameleon-build && ln -sf $XDG_RUNTIME_DIR/kameleon-build build

We now create a fresh new recipe, that we name my-debian for instance, from the from_grid5000_environment/debian10-x64-custom recipe template.

Terminal.png node:
kameleon new my-debian grid5000/from_grid5000_environment/debian10-x64-custom
Edit and adapt the recipe

We open the recipe file my-debian.yaml in an editor, and adapte it as suggested in the commented lines:

 1 #==============================================================================
 2 # vim: softtabstop=2 shiftwidth=2 expandtab fenc=utf-8 cc=81 tw=80
 3 #==============================================================================
 4 #
 5 # DESCRIPTION: Customized Grid'5000 environment build on top of an existing
 6 #              environment
 7 #
 8 #==============================================================================
 9 # This recipe extends another. To look at the step involed, run:
10 #   kameleon dryrun my-debian.yaml
11 # To see the variables that you can override, use the following command:
12 #   kameleon info my-debian.yaml
13 ---
14 extend: grid5000/from_grid5000_environment/debian10-x64-custom.yaml
15 
16 global:
17   ### Grid'5000 environment information
18   ## (Uncomment and change any line if needed)
19 
20   ## Frontend to run kaenv3 on
21   #grid5000_frontend: "frontend"
22 
23   ## Site used in the build
24   #grid5000_site: "grenoble"
25 
26   ## Environment to build from
27   #grid5000_environment_import_name: "debian10-x64-min"
28   #grid5000_environment_import_user: "deploy"
29   #grid5000_environment_import_version: ""
30 
31   ## New environment description
32   #grid5000_environment_export_name: "$${kameleon_recipe_name}"
33   #grid5000_environment_export_format: "tar.gz"
34   #grid5000_environment_export_description: "Customized $${grid5000_environment_import_name}"
35 
36   ## Set where to store the environment and the assiated kadeploy URL base
37   #grid5000_environment_export_dir: "$HOME/public/"
38   #grid5000_environment_export_baseurl: "local://$HOME/public/"
39 
40   ## Optionaly, the environment postinstall script can be changed, e.g. to
41   ## enable NFS homes, LDAP accounts, if not enabled in the imported env.
42   #grid5000_environment_export_postinstall_script: "g5k-postinstall --net debian --fstab nfs --restrict-user current"
43 
44   ## Optionaly, an additional postinstall can be given, e.g. to do some custom
45   ## operations. Use the following variables to set the archive name and script.
46   #grid5000_environment_export_additional_postinstall_archive: "$${kameleon_recipe_name}-additional-postinstall.tar.gz"
47   #grid5000_environment_export_additional_postinstall_script: "additional_postinstall.sh"
48   ## The recipe will have to prepare the additional postinstall content in a
49   ## directory to create in the local context and name "additional_postinstall"
50   ## by default (next line to change it). The archive is created by the export.
51   #grid5000_environment_export_additional_postinstall_dir: "additional_postinstall"
52 
53   ### You can add below any other global variable definition
54   ## See the variables which can be overloaded, by running:
55   ##   kameleon info my-debian.yaml
56   ## Or define any new variable you would need. e.g.:
57   #my_variable: my_value
58 
59 bootstrap:
60   ### The bootstrap section takes in charge the import of the Grid'5000
61   ## environment to customize. No modification should be needed here.
62   - "@base"
63 
64 setup:
65   ### The setup section is where to place your customization. Add all steps
66   ## required by your customization.
67   ## The following is given as example only, replace with your steps.
68   - a_customization_step:
69     - microstep1:
70       - exec_in: echo "Hello world!"
71     - microstep1:
72       # This breakpoint will stop the build for inspecting the environment
73       - breakpoint
74 
75 export:
76   ### The export section takes in charge the export of your customized Grid'5000
77   ## environment. No modification should be needed here.
78   - "@base"
  • You must at least set the grid5000_environment_import_name variable in line 27, for instance with the value debian10-x64-big if you would like to base your customized environment on top of the debian10-x64-big environment provided by the Grid'5000 staff.
  • Then add steps in the setup section for your needs.
Note.png Note

Unless you know what you are doing, you should only have to modify the global and setup sections to customize your environment.

We can now start the build !
Terminal.png node:
kameleon build my-debian

Rebuild a Grid'5000 environment with customizations

In this section, we build a customized environment by building it from scratch, which means the build process will begin by running the operating system standard installer (CDROM image). This takes over the exact same build process of the Grid'5000 supported environment (-min-big), to which you will just add some extra steps.

For your customization, you have 2 options:

  • The easiest is to modify the kameleon recipe to insert custom steps.
  • The alternative if you know and prefer puppet, is to add or modify the Puppet classes involved within the build.
Note.png Note

Grid'5000's way of configuring complex environments is by using Puppet. Grid'5000 supported environments, such as debian10-x64-big, use first kameleon to setup a minimal system (in a VM), then kameleon lets the hard work of configuration to Puppet. Then kameleon takes back the hand to export the build system it to various formats (kadeploy environment, VM qcow2 file). Note that this is different for distributions which Grid'5000 only provides -min environment for: no puppet is involved in that case.

First example: Customize variables

In this example, we will customize the default template using the Kameleon variables.

We will use the debian10_custom.yaml recipe that we created in the previous section.

Kameleon use variables prefixed by $$ like $$my_variable. Among other things, the info command allows you to see all the defined variables in your recipe:

Terminal.png localhost:
kameleon info debian10_custom.yaml

In debian10_custom.yaml file, modify Kameleon variables by adding a new global variable in the 'global' part. For instance, we will customize the g5k_tgz_path so kadeploy will know where to find our new image:

global:
  g5k_tgz_path: /home/<YOUR_G5K_USER>/my_g5k_images/debian10_custom.tgz

To build the new image, use (or you may go directly to next section):

Terminal.png localhost:
kameleon build debian10_custom.yaml --enable-cache

Second example: Install the NAS Benchmarks

The NAS benchmarks are commonly used to benchmark HPC application using MPI or OpenMP. In this example we will download and configure the NAS package and build the MPI FT benchmark.

To do so we will create a step file that will called from the recipe. First let's create the ./steps in our kameleon workspace (~/my_recipes in this tutorial).

Terminal.png localhost:
mkdir steps
Note.png Note

It is a good practice to add your steps in this folder rather

than inside the imported templates steps: it is easier to

know what is my work and what is coming from the template.

Define the content of that step in steps/NAS_benchmark.yaml. You can notice that a Kameleon variable is used to defined the NAS_Home.

- NAS_home: /root
- install_NAS_bench:
  # install dependencies
  - exec_in: apt-get -y install openmpi-bin libopenmpi-dev make gfortran gcc
  - download_file_in:
    - https://www.nas.nasa.gov/assets/npb/NPB3.3.1.tar.gz
    - $$NAS_home/NPB3.3.1.tar.gz
  - exec_in: cd $$NAS_home && tar xf NPB3.3.1.tar.gz
- configure_make_def:
  - exec_in: |
      cd $$NAS_home/NPB3.3.1/NPB3.3-MPI/
      cp config/make.def{.template,}
      sed -i 's/^MPIF77.*/MPIF77 = mpif77/' config/make.def
      sed -i 's/^MPICC.*/MPICC = mpicc/' config/make.def
      sed -i 's/^FFLAGS.*/FFLAGS  = -O -mcmodel=medium/' config/make.def
- compile_different_MPI_bench:
  - exec_in: |
      cd $$NAS_home/NPB3.3.1/NPB3.3-MPI/
      for nbproc in 1 2 4 8 16 32
      do
        for class in B C D
        do
          for bench in is lu ft
          do
            # Not all IS bench are compiling but we get 48 working
            make -j 4 $bench NPROCS=$nbproc CLASS=$class || true
          done
        done
      done


To add this step to your debian10_custom.yaml recipe, modify Kameleon build's process by adding a new step NAS_benchmark (named after the file name) to the 'setup' part, in addition to inherited steps from parent recipe (@base). It is possible to override the NAS_Home variable in the recipe. Here we chose to put it in /root/workdir for example, but first we use the inline step declaration to create the folder inside the image.

setup:
  - "@base"
  - create_my_working_directory:
    - create_the_folder:
      - exec_in: mkdir /root/workdir
  - NAS_benchmark:
    - NAS_Home: /root/workdir

To build the new image, use (or you may go directly to next section):

Terminal.png localhost:
kameleon build debian10_custom.yaml --enable-cache

Third example: Add a file

Let's add a file to your image. You can access the steps/data folder inside Kameleon recipes using the $$kameleon_data_dir variable.

In this example, we will add a script that clears logs to in your image.

First, write a step that copies a script and executes it. This step must be located at steps/clean_logs.yaml:

- script_path: /usr/local/sbin
- import_script:
  - local2in:
    - $$kameleon_data_dir/$$script_file_name
    - $$script_path/$$script_file_name
  - exec_in: chmod u+x $$script_path/$$script_file_name
- run_script:
  - exec_in: $$script_path/$$script_file_name
Note.png Note

In this step we are using the alias command local2in provided by Kameleon. See documentation of commands and alias for more details.

Here is an example of a cleaning script that must be copied in steps/data/debian_log_cleaner.sh.

#!/bin/sh
# This is my cleaning script 'cause I don't trust G5K
systemctl stop rsyslog
rm -rf /var/log/*.log*
rm -f /root/.bash_history
Note.png Note

Script content does not really matter, it is an example. Of course you can run these commands directly inside the recipe

Finally, we call that step by modifying the setup section of the recipe (debian10_custom.yaml). We set the variables script_file_name to select the script in the data folder.

  - clean_logs
    - script_file_name: debian_log_cleaner.sh

To build the new image, use (or you may go directly to next section):

Terminal.png localhost:
kameleon build debian10_custom.yaml --enable-cache

Fourth example: Modify export format

Suppose now that you want to export your image to VDI (as used by virtualbox) in addition to the qcow2 and tar.gz format. To do so, we will replace the global section of our debian10_custom.yaml recipe. If we look a the parent recipe's export section in file grid5000/debian10-x64-common.yaml), we see a appliance_formats

In debian10_custom.yaml file we then put:

global:
   appliance_formats: qcow2 tar.gz
Note.png Note

Allowed formats are : tar.gz, tar.bz2, tar.xz, tar.lzo, qcow, qcow2, qed, vdi, raw, vmdk

To build the new image, use:

Terminal.png localhost:
kameleon build debian10_custom.yaml --enable-cache

Modifying an image using Puppet

In Grid'5000 environments building process, Puppet is called by Kameleon after it creates a minimal operating system. Puppet recipes can be found in grid5000/steps/data/setup/puppet/.


Add a package to base variant

In this example, we will re-generate a base variant adding packages iperf and emacs.

If you don't want to mess with standard recipe, you should work in a new workspace:

Terminal.png localhost:
mkdir modified_recipes_puppet && cd modified_recipes_puppet

Then, instantiate a debian10-x64-base template:

Terminal.png localhost:
kameleon new my-debian10-x64-base grid5000/debian10-x64-base.yaml

I add definitions to grid5000/steps/data/setup/puppet/modules/env/manifests/commonpackages.pp

class env::commonpackages{
}
... 
class env::commonpackages::iperf{
  package{ 'iperf':
    ensure => installed;
  }
}
class env::commonpackages::emacs{
  package{ 'emacs':
    ensure => installed;
  }
}
...

To build the new image, use (or you may go directly to next section):

Terminal.png localhost:
kameleon build my-debian10-x64-base.yaml --enable-cache

Create a new variant

Previous section have shown how modifying base image. This may be suitable for minor modifications, but for bigger ones, it is recommended to create a new variant. Having your own variant will allow keeping your own set of customization separated from Grid'5000 recipes, which ease their maintenance (for example if Grid'5000 recipes are updated).

In this example situation, we consider that you need to install apache2, on your nodes and want to integrate it in your image. You have to create a linux user (www-data), add an apache2 configuration file, add your application (here, it's a simple html file), and ensure the service apache2 is running and enabled (starts at boot time). We consider the environment you work with usually is base. We will then extend a base environment with modifications listed before.

Note.png Note

This part is an example of Puppet usage. If you want to know more about Puppet, have a look at Puppet documentation


First, declare a new Kameleon template to build our debian10-x64-webserv image, based on debian10-x64-common template, the Kameleon template which all Grid'5000 image inherits from:

Terminal.png localhost:
kameleon new debian10-x64-webserv grid5000/debian10-x64-common.yaml

Create a new Puppet module apache2:

Terminal.png localhost:
mkdir grid5000/steps/data/setup/puppet/modules/apache2
Terminal.png localhost:
mkdir grid5000/steps/data/setup/puppet/modules/apache2/manifests
Terminal.png localhost:
mkdir grid5000/steps/data/setup/puppet/modules/apache2/files

Here is an example of content for grid5000/steps/data/setup/puppet/modules/apache2/manifests/init.pp:

# Module apache2

class apache2 ( ) {

  package {
    "apache2":
      ensure  => installed;
  }
  user {
    "www-data":
      ensure   => present;
  }
  file {
    "/var/www/my_application":
      ensure   => directory,
      owner    => www-data,
      group    => www-data,
      mode     => '0644';
    "/var/www/my_application/index.html":
      ensure   => file,
      owner    => www-data,
      group    => www-data,
      mode     => '0644',
      source   => 'puppet:///modules/apache2/index.html',
      require  => File['/var/www/my_application'];
    "/etc/apache2/sites-available/my_application.conf":
      ensure   => file,
      owner    => root,
      group    => root,
      mode     => '0644',
      source   => 'puppet:///modules/apache2/my_application.conf',
      require  => Package['apache2'];
    "/etc/apache2/sites-enabled/my_application.conf":
      ensure   => link,
      target   => '../sites-available/my_application.conf',
      require  => Package['apache2'],
      notify   => Service['apache2'];
  }
  service {
    "apache2":
      ensure   => running,
      enable   => true,
      require  => Package['apache2'];
  }
}

Files my_application.conf and index.html must be stored in grid5000/steps/data/setup/puppet/modules/apache2/files/

grid5000/steps/data/setup/puppet/modules/apache2/files/my_application.conf:

<VirtualHost *:80>

    ServerName my_application

    DocumentRoot /var/www/my_application

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog /var/log/apache2/access.log combined

</VirtualHost>

grid5000/steps/data/setup/puppet/modules/apache2/files/index.html:

<html>
    <head>
        <title>Hello World!</title>
    </head>
    <body>
        <P>I &#60;3 Grid'5000!</P>
    </body>
</html>


We will now integrate this module in a new variant called webserv that extends base variant.

First we must create a file webserv.pp here : grid5000/steps/data/setup/puppet/modules/env/manifests/webserv.pp:

# This file contains the apache2 class used to configure a user environment based on base variant, that contains apache2.

class env::webserv ( ) {

  class { "env::base": } # we include base variant here without overloading any of it's default parameters
  class { "apache2": }
}

To have it included by the actual Puppet setup, we must also create grid5000/steps/data/setup/puppet/manifests/webserv.pp:

# User env containing apache2
# All recipes are stored in env module. Here called with webserv variant parameter.

class { 'env':
  given_variant    => 'webserv';
}

And finally, modify grid5000/steps/data/setup/puppet/modules/env/manifests/init.pp to include your variant:

 case $variant {
   'min' :  { include env::min }
   'base':  { include env::base }
   'webserv': { include env::webserv }
   'nfs' :  { include env::nfs }
   'prod':  { include env::prod }
   'big' :  { include env::big }
   'xen' :  { include env::xen }
   default: { notify {"flavor $variant is not implemented":}}
 }


image, based on debian10-x64-commo, template, the Kameleon template which all Grid'5000 image inherits:
Terminal.png localhost:
kameleon new debian10-x64-webserv grid5000/debian10-x64-common.yaml

Then, tell Kameleon debian10-x64-webserv template to build our the webserv variant by modifying global section of recipe debian10-x64-webserv.yaml:

---
extend: grid5000/debian10-x64-common.yaml

global:
    # You can see the base template `grid5000/debian10-x64-common.yaml` to know the
    # variables that you can override
  variant: webserv

bootstrap:
  - "@base"

setup:
  - "@base"

export:
  - "@base"


Finally, you can launch a build :

Terminal.png localhost:
kameleon build debian10-x64-webserv.yaml --enable-cache