Custom-kernel-environment

In this tutorial, we describe how to create a kameleon recipe to build a kadeploy environment that embeds a customized Linux kernel.

About Kameleon

Quoting [1]: Thanks to Kameleon, one can write recipes that describe how to create, step by step, customized operating systems in any desired target format, and then cook them (build them), just like GNU make cooks sources using a Makefile to build binary programs.

Read more on the Environment_creation page.

Setup the workspace

Prepare the kameleon tool

First, get a job on a node

While the kameleon tool can be installed and run anyware, the easiest is to run it on a Grid'5000 node where is it already installed. This may avoid dependency issues.

pneyron@fgrenoble:~$ oarsub -I -l walltime=2
# Filtering out exotic resources (yeti, troll, servan, drac).
OAR_JOB_ID=2514218
# Interactive mode: waiting...
# Starting...
pneyron@dahu-12:~$

Then, add Grid'5000's environment recipes as a kameleon templates repository

If not already added, do as follows:

pneyron@dahu-12:~$ kameleon repo add grid5000 https://gitlab.inria.fr/grid5000/environments-recipes
Cloning into '/home/pneyron/.kameleon.d/repos/grid5000'...
warning: redirecting to https://gitlab.inria.fr/grid5000/environments-recipes.git/
remote: Enumerating objects: 23035, done.
remote: Counting objects: 100% (2467/2467), done.
remote: Compressing objects: 100% (434/434), done.
remote: Total 23035 (delta 2004), reused 2431 (delta 1987), pack-reused 20568
Receiving objects: 100% (23035/23035), 2.83 MiB | 21.79 MiB/s, done.
Resolving deltas: 100% (14477/14477), done.

Else, you may just update it:

pneyron@dahu-12:~$ kameleon repo update grid5000
...

We can run kameleon template list to list the available templates.

Create your environment recipe

We create a debiantesting-custom-kernel recipe, which will build on top of the debiantesting-min environment of Grid'5000.

While we could extend the debiantesting-x64-min recipe, which would imply building our environment from scratch (takes more time), we prefer using the from_grid5000_environment/base recipe, which will let kameleon extract the already built debiantesting-x64-min environment image tarball.

Create the debian12-custom-kernel recipe, which extends from_grid5000_environment/base.yaml

pneyron@dahu-12:~$ mkdir debiantesting-custom-kernel
pneyron@dahu-12:~$ cd debiantesting-custom-kernel
pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon new debiantesting-custom-kernel grid5000/from_grid5000_environment/base
      create  grid5000/from_grid5000_environment/base.yaml
      create  grid5000/steps/backend/qemu.yaml
      create  grid5000/steps/backend/VM.yaml
      create  grid5000/steps/backend/chroot.yaml
      create  grid5000/steps/aliases/defaults.yaml
      create  grid5000/steps/checkpoints/qemu.yaml
      create  grid5000/steps/bootstrap/prepare_ssh_to_out_context.yaml
      create  grid5000/steps/bootstrap/download_upstream_tarball.yaml
      create  grid5000/steps/bootstrap/create_appliance.yaml
      create  grid5000/steps/bootstrap/prepare_appliance.yaml
      create  grid5000/steps/bootstrap/start_qemu.yaml
      create  grid5000/steps/disable_checkpoint.yaml
      create  grid5000/steps/export/save_appliance_VM.yaml
      create  grid5000/steps/export/export_modified_g5k_env.yaml
      create  grid5000/steps/data/helpers/create_appliance.py
      create  grid5000/steps/data/qemu-sendkeys.rb
      create  grid5000/steps/data/helpers/export_appliance.py
      create  grid5000/steps/data/helpers/kaenv-customize.py
      create  grid5000/steps/env/bashrc
      create  grid5000/steps/env/functions.sh
      create  debiantesting-custom-kernel.yaml
pneyron@dahu-12:~/debiantesting-custom-kernel$

The command imports not only the grid5000/from_grid5000_environment/base.yaml recipe that we will extend, but also all its dependencies.

	Note
	When taking over a previous work, we may want to update the template files, which may have updates. We do so using the kameleon repo update grid5000 && kameleon template import grid5000/from_grid5000_environment/base.

Initiate the git repository for the work

At this stage, it's a good idea to start versioning our work in Git. We init the git repository and add the already created files, then commit.

pneyron@dahu-12:~/debiantesting-custom-kernel$ git init
...
Initialized empty Git repository in /home/pneyron/debiantesting-custom-kernel/.git/
pneyron@dahu-12:~/debiantesting-custom-kernel$ git add debiantesting-custom-kernel.yaml
pneyron@dahu-12:~/debiantesting-custom-kernel$ git add grid5000/
pneyron@dahu-12:~/debiantesting-custom-kernel$ git commit -m "First commit after running 'kameleon new ...'"
[master (root-commit) 5eec2d1] First commit after running 'kameleon new ...'
 21 files changed, 2267 insertions(+)
 create mode 100644 debiantesting-custom-kernel.yaml
 create mode 100644 grid5000/from_grid5000_environment/base.yaml
 create mode 100644 grid5000/steps/aliases/defaults.yaml
 create mode 100644 grid5000/steps/backend/VM.yaml
 create mode 100644 grid5000/steps/backend/chroot.yaml
 create mode 100644 grid5000/steps/backend/qemu.yaml
 create mode 100644 grid5000/steps/bootstrap/create_appliance.yaml
 create mode 100644 grid5000/steps/bootstrap/download_upstream_tarball.yaml
 create mode 100644 grid5000/steps/bootstrap/prepare_appliance.yaml
 create mode 100644 grid5000/steps/bootstrap/prepare_ssh_to_out_context.yaml
 create mode 100644 grid5000/steps/bootstrap/start_qemu.yaml
 create mode 100644 grid5000/steps/checkpoints/qemu.yaml
 create mode 100755 grid5000/steps/data/helpers/create_appliance.py
 create mode 100755 grid5000/steps/data/helpers/export_appliance.py
 create mode 100755 grid5000/steps/data/helpers/kaenv-customize.py
 create mode 100755 grid5000/steps/data/qemu-sendkeys.rb
 create mode 100644 grid5000/steps/disable_checkpoint.yaml
 create mode 100644 grid5000/steps/env/bashrc
 create mode 100644 grid5000/steps/env/functions.sh
 create mode 100644 grid5000/steps/export/export_modified_g5k_env.yaml
 create mode 100644 grid5000/steps/export/save_appliance_VM.yaml
pneyron@dahu-12:~/debiantesting-custom-kernel$

We may already push the git repository to a git remote, e.g., a Gitlab project, but this is out of the scope of this tutorial. Please mind doing it whenever you want, e.g., to share your work with others.

Set the upstream environment information

At first, we just have to set the upstream environment we want to import the image tarball from. We edit the grid5000_environment_import_name kameleon global variable in the recipe file: debiantesting-custom-kernel.yaml, as shown by a git diff afterward.

pneyron@dahu-12:~/debiantesting-custom-kernel$ git diff
diff --git a/debiantesting-custom-kernel.yaml b/debiantesting-custom-kernel.yaml
index 3cf22ac..b5805b8 100644
--- a/debiantesting-custom-kernel.yaml
+++ b/debiantesting-custom-kernel.yaml
@@ -27,7 +27,7 @@ global:
   #grid5000_site: "grenoble"
 
   ## Environment to build from
-  #grid5000_environment_import_name: "debian11-min"
+  grid5000_environment_import_name: "debiantesting-min"
   #grid5000_environment_import_user: "deploy"
   #grid5000_environment_import_version: ""
   #grid5000_environment_import_arch: "x86_64"

Then we commit the change:

pneyron@dahu-12:~/debiantesting-custom-kernel$ git commit -m 'Set grid5000_environment_import_name: "debiantesting-min"' -a
[master 1a3f1d8] Set grid5000_environment_import_name: "debiantesting-min"
 1 file changed, 1 insertion(+), 1 deletion(-)

Try and guess a first build

First we can run a dryrun build of the recipe to check that the recipe is syntax is valid, and see the steps of the build:

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon build --dryrun debiantesting-custom-kernel.yaml
debiantesting-custom-kernel (/home/pneyron/debiantesting-custom-kernel/debiantesting-custom-kernel.yaml)
[Bootstrap]
  _init_bootstrap (internal)
  --> 1 _init_0_create_appliance
  --> 2 _init_1_create_appliance
  --> 3 _init_2_generate_ssh_keys
  --> 4 _init_3_inject_ssh_private_key
  --> 5 _init_4_start_vm
  --> 6 _init_5_start_vm
  --> 7 _init_6_start_vm
  --> 8 _init_7_start_vm_synchrone
  --> 9 _init_8_start_vm_synchrone
  --> 10 _init_9_start_vm_synchrone
  --> 11 _init_10_disable_checkpoint
  prepare_ssh_to_out_context (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/prepare_ssh_to_out_context.yaml)
  --> 12 select_empty_port
  --> 13 prepare_ssh_config
  download_upstream_tarball (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/download_upstream_tarball.yaml)
  --> 14 download
  create_appliance (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/create_appliance.yaml)
  --> 15 create_appliance
  prepare_appliance (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/prepare_appliance.yaml)
  --> 16 generate_ssh_keys
  --> 17 inject_ssh_private_key
  --> 18 add_insecure_key_to_ssh_config
  start_qemu (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/start_qemu.yaml)
  --> 19 start_vm
  --> 20 start_vm_synchrone
  --> 21 _clean_0_start_vm_synchrone
  _clean_bootstrap (internal)
  --> 22 _clean_1_start_vm
  --> 23 _clean_0_download
[Setup]
  _init_setup (internal)
  a_customization_step (/home/pneyron/debiantesting-custom-kernel/debiantesting-custom-kernel.yaml)
  --> 24 microstep1
  --> 25 microstep1
  _clean_setup (internal)
  --> 26 _clean_0_start_vm
[Export]
  _init_export (internal)
  --> 27 _init_0_inject_ssh_private_key
  disable_checkpoint (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/disable_checkpoint.yaml)
  --> 28 disable_checkpoint
  save_appliance_VM (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/export/save_appliance_VM.yaml)
  --> 29 save_appliance
  export_modified_g5k_env (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/export/export_modified_g5k_env.yaml)
  --> 30 create_kaenv_file
  --> 31 create_additional_postinstall_archive
  --> 32 export_files
  _clean_export (internal)
  --> 33 _clean_1_start_vm
  --> 34 _clean_0_delete_initial_image_at_the_end

1. Adapt recipe: set the upstream environment to fetch the tarball from
2. First test that it builds
3. Add First step to download kernel sources
4. Test
5. Add step to fetch the debian kernel configuration
6. Add step to dive in make nconfig
7. Add step to dive in
8. Add step to configure option
9. Add step to dive in
10. Add step to compile
11. Add step to finish work