Custom-kernel-environment

From Grid5000
Jump to navigation Jump to search

In this tutorial, we describe how to create a kameleon recipe to build a kadeploy environment that embeds a customized Linux kernel.

About Kameleon

Quoting Kameleon's documentation: Thanks to Kameleon, one can write recipes that describe how to create, step by step, customized operating systems in any desired target format, and then cook them (build them), just like GNU make cooks sources using a Makefile to build binary programs.

Also read the Environment creation page.

Setup the workspace

Prepare the kameleon tool

First, get a job on a node

While the kameleon tool can be installed and run anyware, the easiest is to run it on a Grid'5000 node where is it already installed. This may avoid dependency issues. 

pneyron@fgrenoble:~$ oarsub -I -l walltime=2
# Filtering out exotic resources (yeti, troll, servan, drac).
OAR_JOB_ID=2514218
# Interactive mode: waiting...
# Starting...
pneyron@dahu-12:~$
Warning.png Warning

We will use the job's node (in the example: dahu-12) for the whole tutorial: do not end the job, do not exit its shell!

Then, add Grid'5000's environment recipes as a kameleon templates repository

If not already added, do as follows: 

pneyron@dahu-12:~$ kameleon repo add grid5000 https://gitlab.inria.fr/grid5000/environments-recipes
Cloning into '/home/pneyron/.kameleon.d/repos/grid5000'...
warning: redirecting to https://gitlab.inria.fr/grid5000/environments-recipes.git/
remote: Enumerating objects: 23035, done.
remote: Counting objects: 100% (2467/2467), done.
remote: Compressing objects: 100% (434/434), done.
remote: Total 23035 (delta 2004), reused 2431 (delta 1987), pack-reused 20568
Receiving objects: 100% (23035/23035), 2.83 MiB | 21.79 MiB/s, done.
Resolving deltas: 100% (14477/14477), done.

Else, we may just update it: 

pneyron@dahu-12:~$ kameleon repo update grid5000
...

We can run kameleon template list to list the available templates...ᗧ･･････････

Create your environment recipe

We create a debiantesting-custom-kernel recipe, which will build on top of the debiantesting-min environment of Grid'5000.

While we could extend the debiantesting-x64-min recipe, which would imply building our environment from scratch (including the run of the OS installer, thus takes a lot more time), we prefer using the from_grid5000_environment/base recipe as our template, which will let kameleon extract the already built debiantesting-x64-min environment image tarball and allow adding customizations on top of it.

Create the debiantesting-custom-kernel recipe, which extends from_grid5000_environment/base.yaml
pneyron@dahu-12:~$ mkdir debiantesting-custom-kernel
pneyron@dahu-12:~$ cd debiantesting-custom-kernel
pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon new debiantesting-custom-kernel grid5000/from_grid5000_environment/base
      create  grid5000/from_grid5000_environment/base.yaml
      create  grid5000/steps/backend/qemu.yaml
      create  grid5000/steps/backend/VM.yaml
      create  grid5000/steps/backend/chroot.yaml
      create  grid5000/steps/aliases/defaults.yaml
      create  grid5000/steps/checkpoints/qemu.yaml
      create  grid5000/steps/bootstrap/prepare_ssh_to_out_context.yaml
      create  grid5000/steps/bootstrap/download_upstream_tarball.yaml
      create  grid5000/steps/bootstrap/create_appliance.yaml
      create  grid5000/steps/bootstrap/prepare_appliance.yaml
      create  grid5000/steps/bootstrap/start_qemu.yaml
      create  grid5000/steps/disable_checkpoint.yaml
      create  grid5000/steps/export/save_appliance_VM.yaml
      create  grid5000/steps/export/export_modified_g5k_env.yaml
      create  grid5000/steps/data/helpers/create_appliance.py
      create  grid5000/steps/data/qemu-sendkeys.rb
      create  grid5000/steps/data/helpers/export_appliance.py
      create  grid5000/steps/data/helpers/kaenv-customize.py
      create  grid5000/steps/env/bashrc
      create  grid5000/steps/env/functions.sh
      create  debiantesting-custom-kernel.yaml
pneyron@dahu-12:~/debiantesting-custom-kernel$

The command imports not only the grid5000/from_grid5000_environment/base.yaml recipe that we will extend, but also all its dependencies.

Note.png Note

When taking over a previous work, we may want to update the template files, which may have updates. We can do so with the following command: kameleon repo update grid5000 && kameleon template import grid5000/from_grid5000_environment/base.

Initiate the git repository for the work

At this stage, it's a good idea to start versioning our work in Git. We init the git repository and add the already created files, then commit. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ git init
...
Initialized empty Git repository in /home/pneyron/debiantesting-custom-kernel/.git/
pneyron@dahu-12:~/debiantesting-custom-kernel$ git add debiantesting-custom-kernel.yaml
pneyron@dahu-12:~/debiantesting-custom-kernel$ git add grid5000/
pneyron@dahu-12:~/debiantesting-custom-kernel$ git commit -m "First commit after running 'kameleon new ...'"
[master (root-commit) 5eec2d1] First commit after running 'kameleon new ...'
 21 files changed, 2267 insertions(+)
 create mode 100644 debiantesting-custom-kernel.yaml
 create mode 100644 grid5000/from_grid5000_environment/base.yaml
 create mode 100644 grid5000/steps/aliases/defaults.yaml
 create mode 100644 grid5000/steps/backend/VM.yaml
 create mode 100644 grid5000/steps/backend/chroot.yaml
 create mode 100644 grid5000/steps/backend/qemu.yaml
 create mode 100644 grid5000/steps/bootstrap/create_appliance.yaml
 create mode 100644 grid5000/steps/bootstrap/download_upstream_tarball.yaml
 create mode 100644 grid5000/steps/bootstrap/prepare_appliance.yaml
 create mode 100644 grid5000/steps/bootstrap/prepare_ssh_to_out_context.yaml
 create mode 100644 grid5000/steps/bootstrap/start_qemu.yaml
 create mode 100644 grid5000/steps/checkpoints/qemu.yaml
 create mode 100755 grid5000/steps/data/helpers/create_appliance.py
 create mode 100755 grid5000/steps/data/helpers/export_appliance.py
 create mode 100755 grid5000/steps/data/helpers/kaenv-customize.py
 create mode 100755 grid5000/steps/data/qemu-sendkeys.rb
 create mode 100644 grid5000/steps/disable_checkpoint.yaml
 create mode 100644 grid5000/steps/env/bashrc
 create mode 100644 grid5000/steps/env/functions.sh
 create mode 100644 grid5000/steps/export/export_modified_g5k_env.yaml
 create mode 100644 grid5000/steps/export/save_appliance_VM.yaml
pneyron@dahu-12:~/debiantesting-custom-kernel$

We may already push the git repository to a git remote, e.g., a Gitlab project, but this is out of the scope of this tutorial. Please mind doing it whenever you want, e.g., to share your work with others.

Warning.png Warning

In the following, we will often display a git diff to show modifications to be made to the recipe, as if they were already committed.

Customize the recipe

Set the upstream environment information

At first, we just have to set the upstream environment we want to import the image tarball from. We edit the grid5000_environment_import_name kameleon global variable in the recipe file: debiantesting-custom-kernel.yaml, as shown by a diff afterward. 

diff --git a/debiantesting-custom-kernel.yaml b/debiantesting-custom-kernel.yaml
index 3cf22ac..b5805b8 100644
--- a/debiantesting-custom-kernel.yaml
+++ b/debiantesting-custom-kernel.yaml
@@ -27,7 +27,7 @@ global:
   #grid5000_site: "grenoble"
 
   ## Environment to build from
-  #grid5000_environment_import_name: "debian11-min"
+  grid5000_environment_import_name: "debiantesting-min"
   #grid5000_environment_import_user: "deploy"
   #grid5000_environment_import_version: ""
   #grid5000_environment_import_arch: "x86_64"

Then we commit the change: 

pneyron@dahu-12:~/debiantesting-custom-kernel$ git commit -m 'Set grid5000_environment_import_name: "debiantesting-min"' -a
[master 1a3f1d8] Set grid5000_environment_import_name: "debiantesting-min"
 1 file changed, 1 insertion(+), 1 deletion(-)

Try a first build, enable checkpointing

First we can run a dryrun build of the recipe to check that the recipe syntax is valid, and see the steps that will be involved in the build: 

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon build debiantesting-custom-kernel.yaml --dryrun
debiantesting-custom-kernel (/home/pneyron/debiantesting-custom-kernel/debiantesting-custom-kernel.yaml)
[Bootstrap]
  _init_bootstrap (internal)
  --> 1 _init_0_create_appliance
  --> 2 _init_1_create_appliance
  --> 3 _init_2_generate_ssh_keys
  --> 4 _init_3_inject_ssh_private_key
  --> 5 _init_4_start_vm
  --> 6 _init_5_start_vm
  --> 7 _init_6_start_vm
  --> 8 _init_7_start_vm_synchrone
  --> 9 _init_8_start_vm_synchrone
  --> 10 _init_9_start_vm_synchrone
  --> 11 _init_10_disable_checkpoint
  prepare_ssh_to_out_context (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/prepare_ssh_to_out_context.yaml)
  --> 12 select_empty_port
  --> 13 prepare_ssh_config
  download_upstream_tarball (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/download_upstream_tarball.yaml)
  --> 14 download
  create_appliance (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/create_appliance.yaml)
  --> 15 create_appliance
  prepare_appliance (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/prepare_appliance.yaml)
  --> 16 generate_ssh_keys
  --> 17 inject_ssh_private_key
  --> 18 add_insecure_key_to_ssh_config
  start_qemu (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/bootstrap/start_qemu.yaml)
  --> 19 start_vm
  --> 20 start_vm_synchrone
  --> 21 _clean_0_start_vm_synchrone
  _clean_bootstrap (internal)
  --> 22 _clean_1_start_vm
  --> 23 _clean_0_download
[Setup]
  _init_setup (internal)
  a_customization_step (/home/pneyron/debiantesting-custom-kernel/debiantesting-custom-kernel.yaml)
  --> 24 microstep1
  --> 25 microstep1
  _clean_setup (internal)
  --> 26 _clean_0_start_vm
[Export]
  _init_export (internal)
  --> 27 _init_0_inject_ssh_private_key
  disable_checkpoint (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/disable_checkpoint.yaml)
  --> 28 disable_checkpoint
  save_appliance_VM (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/export/save_appliance_VM.yaml)
  --> 29 save_appliance
  export_modified_g5k_env (/home/pneyron/debiantesting-custom-kernel/grid5000/steps/export/export_modified_g5k_env.yaml)
  --> 30 create_kaenv_file
  --> 31 create_additional_postinstall_archive
  --> 32 export_files
  _clean_export (internal)
  --> 33 _clean_1_start_vm
  --> 34 _clean_0_delete_initial_image_at_the_end

A lot of information, but we are mainly interested in the setup section, which for now does not contain anything very relevant.

Warning.png Warning

Before our first real build, in order to speed up the build, we can symlink the kameleon build directory to /tmp, to use the SSD instead of NFS (home directory) . 

pneyron@dahu-12:~/debiantesting-custom-kernel$ mkdir -p /tmp/kameleon && ln -s /tmp/kameleon build

We now run a first build.

Note.png Note

Since we know that our work will follow a methodology of trials and errors, we enable kameleon's checkpointing mechanism (-c command switch), so that every build may restart from a previously built state. Of course, activating the checkpointing has an overhead, so it's a threadoff to evaluate.

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon build debiantesting-custom-kernel.yaml -c
...
Step 19: bootstrap/start_qemu/start_vm
--> Running the step...
[local] Qemu start VM, VM shutdown in setup section cleaning
[local] Starting qemu...
[local] VNC port: 0
[local] Waiting for SSH to become available in VM for out_context...(89s)
...
Step 24: setup/a_customization_step/microstep1
--> Running the step...
[in] The in_context has been initialized
[in] Hello world!
--> Creating checkpoint: 'setup/a_customization_step/microstep1' (912c3ebd1d1c)
Step 25: setup/a_customization_step/microstep1
--> Running the step...
Kameleon breakpoint!
Press [c] to continue with execution
Press [a] to abort execution
Press [l] to switch to local_context shell
Press [o] to switch to out_context shell
Press [i] to switch to in_context shell
answer ? [c/a/l/o/i]:

The output is very verbose, so we highlight only parts of it.

  • Step 19 starts the step where the Qemu VM in which our environment is "cooked". As written, it is possible to connect a VNC client to it (needs Grid'5000 VPN or a SSH tunnel), to see the VM's terminal console. Here, it would show the VM boot and then a login prompt, so it is not so useful.
  • Step 24 begins the setup section of the recipe. This step just writes "Hello World!".
  • Step 25 is more interesting as it sets a breakpoint, allowing us to enter the build contexts for inspection.
    • Here, only the in context is interesting, because the recipe does not use an out context (actually, out and in are the same here). Typing i+ENTER gives a shell inside the VM. Ctrl-D exits from the shell. Please note that commands run in the shell are note recorded in the recipe. That's just for debugging.
    • If we want to go on with the build, we type c+ENTER
    • However, we can also just abort, because finishing the build is useless at this stage.
Step 25: setup/a_customization_step/microstep1
--> Running the step...
Kameleon breakpoint!
Press [c] to continue with execution
Press [a] to abort execution
Press [l] to switch to local_context shell
Press [o] to switch to out_context shell
Press [i] to switch to in_context shell
answer ? [c/a/l/o/i]:  a
User choice: [a] abort
Aborted...
Waiting for cleanup before exiting...
Cleaning setup section
[local] Executing a graceful shutdown of the qemu VM via the monitor socket...
[local] Waiting for qemu virtual machine to shutdown...(98s)
Cleaning export section
Error: Execution aborted...

Since we activated the checkpointing mechanism, we can look at the recorded checkpoints by adding the -l command switch: 

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon build debiantesting-custom-kernel.yaml -c -l
The following checkpoints are available for recipe 'debiantesting-custom-kernel':
ID           | STEP                                            
-------------|-------------------------------------------------
6ba05299507c | bootstrap/start_qemu/start_vm_synchrone         
2533d6a21027 | bootstrap/start_qemu/_clean_0_start_vm_synchrone
912c3ebd1d1c | setup/a_customization_step/microstep1

(Not all steps have a checkpoint.)

First steps in the setup section: Download kernel sources

We now add steps to download the Linux kernel sources and extract them in the build VM.

  • First, we add some global variables to parameterize everything. As a reminder, variables must then be used with the $${variable} syntax.
  • In a first macrostep, we install with apt some tools that will be required for the recipe (we use the apt-get_in alias, defined in grid5000/steps/aliases/defaults.yaml.
  • Then, we download and extract the Linux kernel tarball (of course it could do more, e.g., apply some patches, etc...)
  • Finally, we rewrite a bit the debug step, with the breakpoint that is still useful.
diff --git a/debiantesting-custom-kernel.yaml b/debiantesting-custom-kernel.yaml
index b5805b8..b651018 100644
--- a/debiantesting-custom-kernel.yaml
+++ b/debiantesting-custom-kernel.yaml
@@ -79,6 +79,11 @@ global:
   ##   kameleon info debiantesting-custom-kernel.yaml
   ## Or define any new variable you would need. e.g.:
   #my_variable: my_value
+  kernel_version: 6.15.3
+  kernel_version_suffix: -custom
+  src_dir: /usr/src
+  kernel_src_dir: $${src_dir}/linux-$${kernel_version}
+  kernel_src_url: https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-$${kernel_version}.tar.xz
 
 bootstrap:
   ### The bootstrap section takes in charge the import of the Grid'5000
@@ -86,14 +91,23 @@ bootstrap:
   - "@base"
 
 setup:
-  ### The setup section is where to place your customization. Add all steps
-  ## required by your customization.
-  ## The following is given as example only, replace with your steps.
-  - a_customization_step:
-    - microstep1:
-      - exec_in: echo "Hello world!"
-    - microstep1:
-      # This breakpoint will stop the build for inspecting the environment
+  - prerequisites:
+    - install_tools:
+      - apt-get_in: update
+      - apt-get_in: install make gcc xxd flex bison python3 bc rsync kmod libelf-dev libssl-dev xz-utils
+  - install_kernel_sources:
+    - download_kernel_sources:
+      - exec_in: |
+          set -e
+          case $${kernel_src_url} in
+            *xz)
+              TAR_OPT=-J
+              ;;
+          esac
+          wget $${kernel_src_url} -O- | tar -xv $TAR_OPT -C $${src_dir}
+  - debug:
+    - dive_in:
+      - on_checkpoint: disabled
       - breakpoint
 
 export:
Note.png Note

Please note the step definition: 

     - exec_in: |
         set -e
         first_command
         second_command
         ...
  • The YAML syntax with | requires 4 spaces of indentation on the next lines.
  • We use set -e so that any command makes the step fail, not just the last one.

Now, before trying to build again, we may run kameleon info to see the resolution of the variables. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon info debiantesting-custom-kernel.yaml 
--------------------
[Name]
 -> debiantesting-custom-kernel
[Path]
 -> /home/pneyron/debiantesting-custom-kernel/debiantesting-custom-kernel.yaml
[Description]
...
[Variables]
 -> appliance_filename: /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/debiantesting-custom-kernel
 -> appliance_formats: tar.zst
 -> appliance_tar_compression_level: 9
 -> appliance_tar_excludes: ./etc/fstab ./root/.bash_history ./root/kameleon_workdir ./root/.ssh ./var/tmp/* ./tmp/* ./var/log/* ./dev/* ./proc/* ./run/* ./sys/*
 -> arch: x86_64
 -> backend: qemu
 -> checkpoint_disabled_file: /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/checkpoint_disabled
 -> checkpointing_enabled: false
 -> disk_device: /dev/vda
 -> distrib: debian
 -> filesystem_type: ext4
 -> grid5000_environment_export_additional_postinstall_archive: 
 -> grid5000_environment_export_additional_postinstall_dir: additional_postinstall
 -> grid5000_environment_export_additional_postinstall_script: 
 -> grid5000_environment_export_baseurl: local://$HOME/public/
 -> grid5000_environment_export_description: Customized debiantesting-min
 -> grid5000_environment_export_dir: $HOME/public/
 -> grid5000_environment_export_format: tar.zst
 -> grid5000_environment_export_name: debiantesting-custom-kernel
 -> grid5000_environment_export_visibility: shared
 -> grid5000_environment_import_arch: x86_64
 -> grid5000_environment_import_name: debiantesting-min
 -> grid5000_environment_import_user: deploy
 -> grid5000_environment_import_version: 
 -> grid5000_frontend: frontend
 -> grid5000_site: grenoble
 -> image_disk: /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/base_debiantesting-custom-kernel
 -> image_format: qcow2
 -> image_size: 20G
 -> in_context: {"cmd"=>"ssh -F /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/ssh_config debiantesting-custom-kernel -t /bin/bash", "proxy_cache"=>"10.0.2.2", "workdir"=>"/root/kameleon_workdir", "interactive_cmd"=>"ssh -F /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/ssh_config debiantesting-custom-kernel -t /bin/bash"}
 -> include_steps: ["debian/11", "debian"]
 -> kameleon_cwd: /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel
 -> kameleon_recipe_dir: /home/pneyron/debiantesting-custom-kernel
 -> kameleon_recipe_name: debiantesting-custom-kernel
 -> kameleon_short_uuid: e46908ed1837
 -> kameleon_uuid: 1ff6d267-8c6f-408f-bf41-e46908ed1837
 -> kernel_args: quiet
 -> kernel_src_dir: /usr/src/linux-6.15.3
 -> kernel_src_url: https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.15.3.tar.xz
 -> kernel_version: 6.15.3
 -> kernel_version_suffix: -custom
 -> local_ip: 10.0.2.2
 -> out_context: {"cmd"=>"ssh -F /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/ssh_config debiantesting-custom-kernel -t /bin/bash", "proxy_cache"=>"10.0.2.2", "workdir"=>"/root/kameleon_workdir", "interactive_cmd"=>"ssh -F /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/ssh_config debiantesting-custom-kernel -t /bin/bash"}
 -> persistent_cache: false
 -> proxy_in: 
 -> proxy_local: 
 -> proxy_out: 
 -> qemu_append_cmdline: 
 -> qemu_arch: x86_64
 -> qemu_checkpoint_timeout: 300
 -> qemu_cpus: $(nproc)
 -> qemu_enable_kvm: true
 -> qemu_initrd_path: 
 -> qemu_iso_path: 
 -> qemu_kernel_path: 
 -> qemu_memory_size: 2G
 -> qemu_monitor_socket: /tmp/kameleon_qemu_monitor.debiantesting-custom-kernel.e46908ed1837.socket
 -> qemu_netdev_user_options: dnssearch=grenoble.grid5000.fr,dnssearch=grid5000.fr
 -> qemu_pidfile: /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/qemu.pid
 -> release: 11
 -> rootfs: /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/rootfs
 -> src_dir: /usr/src
 -> ssh_config_file: /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/ssh_config

Then we run the build again. We will keep using the -c switch to enable checkpointing (we'll do until the recipe is finalized, for the final build).

Thanks to the breakpoint, we may look into the in context that the /usr/src/ directory indeed contains the Linux kernel sources. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon build debiantesting-custom-kernel.yaml -c
...
[in] linux-6.15.3/virt/lib/
[in] linux-6.15.3/virt/lib/Kconfig
[in] linux-6.15.3/virt/lib/Makefile
[in] linux-6.15.3/virt/lib/irqbypass.c
--> Creating checkpoint: 'setup/install_kernel_sources_and_config/download_kernel_sources' (0d95471fe7c6)
Step install_kernel_sources_and_config took: 8 secs
Step 26: setup/debug/dive_in
--> Running the step...
Kameleon breakpoint!
Press [c] to continue with execution
Press [a] to abort execution
Press [l] to switch to local_context shell
Press [o] to switch to out_context shell
Press [i] to switch to in_context shell
answer ? [c/a/l/o/i]:  i
User choice: [i] launch in_context
Starting interactive shell
(in_context) root@debian: /root/kameleon_workdir # ls /usr/src/linux-6.15.3/
COPYING  Documentation	Kconfig   MAINTAINERS  README  block  crypto   fs	init	  ipc	  lib  net   samples  security	tools  virt
CREDITS  Kbuild		LICENSES  Makefile     arch    certs  drivers  include	io_uring  kernel  mm   rust  scripts  sound	usr
(in_context) root@debian: /root/kameleon_workdir # exit
exit
Saved ENV in /root/kameleon_workdir/kameleon_scripts/in/bash_env file
Connection to 127.0.0.1 closed.
Getting back to Kameleon...
Press [c] to continue with execution
Press [a] to abort execution
Press [l] to switch to local_context shell
Press [o] to switch to out_context shell
Press [i] to switch to in_context shell
answer ? [c/a/l/o/i]:  a
User choice: [a] abort
Aborted...
Waiting for cleanup before exiting...
Cleaning setup section
[local] Executing a graceful shutdown of the qemu VM via the monitor socket...
[local] Waiting for qemu virtual machine to shutdown...(97s)
Cleaning export section
Error: Execution aborted...
pneyron@dahu-12:~/debiantesting-custom-kernel$

Finally, we can look at the new recorded checkpoints. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon build debiantesting-custom-kernel.yaml -cl
The following checkpoints are available for recipe 'debiantesting-custom-kernel':
ID           | STEP                                                        
-------------|-------------------------------------------------------------
6ba05299507c | bootstrap/start_qemu/start_vm_synchrone                     
2533d6a21027 | bootstrap/start_qemu/_clean_0_start_vm_synchrone            
7e945cc37e22 | setup/prerequisites/install_tools                           
0d95471fe7c6 | setup/install_kernel_sources_and_config/download_kernel_s...
Note.png Note

By default kameleon build -c restarts from the latest available checkpoint that is not invalidated by a newer change in the recipe. However, it is possible to specify an earlier checkpoint with the -F option. More info in kameleon build help.

Since we are happy with our changes, we commit to git. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ git commit -m 'Download kernel' -a

More setup steps: Configure and build kernel

We now add the steps to configure the kernel and build it.

Note.png Note

We use YAML anchors (&1 and *1) to avoid repeating the dive_in code, to add additional breakpoints.

diff --git a/debiantesting-custom-kernel.yaml b/debiantesting-custom-kernel.yaml
index 74d5c69..a00c39b 100644
--- a/debiantesting-custom-kernel.yaml
+++ b/debiantesting-custom-kernel.yaml
@@ -93,6 +93,10 @@ global:
   kernel_src_dir: $${src_dir}/linux-$${kernel_version}
   kernel_src_url: https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-$${kernel_version}.tar.xz

+  kernel_make_cmd: "make -C $${kernel_src_dir} -j $(nproc) LOCALVERSION=$${kernel_version_suffix}"
+
+  dive: yes
+
 bootstrap:
   ### The bootstrap section takes in charge the import of the Grid'5000
   ## environment to customize. No modification should be needed here.
@@ -116,9 +120,27 @@ setup:
           esac
           wget $${kernel_src_url} -O- | tar -xv $TAR_OPT -C $${src_dir}
   - debug:
-    - dive_in:
+    - dive_in: &1
       - on_checkpoint: disabled
-      - breakpoint
+      - test:
+        - exec_local: test '$${dive}' = 'yes'
+        - group:
+          - apt-get_in: install libncurses-dev
+          - exec_local: |
+              echo "Kernel make command: $${kernel_make_cmd}"
+          - breakpoint
+  - configure_kernel:
+    - apply_default_kernel_config:
+      - exec_in: $${kernel_make_cmd} defconfig
+    - dive_in_again: *1
+  - build_and_install_kernel:
+    - build_kernel_objects:
+      - exec_in: $${kernel_make_cmd}
+      - exec_in: $${kernel_make_cmd} modules
+    - install_kernel_objects:
+      - exec_in: $${kernel_make_cmd} modules_install
+      - exec_in: $${kernel_make_cmd} install
+    - dive_in_again: *1

 export:
   ### The export section takes in charge the export of your customized Grid'5000

Then we build the recipe again: the kernel compilation will be part of it this time!

We see that the checkpoint mechanism avoids redoing some steps (see the "--> Checkpoint ahead, do nothing" messages)! 

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon build debiantesting-custom-kernel.yaml -c
Creating kameleon build directory: /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel
Starting build recipe 'debiantesting-custom-kernel.yaml'
[local] The local_context has been initialized
Restoring last build from step: setup/configure_kernel/apply_default_kernel_config
Step 1: bootstrap/_init_bootstrap/_init_0_create_appliance
--> Checkpoint ahead, do nothing
...
Step 24: setup/prerequisites/install_tools
--> Checkpoint ahead, do nothing
Step prerequisites took: 0 secs
Step 25: setup/install_kernel_sources_and_config/download_kernel_sources
--> Checkpoint ahead, do nothing
Step install_kernel_sources_and_config took: 0 secs
Step 26: setup/debug/dive_in
--> Running the step...
[in] The in_context has been initialized
[in] Reading package lists...
[in] Building dependency tree...
[in] Reading state information...
[in] The following additional packages will be installed:
[in]   libgpm2 libncurses6 libncursesw6 libtinfo6 ncurses-base ncurses-bin
[in]   ncurses-term
...
[local] Kernel make command: make -C /usr/src/linux-6.15.3 -j 64 LOCALVERSION=-custom
Kameleon breakpoint!
Press [c] to continue with execution
Press [a] to abort execution
Press [l] to switch to local_context shell
Press [o] to switch to out_context shell
Press [i] to switch to in_context shell
answer ? [c/a/l/o/i]:  c
User choice: [c] continue
--> Do not create a checkpoint for this microstep: disabled in the microstep definition
Step debug took: 39 secs
Step 27: setup/configure_kernel/apply_default_kernel_config
--> Running the step...
[in] make: Entering directory '/usr/src/linux-6.15.3'
[in]   HOSTCC  scripts/basic/fixdep
[in]   HOSTCC  scripts/kconfig/conf.o
[in]   HOSTCC  scripts/kconfig/confdata.o
[in]   HOSTCC  scripts/kconfig/expr.o
[in]   LEX     scripts/kconfig/lexer.lex.c
[in]   YACC    scripts/kconfig/parser.tab.[ch]
[in]   HOSTCC  scripts/kconfig/menu.o
[in]   HOSTCC  scripts/kconfig/preprocess.o
[in]   HOSTCC  scripts/kconfig/util.o
[in]   HOSTCC  scripts/kconfig/symbol.o
[in]   HOSTCC  scripts/kconfig/lexer.lex.o
[in]   HOSTCC  scripts/kconfig/parser.tab.o
[in]   HOSTLD  scripts/kconfig/conf
[in] *** Default configuration is based on 'x86_64_defconfig'
[in] #
[in] # configuration written to .config
[in] #
[in] make: Leaving directory '/usr/src/linux-6.15.3'
--> Creating checkpoint: 'setup/configure_kernel/apply_default_kernel_config' (16a0b1967ce4)
Step 28: setup/configure_kernel/dive_in_again
--> Running the step...
[in] Reading package lists...
[in] Building dependency tree...
[in] Reading state information...
[in] libncurses-dev is already the newest version (6.5+20250216-2).
[in] 0 upgraded, 0 newly installed, 0 to remove and 181 not upgraded.
[in] W: --force-yes is deprecated, use one of the options starting with --allow instead.
[local] Kernel make command: make -C /usr/src/linux-6.15.3 -j 64 LOCALVERSION=-custom
Kameleon breakpoint!
Press [c] to continue with execution
Press [a] to abort execution
Press [l] to switch to local_context shell
Press [o] to switch to out_context shell
Press [i] to switch to in_context shell
answer ? [c/a/l/o/i]:  c
User choice: [c] continue
--> Do not create a checkpoint for this microstep: disabled in the microstep definition
Step configure_kernel took: 9 secs
Step 29: setup/build_and_install_kernel/build_kernel_objects
--> Running the step...
[in] make: Entering directory '/usr/src/linux-6.15.3'
[in]   GEN     arch/x86/include/generated/asm/orc_hash.h
[in]   UPD     include/config/kernel.release
[in]   SYSHDR  arch/x86/include/generated/uapi/asm/unistd_32.h
[in]   UPD     include/generated/uapi/linux/version.h
...
[in]   CC      security/selinux/avc.o
[in]   AR      arch/x86/virt/built-in.a
[in]   CC      arch/x86/realmode/rm/wakemain.o
[in]   GEN     usr/initramfs_data.cpio
[in]   CC      block/elevator.o
[in]   COPY    usr/initramfs_inc_data
[in]   AS      usr/initramfs_data.o
[in] gcc: fatal error: Killed signal terminated program cc1
[in] compilation terminated.
[in] gcc: fatal error: Killed signal terminated program cc1
...

At this stage, we notice that the kernel compilation is failing!.

We hit Ctrl-C to stop the compilation, and abort kameleon.

Warning.png Warning

Sometimes, aborting kameleon takes a lot of time or fails. This can be fixed by killing (possibly several times) the qemu process from another shell on the node: 

pneyron@dahu-12:~/debiantesting-custom-kernel$ killall qemu-system-x86_64 
qemu-system-x86_64: no process found

The compilation failure turns out to be because the build VM does not have enough memory for the compilation.

We fix this in the recipe, and also increase the VM disk size at the same time. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ git diff
index fa82600..f61278b 100644
--- a/debiantesting-custom-kernel.yaml
+++ b/debiantesting-custom-kernel.yaml
@@ -70,9 +70,9 @@ global:
   ## Make qemu use UEFI (required for aarch64 and ppc64).
   #qemu_uefi: true
   ## Adapt Qemu memory size (may be needed depending on what's done in setup)
-  #qemu_memory_size: 64G
+  qemu_memory_size: 64G
   ## Adapt Qemu disk image size (may be needed for bigger environments)
-  #image_size: 50G
+  image_size: 50G
 
   ### You can add below any other global variable definition
   ## See the variables which can be overloaded, by running:

Then we run the kameleon build again. Unfortunately, this change causes the build to start over from the beginning (checkpoints are invalidated). 

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon build debiantesting-custom-kernel.yaml -c
...
[in]   INSTALL /lib/modules/6.15.3-custom/kernel/net/netfilter/xt_MASQUERADE.ko
[in]   INSTALL /lib/modules/6.15.3-custom/kernel/net/netfilter/xt_addrtype.ko
[in]   INSTALL /lib/modules/6.15.3-custom/kernel/net/ipv4/netfilter/iptable_nat.ko
[in]   DEPMOD  /lib/modules/6.15.3-custom
[in] make: Leaving directory '/usr/src/linux-6.15.3'
[in] make: Entering directory '/usr/src/linux-6.15.3'
[in]   INSTALL /boot
[in] run-parts: executing /etc/kernel/postinst.d/initramfs-tools 6.15.3-custom /boot/vmlinuz-6.15.3-custom
[in] update-initramfs: Generating /boot/initrd.img-6.15.3-custom
[in] W: Possible missing firmware /lib/firmware/rtl_nic/rtl8126a-3.fw for built-in driver r8169
...
[in] run-parts: executing /etc/kernel/postinst.d/zz-update-grub 6.15.3-custom /boot/vmlinuz-6.15.3-custom
[in] make: Leaving directory '/usr/src/linux-6.15.3'
--> Creating checkpoint: 'setup/build_and_install_kernel/install_kernel_objects' (b4dd1c0e36e2)
...
Step 35: export/save_appliance_VM/save_appliance
--> Running the step...
[local] INFO: Creating /home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/debiantesting-custom-kernel.tar.zst
--> Do not create a checkpoint for this microstep: disabled in backend
Step save_appliance_VM took: 114 secs
Step 36: export/export_modified_g5k_env/create_kaenv_file
--> Running the step...
--> Do not create a checkpoint for this microstep: disabled in backend
Step 37: export/export_modified_g5k_env/create_additional_postinstall_archive
--> Running the step...
--> Do not create a checkpoint for this microstep: disabled in backend
Step 38: export/export_modified_g5k_env/export_files
--> Running the step...
[local] Copying grid5000 environment files to /home/pneyron/public/
[local] '/home/pneyron/debiantesting-custom-kernel/build/debiantesting-custom-kernel/debiantesting-custom-kernel.tar.zst' -> '/home/pneyron/public/debiantesting-custom-kernel.tar.zst'
[local] 'debiantesting-custom-kernel.dsc' -> '/home/pneyron/public/debiantesting-custom-kernel.dsc'
--> Do not create a checkpoint for this microstep: disabled in backend
Step export_modified_g5k_env took: 4 secs
Step 39: export/_clean_export/_clean_1_start_vm
--> Running the step...
--> Do not create a checkpoint for this microstep: disabled in backend
Step 40: export/_clean_export/_clean_0_delete_initial_image_at_the_end
--> Skip microstep as requested when checkpointing is activated
Step _clean_export took: 0 secs

Successfully built 'debiantesting-custom-kernel.yaml'
Total duration: 797 secs

Now the kernel compilation is ok, and the recipe build completes!

We commit to git. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ git commit -m 'Kernel compilation ok, build ok' -a

Check the generated environment

We can look at the generated environment description file /home/pneyron/public/debiantesting-custom-kernel.dsc. 

---
arch: x86_64
author: Pierre Neyron
boot:
  initrd: /initrd.img
  kernel: /vmlinuz
  kernel_params: modprobe.blacklist=nouveau
description: Customized debiantesting-min
destructive: false
filesystem: ext4
image:
  compression: zstd
  file: local:///home/pneyron/public//debiantesting-custom-kernel.tar.zst
  kind: tar
multipart: false
name: debiantesting-custom-kernel
os: linux
partition_type: 131
postinstalls:
- archive: server:///grid5000/postinstalls/g5k-postinstall.tgz
  compression: gzip
  script: g5k-postinstall --net debian --disk-aliases
version: 2025070620
visibility: shared

As we can see, the environment kernel image and initrd are /vmlinuz and /initrd.img, which have to be symlinks to our compiled kernel.

Let's check if it's actually the case in our environment tarball. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ tar tvf /home/pneyron/public/debiantesting-custom-kernel.tar.zst | grep -e vmlinuz -e initrd.img 
-rw-r--r-- 0/0        49451473 2025-07-06 21:54 ./boot/initrd.img-6.12.11-amd64
-rw-r--r-- 0/0        19253105 2025-07-06 21:59 ./boot/initrd.img-6.15.3-custom
-rw-r--r-- 0/0        10028992 2025-01-25 21:15 ./boot/vmlinuz-6.12.11-amd64
-rw-r--r-- 0/0        13902848 2025-07-06 21:59 ./boot/vmlinuz-6.15.3-custom
lrwxrwxrwx 0/0               0 2025-02-03 10:03 ./initrd.img -> boot/initrd.img-6.12.11-amd64
lrwxrwxrwx 0/0               0 2025-02-03 10:03 ./initrd.img.old -> boot/initrd.img-6.12.11-amd64
-rw-rw-r-- 0/0            1196 2025-06-19 15:41 ./usr/src/linux-6.15.3/arch/mips/boot/compressed/calc_vmlinuz_load_addr.c
lrwxrwxrwx 0/0               0 2025-02-03 10:03 ./vmlinuz -> boot/vmlinuz-6.12.11-amd64
lrwxrwxrwx 0/0               0 2025-02-03 10:03 ./vmlinuz.old -> boot/vmlinuz-6.12.11-amd64

Not good.

There is 2 ways to fix that issue:

  • Add a step in the recipe to fix the /vmlinuz and /initrd.img symlinks in the VM before the environment tarball is exported.
  • Modify the recipe to set the good path for kernel and initrd in the environment description, to the actual files in the /boot directory.

We do the latter: 

pneyron@dahu-12:~/debiantesting-custom-kernel$ git diff
diff --git a/debiantesting-custom-kernel.yaml b/debiantesting-custom-kernel.yaml
index 55ab026..704fc72 100644
--- a/debiantesting-custom-kernel.yaml
+++ b/debiantesting-custom-kernel.yaml
@@ -44,8 +44,8 @@ global:
 
   ## Optionaly, set the kernel image, initrd and kernel params (command line)
   ## if the default /vmlinuz ans /initrd.img symlinks are not ok
-  #grid5000_environment_export_boot_kernel: "/boot/vmlinuz-x.y.z"
-  #grid5000_environment_export_boot_initrd: "/boot/initrd.img-x.y.z"
+  grid5000_environment_export_boot_kernel: "/boot/vmlinuz-$${kernel_version}$${kernel_version_suffix}"
+  grid5000_environment_export_boot_initrd: "/boot/initrd.img-$${kernel_version}$${kernel_version_suffix}"
   #grid5000_environment_export_boot_kernel_params: "..."
 
   ## Optionaly, the environment postinstall script can be changed, e.g. to

We build again, but fortunately, this time, only the latest steps have to be redone, thanks to the checkpoints.

We commit again to git. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ git commit -m 'Fix kernel files' -a

Test the deployment

Now that we have built the recipe with a working new kernel, we can test the resulting environment by deploying it on a node.

In another terminal on the frontend, we create a new job for our deployment. 

pneyron@fgrenoble:~$ oarsub -I -t deploy
# Filtering out exotic resources (yeti, troll, servan, drac).
# Set walltime to default (3600 s).
OAR_JOB_ID=2514234
# Interactive mode: waiting...
# Starting...
pneyron@fgrenoble:~$ kadeploy3 -a public/debiantesting-custom-kernel.dsc
...

While the deployment is running, we can look at the node's console with kaconsole3 to see if the kernel boots ok.

...

It turns out that it's not booting!

That's because the default vanilla kernel configuration is not sufficient to boot the hardware.

Back to modifying the recipe, we add some steps to configure a valid configuration: We fetch Debian's one, but disable some useless features.

Note.png Note

We could also enable some features with ./scripts/config --enable ... or --module.

diff --git a/debiantesting-custom-kernel.yaml b/debiantesting-custom-kernel.yaml
index e53d5cd..430d261 100644
--- a/debiantesting-custom-kernel.yaml
+++ b/debiantesting-custom-kernel.yaml
@@ -93,6 +93,11 @@ global:
   src_dir: /usr/src
   kernel_src_dir: $${src_dir}/linux-$${kernel_version}
   kernel_src_url: https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-$${kernel_version}.tar.xz
+  kernel_arch: x86
+
+  debian_version: sid
+  debian_arch: amd64
+  debian_kernel_package: linux-image-$${debian_arch}

   kernel_make_cmd: "make -C $${kernel_src_dir} -j $(nproc) LOCALVERSION=$${kernel_version_suffix}"

@@ -131,8 +136,43 @@ setup:
               echo "Kernel make command: $${kernel_make_cmd}"
           - breakpoint
   - configure_kernel:
+    - download_and_extract_debian_kernel_config:
+      - exec_in: |
+          set -e
+          mkdir -p /tmp/deb
+          cd /tmp/deb
+          # Download Debian's kernel package in order to retrieve Debian's kernel configuration
+          wget -nv -r -l3 -nd -H -D packages.debian.org,ftp.fr.debian.org,security.debian.org \
+               --accept-regex "(packages\.debian\.org/$${debian_version}/(|$${debian_arch}/)linux-image(|/download)|(ftp\.fr|security)\.debian\.org/debian(|-update|-security)/pool/(|updates/)main/l/linux-)" \
+               https://packages.debian.org/$${debian_version}/$${debian_kernel_package}
+          rm -f robot.txt
+          dpkg -X $(ls -S | head -n1) .
+          cp -v boot/config-* $${kernel_src_dir}/arch/$${kernel_arch}/configs/debian_defconfig
+          rm -r /tmp/deb/
     - apply_default_kernel_config:
-      - exec_in: $${kernel_make_cmd} defconfig
+      - exec_in: $${kernel_make_cmd} debian_defconfig
+    - disable_some_useless_kernel_features:
+      - on_checkpoint: disabled
+      - exec_in: |
+          set -e
+          cd $${kernel_src_dir}
+          set -x
+          ./scripts/config --disable USB_SUPPORT
+          ./scripts/config --disable WLAN
+          ./scripts/config --disable WIRELESS
+          ./scripts/config --disable WIMAX
+          ./scripts/config --disable BT
+          ./scripts/config --disable NFC
+          ./scripts/config --disable HAMRADIO
+          ./scripts/config --disable SOUND
+          ./scripts/config --disable RFKILL
+          ./scripts/config --disable CAN
+          ./scripts/config --disable W1
+          ./scripts/config --disable GNSS
+          ./scripts/config --disable PARPORT
+          ./scripts/config --disable MEMSTICK
+          ./scripts/config --disable SOUNDWIRE
+          set +x
     - dive_in_again: *1
   - build_and_install_kernel:
     - build_kernel_objects:
Note.png Note

Thanks to the breakpoint, we can dive in the in context and run the kernel configuration tool to find the kernel configuration symbols. 

Step 26: setup/debug/dive_in
--> Running the step...
Kameleon breakpoint!
Press [c] to continue with execution
Press [a] to abort execution
Press [l] to switch to local_context shell
Press [o] to switch to out_context shell
Press [i] to switch to in_context shell
answer ? [c/a/l/o/i]:  i
User choice: [i] launch in_context
Starting interactive shell
(in_context) root@debian: /root/kameleon_workdir # make -C /usr/src/linux-6.15.3 nconfig
...
In the kernel's make nconfig tool, F8 is very handy for searching kernel configuration symbols that we can then enable, disable, or set as a module with the ./scripts/config tool.

Again, we run the kameleon build, then test a new deployment... It should be fine now!

We commit the recipe to git again. 

pneyron@dahu-12:~/debiantesting-custom-kernel$ git commit -m 'Fix kernel configuration to boot hardware' -a

Final customizations

Next steps involve disabling kexec in the environment, so that we have a cold hardware reboot of the machine when deploying, and enabling the display of the grub menu at boot, so that we may interact with it in kaconsole3.

These latest changes can be found in the https://gitlab.inria.fr/neyron/debiantesting-custom-kernel repository, which was built throughout the writing of this tutorial.

Last but not least, we can finally build the environment without the checkpoints and without the debug breakpoints (by setting the dive global variable to no). 

pneyron@dahu-12:~/debiantesting-custom-kernel$ kameleon build debiantesting-custom-kernel -g dive:no

Let's now get a look at the description of the environment we generated: our latest changes are indeed translated in it. 

pneyron@fgrenoble:~/debiantesting-custom-kernel$ cat ~/public/debiantesting-custom-kernel.dsc 
---
arch: x86_64
author: Pierre Neyron
boot:
  initrd: /boot/initrd.img-6.15.3-custom
  kernel: /boot/vmlinuz-6.15.3-custom
  kernel_params: modprobe.blacklist=nouveau
custom_variables:
  BOOTLOADER_SHOW_MENU: '1'
description: Customized debiantesting-min
destructive: false
filesystem: ext4
image:
  compression: zstd
  file: local:///home/pneyron/public//debiantesting-custom-kernel.tar.zst
  kind: tar
multipart: false
name: debiantesting-custom-kernel
options:
  kexec: false
  trust_previous_env: false
os: linux
partition_type: 131
postinstalls:
- archive: server:///grid5000/postinstalls/g5k-postinstall.tgz
  compression: gzip
  script: g5k-postinstall --net debian --disk-aliases
version: 2025070811
visibility: shared

We can deploy it and indeed see in kaconsole3 the effects of our changes.

Next

At this stage, you likely want to integrate further customization of the Linux kernel for a real-world experiment use case. Anyways, the final version of the recipe can be committed to git and shared, providing a traceable, scripted description of how you built the system that is deployed for your experiment.

Note.png Note

The recipe builds on top of the image tarball of another environment, here debiantesting-min. For the sake of full traceability of how our system is built, you may also share the debiantesting-x64-min.yaml recipe from which that tarball is itself built (see /etc/grid5000/release for the exact version of the recipe, which is tagged in https://gitlab.inria.fr/grid5000/environments-recipes/-/tags).

Please also note that you can share both the recipe and the result of the build: the kadeploy environment itself.

A few reminders about the kadeploy environment:

  • It can optionally be recorded in the kadeploy registry, using the kaenv3 command.
  • The environment image tarball must not be deleted: only the environment description is stored in the registry.
  • The environment may be used on any Grid'5000 site/cluster.
Retrieved from "https://www.grid5000.fr/mediawiki/index.php?title=Custom-kernel-environment&oldid=106885"