PMEM-environment

From Grid5000
Revision as of 19:58, 24 November 2025 by Pneyron (talk | contribs) (Pneyron moved page User:Pneyron/PMEM-environment to PMEM-environment: Make it public)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search


Warning.png Warning

This tutorial was written before the release of Debian 11 Bullseye and kadeploy 3.7. Some parts are outdated, but it is still a good example of how to write a kameleon recipe for experimenting on PMEMs.

Generating an environment with the PMEM tools

We will build a PMEM environment thanks to the kameleon tool.

Kameleon is one of the tools to build Grid'5000 environments, along with tgz-g5k. Contrary to tgz-g5k, it allows to script the environment build process, and is lighter (build environment in a VM, rather than on a deployed node).

Kameleon's documentation:

Prepare the kameleon tool

Warning.png Warning

The kameleon software and required dependencies are now installed in the default environment on the nodes. It is not anymore required to install it with gem. If it was installed with gem in the past, it should be removed: 

gem uninstall --user-install -x kameleon-builder ffi psych

First, get a job on a node (any x86-64 node could do)
pneyron@fgrenoble:~$ oarsub -I
# Filtering out exotic resources (drac, troll, yeti).
# Set default walltime to 3600.
OAR_JOB_ID=1931031
# Interactive mode: waiting...
# Starting...
[OAR] OAR_JOB_ID=1931031
[OAR] Your nodes are:
      dahu-20.grenoble.grid5000.fr*32
Add the Grid'5000 environment recipes to your kameleon repositories
pneyron@dahu-20:~$ kameleon repo add grid5000 https://gitlab.inria.fr/grid5000/environments-recipes
Cloning into '/home/pneyron/.kameleon.d/repos/grid5000'...
warning: redirecting to https://gitlab.inria.fr/grid5000/environments-recipes.git/
remote: Enumerating objects: 144, done.
remote: Counting objects: 100% (144/144), done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 15154 (delta 67), reused 110 (delta 44), pack-reused 15010
Receiving objects: 100% (15154/15154), 1.90 MiB | 14.11 MiB/s, done.
Resolving deltas: 100% (9391/9391), done.
In the case your already added the repository previously, mind updating it
pneyron@dahu-20:~$ kameleon repo update grid5000
[...]

Create a customized environment recipe

We create a debiantesting-x64-pmem recipe, which will build on top of the debiantesting-x64-min environment, which is maintained by the Grid'5000 tech team.

Using Debian testing and not Debian stable version 10 was important, since the support of PMEM in Debian 10 lacks the latest functionalities provided by more recent Linux kernels (e.g. kmem). After the release of Debian 11, this might be a bit different but we keep using Debian testing in order to get the latest version of software.

Instead of extending the debiantesting-x64-min recipe and thus building our environment from scratch (which means running the debian installer, etc), we choose to use the from_grid5000_environment/base recipe, which will let kameleon extract the already built debiantesting-x64-min environment tarball and add our stuff on top of it. As a reminder, kameleon does all that in a VM.

Create the debiantesting-x64-pmem recipe, which extends from_grid5000_environment/base
pneyron@dahu-20:~$ mkdir debiantesting-x64-pmem
pneyron@dahu-20:~$ cd debiantesting-x64-pmem
pneyron@dahu-20:~/debiantesting-x64-pmem$ kameleon new debiantesting-x64-pmem grid5000/from_grid5000_environment/base
      create  grid5000/from_grid5000_environment/base.yaml
      create  grid5000/steps/backend/qemu.yaml
      create  grid5000/steps/backend/VM.yaml
      create  grid5000/steps/aliases/defaults.yaml
      create  grid5000/steps/checkpoints/qemu.yaml
      create  grid5000/steps/bootstrap/prepare_ssh_to_out_context.yaml
      create  grid5000/steps/bootstrap/download_upstream_tarball.yaml
      create  grid5000/steps/bootstrap/create_appliance.yaml
      create  grid5000/steps/bootstrap/prepare_appliance.yaml
      create  grid5000/steps/bootstrap/start_qemu.yaml
      create  grid5000/steps/disable_checkpoint.yaml
      create  grid5000/steps/export/save_appliance_VM.yaml
      create  grid5000/steps/export/create_kadeploy_environment.yaml
      create  grid5000/steps/data/helpers/create_appliance.py
      create  grid5000/steps/data/qemu-sendkeys.rb
      create  grid5000/steps/data/helpers/export_appliance.py
      create  grid5000/steps/data/helpers/kaenv-customize.py
      create  grid5000/steps/env/bashrc
      create  grid5000/steps/env/functions.sh
      create  debiantesting-x64-pmem.yaml
Adapt the debiantesting-x64-pmem recipe to add our stuff

Edit the generated recipe file: debiantesting-x64-pmem.yaml, to make Grid'5000 LDAP and home directories service available, add the PMEM tools and setup an additional postinstall that will switch the PMEM to AppDirect during the deployment (PMEM is in Memory mode by default).

Required changes are high-lighted. 

#==============================================================================
# vim: softtabstop=2 shiftwidth=2 expandtab fenc=utf-8 cc=81 tw=80
#==============================================================================
#
# DESCRIPTION: debiantesting-x64-pmem
#
#==============================================================================
# This recipe extends another. To look at the step involed, run:
#   kameleon dryrun debiantesting-x64-pmem.yaml
# To see the variables that you can override, use the following command:
#   kameleon info debiantesting-x64-pmem.yaml
---
extend: grid5000/from_grid5000_environment/base.yaml

global:
  ### Grid'5000 environment information
  ## (Uncomment and change any line if needed)

  ## Frontend to run kaenv3 on
  #grid5000_frontend: "frontend"

  ## Site used in the build
  #grid5000_site: "grenoble"

  ## Environment to build from
  grid5000_environment_import_name: debiantesting-x64-min
  #grid5000_environment_import_user: "deploy"
  #grid5000_environment_import_version: ""

  ## New environment description
  #grid5000_environment_export_name: "$${kameleon_recipe_name}"
  #grid5000_environment_export_format: "tar.zstd"
  grid5000_environment_export_description: "debian testing with pmem tools"

  ## Set where to store the environment and the assiated kadeploy URL base
  #grid5000_environment_export_dir: "$HOME/public/"
  #grid5000_environment_export_baseurl: "local://$HOME/public/"

  ## Optionaly, the environment postinstall script can be changed, e.g. to
  ## enable NFS homes, LDAP accounts, if not enabled in the imported env.
  grid5000_environment_export_postinstall_script: "g5k-postinstall --net debian --fstab nfs --restrict-user current"

  ## Optionaly, an additional postinstall can be given, e.g. to do some custom
  ## operations. Use the following variables to set the archive name and script.
  grid5000_environment_export_additional_postinstall_archive: "$${kameleon_recipe_name}-postinstall.tar.gz"
  grid5000_environment_export_additional_postinstall_script: "pmem-setup.sh"
  ## The recipe will have to prepare the additional postinstall content in a
  ## directory to create in the local context and name "additional_postinstall"
  ## by default (next line to change it). The archive is created by the export.
  #grid5000_environment_export_additional_postinstall_dir: "additional_postinstall"

  ### Target machine/CPU architecture
  ## If building an environment for another architecture than x86_64, uncomment
  ## and adapt the next lines.
  ## The following works for ARM64 machines, just uncomment for such machines.
  #arch: aarch64
  #qemu_arch: aarch64
  #qemu_uefi: true

  ### You can add below any other global variable definition
  ## See the variables which can be overloaded, by running:
  ##   kameleon info debiantesting-x64-pmem.yaml
  ## Or define any new variable you would need. e.g.:
  #my_variable: my_value

bootstrap:
  ### The bootstrap section takes in charge the import of the Grid'5000
  ## environment to customize. No modification should be needed here.
  - "@base"

setup:
  # We first need to setup Grid'5000 LDAP and NFS, since this is not done in
  # the debiantesting-x64-min environment, nor by g5k-postinstall (See
  # https://intranet.grid5000.fr/bugzilla/show_bug.cgi?id=11839).
  - prepare_user_access:
    - update-system:
      - exec_in: apt-get -y update && apt-get -y upgrade
    - install_nfs_service:
      - exec_in: apt-get install -y nfs-common autofs
    - install_ldap_service:
      - exec_in: apt-get install -y nslcd
    - setup_ldap:
      - exec_in: mkdir -p /etc/ldap/certificates/
      - local2in:
        - /etc/ldap/certificates/ca2019.grid5000.fr.cert
        - /etc/ldap/certificates/ca2019.grid5000.fr.cert
      - local2in:
        - /etc/ldap/ldap.conf 
        - /etc/ldap/ldap.conf 
      - local2in:
        - /etc/nsswitch.conf
        - /etc/nsswitch.conf
      - local2in:
        - /etc/nslcd.conf
        - /etc/nslcd.conf
  # Next, we install the PMEM tools and some other utilities in the environment.
  # Feel free to add more here to have more packages installed or prepare some
  # additional configurations in the environment before deployment.
  - install_tools:
    - install_pmem_tools:
      - exec_in: apt-get install -y ipmctl ndctl daxctl
    - install_some_other_common_tools:
      - exec_in: apt-get install -y sudo vim-nox hwloc-nox htop dstat man bash-completion
  # Last, we prepare an additional postinstall which will configure the PMEM is AppDirect Mode
  # during the deployment, so that the deployed nodes are in AppDirect mode right away,
  # avoiding the need to do the mode switch and reboot again after deploying.
  # We also make the postinstall add the deactivation of the dax_pmem_compat driver, using
  # the daxctl migrate-device-model command.
  # The export section will make an archive of the directory with the script. The archive path
  # will be set in the resulting environment description.
  - prepare_additional_postinstall:
    - create_pmem_setup_script:
      - exec_local: |
          rm -rf $${grid5000_environment_export_additional_postinstall_dir}
          mkdir $${grid5000_environment_export_additional_postinstall_dir}
          cat <<'EOF' > $${grid5000_environment_export_additional_postinstall_dir}/$${grid5000_environment_export_additional_postinstall_script}
          #!/bin/bash
          # Postinstall script, to have the node in the wanted PMEM configuration right away after the deployment (avoid additional reboots)
          #
          # Make stdout&err display in the console as well as in kadeploy logs.
          exec 1> >(tee -a /dev/console)
          exec 2> >(tee -a /dev/console 1>&2)
          set -x -e
          # install ipmctl in the deployment MiniOS
          apt-get update
          apt-get install -y ipmctl
          ipmctl version
          # Look for any defined AppDirect region: if any, this means the node was already configured in AppDirect mode (e.g. in a previous deployment)
          if ipmctl show -region | tee /dev/console | grep -q AppDirect; then
            echo "=> PMEM already in AppDirect Mode"
          else
            echo "=> Switch PMEM to AppDirect Mode..."
            # Switch PMEM to AppDirect, after this the node needs to reboot for the mode switch to actually happen.
            # For that purpose, the kadeploy3 command must be called with the --no-kexec option.
            ipmctl create -force -goal PersistentMemoryType=AppDirect
          fi
          # Installing daxctl
          apt-get install -y daxctl
          # Run the daxctl migrate-device-model command and then copy the generated file to the deployed OS filesystem.
          daxctl migrate-device-model
          cp -v /etc/modprobe.d/daxctl.conf /mnt/dest/etc/modprobe.d/
          EOF
          chmod 755 $${grid5000_environment_export_additional_postinstall_dir}/$${grid5000_environment_export_additional_postinstall_script}
export:
  ### The export section takes in charge the export of your customized Grid'5000
  ## environment. No modification should be needed here.
  - "@base"

One can easily add more in the environment, for instance installing more software in the install_tools step, or adding any other steps in the setup section.

See the Environment creation page for more information on how to write a kameleon recipe.

See also the Advance Kadeploy page to find out how to debug the additional postinstall script, using a kadeploy breakpoint and kaconsole.

Warning.png Warning

Mind that the proposed PMEM configuration (100% AppDrirect, deactivation of the dax_pmem_compat driver) is only an example. This might not be relevant for your experiment !!!

Build the environment

We build on our dahu node (regardless of the fact that it has or not PMEM hardware).

In the job shell, we run: 

[pneyron@dahu-20 debiantesting-x64-pmem](1931031-->58mn)$ kameleon build debiantesting-x64-pmem.yaml
Creating kameleon build directory : /home/pneyron/debiantesting-x64-pmem/build/debiantesting-x64-pmem
Starting build recipe 'debiantesting-x64-pmem.yaml'
Step 1 : bootstrap/_init_bootstrap/_init_0_create_appliance
...
...
...
Step 29 : export/create_kadeploy_environment/create_kaenv_file
--> Running the step...
Step 30 : export/create_kadeploy_environment/copy_files
--> Running the step...
[local] Copying grid5000 environment files to /home/pneyron/public/
[local] '/home/pneyron/debiantesting-x64-pmem/build/debiantesting-x64-pmem/debiantesting-x64-pmem.tar.gz' -> '/home/pneyron/public/debiantesting-x64-pmem.tar.gz'
[local] 'debiantesting-x64-pmem.dsc' -> '/home/pneyron/public/debiantesting-x64-pmem.dsc'
[local] '/home/pneyron/debiantesting-x64-pmem/build/debiantesting-x64-pmem/debiantesting-x64-pmem-postinstall.tar.gz' -> '/home/pneyron/public/debiantesting-x64-pmem-postinstall.tar.gz'
Step create_kadeploy_environment took: 2 secs
Step 31 : export/_clean_export/_clean_1_start_vm
--> Running the step...
Step 32 : export/_clean_export/_clean_0_delete_initial_image_at_the_end
--> Running the step...
Step _clean_export took: 0 secs

Successfully built 'debiantesting-x64-pmem.yaml'
Total duration : 156 secs

Our environment is now ready (in our public webspace) to be deployed on a troll node!

  • environment description file: /home/pneyron/public/debiantesting-x64-pmem.dsc (this is a YAML file, formerly using the .yaml extension)
  • environment image: /home/pneyron/public/debiantesting-x64-pmem.tar.gz
  • environment additional postinstall: /home/pneyron/public/debiantesting-x64-pmem-postinstall.tar.gz
Warning.png Warning

the environment image now use a tar.zst file extension instead of tgz, since zst is now used for the compression instead of gzip

We quit this job, we will create a new one on troll nodes and of type deploy.

Deploy the environment

Reserve a troll node for 2h in a deploy job
pneyron@fgrenoble:~/debiantesting-x64-pmem$ oarsub -t exotic -p "cluster='troll'" -t deploy -l nodes=1,walltime=2 -I
# Include exotic resources in the set of reservable resources (this does NOT exclude non-exotic resources).
OAR_JOB_ID=1931038
# Interactive mode: waiting...
# Starting...
[OAR] OAR_JOB_ID=1931038
[OAR] Your nodes are:
      troll-2.grenoble.grid5000.fr*32

[pneyron@fgrenoble debiantesting-x64-pmem](1931038-->119mn)$
Open the console of the node

Since the deployment is pretty long on the troll nodes, we open the console help our patience... 

pneyron@fgrenoble:~$ kaconsole3 -m troll-2
Deploy the environment on the node

Run the deployment with the --no-kexec option, so that the real reboot makes the PMEM switch to the AppDirect mode take effect during the deployment. 

[pneyron@fgrenoble debiantesting-x64-pmem](1931038-->119mn)$ kadeploy3 --no-kexec -a ~/public/debiantesting-x64-pmem.dsc -f $OAR_NODE_FILE -k
Deployment #D-ea5e7cb0-da6a-4b46-8726-f9f41ee2c115 started
Grab the key file /home/pneyron/.ssh/authorized_keys
Grab the tarball file /home/pneyron/public/debiantesting-x64-pmem.tar.gz
Grab the postinstall file server:///grid5000/postinstalls/g5k-postinstall.tgz
Launching a deployment on troll-2.grenoble.grid5000.fr
...
...
...
End of step Deploy[BootNewEnvHardReboot] after 196s
End of deployment for troll-2.grenoble.grid5000.fr after 526s
End of deployment on cluster troll after 526s
Deployment #D-ea5e7cb0-da6a-4b46-8726-f9f41ee2c115 done

The deployment is successful on nodes
troll-2.grenoble.grid5000.fr
[pneyron@fgrenoble debiantesting-x64-pmem](1931038-->109mn)$

Well done, we can now ssh to the machine with our user account or root. The PMEM is in AppDirect mode, we can use the ipmctl, ndctl or daxctl tools to configure it and begin experimenting.

Retrieved from "https://www.grid5000.fr/mediawiki/index.php?title=PMEM-environment&oldid=108078"