PMEM: Difference between revisions
No edit summary |
No edit summary |
||
Line 1: | Line 1: | ||
Some nodes of Grid'5000 feature the new | __TOC__ | ||
Some [[Hardware#PMEM_size_per_node|nodes of Grid'5000]] feature the new '''Persistent Memory''' technology. As of writing this page, the [[Grenoble:Hardware#troll|troll]] cluster in Grenoble is equipped. | |||
= Forewords = | = Forewords = | ||
Line 49: | Line 50: | ||
See the man pages or [[#References|external documentations]] of the tools to use them. | See the man pages or [[#References|external documentations]] of the tools to use them. | ||
Please mind that '''when a job is terminated, the nodes of the job are automatically reconfigured to the default mode of operation, that is Memory Mode'''. | For instance to change to App Direct mode, with DIMMs interleaved, one can run: | ||
{{Term|location=troll-2|cmd=<code class="command">ipmctl create -goal MemoryMode=100</code>}} | |||
And then reboot. | |||
Reboot time of the machine is pretty long (~ 10 minutes), so be patient. You may look at the console to follow the progress | |||
{{Term|location=fgrenoble|cmd=<code class="command">kaconsole -m </code><code classe="replace">troll-2</code>}} | |||
; Important notes | |||
* Please mind that '''when a job is terminated, the nodes of the job are automatically reconfigured to the default mode of operation, that is Memory Mode'''. | |||
* Please mind that '''sudo-g5k is of no help to experiment with the App Direct mode''', since rebooting the node will terminate the job. Using the App Direct mode requires kadeploying. | |||
= References = | = References = | ||
Line 66: | Line 76: | ||
* VirtIO-PMEM https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.3-VirtIO-PMEM | * VirtIO-PMEM https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.3-VirtIO-PMEM | ||
* https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00074717en_us&docLocale=en_US | * https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00074717en_us&docLocale=en_US | ||
Revision as of 11:53, 18 February 2020
Some nodes of Grid'5000 feature the new Persistent Memory technology. As of writing this page, the troll cluster in Grenoble is equipped.
Forewords
This Persistant Memory technology is known by many different names, e.g.
- nvdimm (generic term, → nvdimm-N with a battery, nvdimm-P...)
- SCM (storage class memory)
- PMM/PMEM
In the rest of this document, we'll use the PMEM acronym.
The current available PMEM technology available in Grid'5000 is Intel's Optane DC Permanent Memory. Other vendors may provide PMEM in the future (IBM, HPE Memristor ?). PMEM has been also available for tests in emulator such as qemu for a long time.
This technology consists in DIMMs (just like DRAM) but offering a different set of characteristics:
- It fills the gap between memory and storage: RAM <x10< PMEM <x100< NVMe SSD in terms of latency
- Persistence: can be use as (persistent) memory or filesystem on steroids
- Byte addressable, zero-copy memory mapping
- No energy consumption on idle (and more than RAM when used)
- Lower price per GB compared to DRAM, larger memory sizes than DRAM
Intel's PMEM settings
Intel's PMEM can be configured in 2 modes:
- Memory
- Just more RAM, no persistence. DRAM serves as cache (disappear for the system viewpoint).
- App direct
- Many choices of configuration:
- DIMMs interleave option in the region (change needs reboot)
- region splits in namespaces (change may need reboot)
- sector, fsdax, devdax, kmem (kmem not available before Linux 5.1)
- Mix mode
- It is also possible to allocate part of the memory to Memory Mode and part of it to App Direct
In order to change the configuration (e.g. from Memory mode to App Direct mode, or vice versa), a reboot of the machine is needed.
Grid'5000 setup for experimentation
The choice in Grid'5000 as been to configure PMEM in Memory Mode by default. That means that the PMEM is in Memory Mode (appear just like RAM) in the Grid'5000 default environment (when not deploying).
We encourage users who wants to experiments with the App direct mode to deploy a very recent system (e.g. Debian testing), in order to benefit from the latest support for PMEM.
To that purpose, jobs need to be of the deploy type, and kadeploy must be used:
Then:
Once a node is deployed, once can connect as root, install the PMEM software and possibly change the configuration and reboot to apply it. The PMEM software are:
- ipmctl: tool to change the config of Intel's PMEM
- ndctl: tool to configure PMEM in App Direct mode
- daxctl: tool to configure the PMEM direct access (dax)
See the man pages or external documentations of the tools to use them.
For instance to change to App Direct mode, with DIMMs interleaved, one can run:
And then reboot.
Reboot time of the machine is pretty long (~ 10 minutes), so be patient. You may look at the console to follow the progress
- Important notes
- Please mind that when a job is terminated, the nodes of the job are automatically reconfigured to the default mode of operation, that is Memory Mode.
- Please mind that sudo-g5k is of no help to experiment with the App Direct mode, since rebooting the node will terminate the job. Using the App Direct mode requires kadeploying.
References
- https://docs.pmem.io/
- https://software.intel.com/pmem
- https://software.intel.com/en-us/articles/quick-start-guide-configure-intel-optane-dc-persistent-memory-on-linux
- https://software.intel.com/en-us/videos/provisioning-intel-optane-dc-persistent-memory-modules-in-linux
- https://www.youtube.com/watch?v=BShO6h8Lc1s
- https://www.youtube.com/watch?v=UTVt_AZmWjM
- https://github.com/intel/ipmctl
- https://hal.inria.fr/hal-02173336/document
- https://nvdimm.wiki.kernel.org/
- https://stevescargall.com/2019/07/09/how-to-extend-volatile-system-memory-ram-using-persistent-memory-on-linux/
- https://www.dell.com/support/manuals/fr/fr/frbsdt1/poweredge-r640/idrac_3.36.36.36_racadm_ar_referenceguide/biospmcreategoalconfigpmpersistentpercentage-read-or-write
- VirtIO-PMEM https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.3-VirtIO-PMEM
- https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00074717en_us&docLocale=en_US