GPUs on Grid5000
Purpose
This page presents how to use GPUs on Grid'5000 and how to install your own NVIDIA drivers and CUDA installation.
In this tutorial, we will first compile and use CUDA examples and in the second part, we will install NVIDIA drivers and compile CUDA 5 from a simple wheezy-x64-base environment.
Pre-requisite
- A basic knowledge of Grid'5000 is require, we suggest you to read Getting Started tutorial first.
- Information about hardware information and GPUs availability can be found on Special:G5KHardware.
GPU on Grid'5000
Using CUDA
Download and compile examples
GPU are available in:
- Grenoble (adonis)
- Lyon (orion)
- Lille (chirloute)
NVIDIA drivers 304.54 and CUDA 5.0 are installed by default on nodes.
You can reserve a node with GPU using OAR GPU property. For Grenoble and Lyon:
or for Lille:
![]() |
Warning |
---|---|
Please note that Lille GPU are shared between nodes and the oarsub command will differ from Grenoble or Lyon. When using GPUs at Lille, you may encounter some trouble, you can read more about Lille GPU on Lille:GPU. |
We will then download CUDA 5.0 samples and install them.
![]() |
node :
|
wget http://git.grid5000.fr/sources/cuda-samples_5.0.35_linux.run -P /tmp/ && cd /tmp |
You will be prompt to accept the EULA and for a installation path, we suggest you to use /tmp/samples/.
Then you can go to installation path, in our case /tmp/samples/
. If you list the directory, you will see a lot of folder starting with a number, these are CUDA examples. CUDA examples are describe in the document named Samples.html
. You might also want to have a look to doc
directory.
We will now compile examples, this will take a little time. From CUDA samples installation directory (/tmp/samples
), run make:
The process is complete when "Finished building CUDA samples" is printed. You should be able to run CUDA examples. We will try the one named Device Query
which is located in /tmp/samples/1_Utilities/deviceQuery/
. This sample enumerates the properties of the CUDA devices present in the system.
This is an example of the result on adonis cluster at Grenoble:
ebertoncello@adonis-2:/tmp/samples$ ./1_Utilities/deviceQuery/deviceQuery 1_Utilities/deviceQuery/deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 2 CUDA Capable device(s) Device 0: "Tesla T10 Processor" CUDA Driver Version / Runtime Version 5.0 / 5.0 CUDA Capability Major/Minor version number: 1.3 Total amount of global memory: 4096 MBytes (4294770688 bytes) (30) Multiprocessors x ( 8) CUDA Cores/MP: 240 CUDA Cores GPU Clock rate: 1296 MHz (1.30 GHz) Memory Clock rate: 800 Mhz Memory Bus Width: 512-bit Max Texture Dimension Size (x,y,z) 1D=(8192), 2D=(65536,32768), 3D=(2048,2048,2048) Max Layered Texture Size (dim) x layers 1D=(8192) x 512, 2D=(8192,8192) x 512 Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 16384 bytes Total number of registers available per block: 16384 Warp size: 32 Maximum number of threads per multiprocessor: 1024 Maximum number of threads per block: 512 Maximum sizes of each dimension of a block: 512 x 512 x 64 Maximum sizes of each dimension of a grid: 65535 x 65535 x 1 Maximum memory pitch: 2147483647 bytes Texture alignment: 256 bytes Concurrent copy and kernel execution: Yes with 1 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): No Device PCI Bus ID / PCI location ID: 12 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > Device 1: "Tesla T10 Processor" CUDA Driver Version / Runtime Version 5.0 / 5.0 CUDA Capability Major/Minor version number: 1.3 Total amount of global memory: 4096 MBytes (4294770688 bytes) (30) Multiprocessors x ( 8) CUDA Cores/MP: 240 CUDA Cores GPU Clock rate: 1296 MHz (1.30 GHz) Memory Clock rate: 800 Mhz Memory Bus Width: 512-bit Max Texture Dimension Size (x,y,z) 1D=(8192), 2D=(65536,32768), 3D=(2048,2048,2048) Max Layered Texture Size (dim) x layers 1D=(8192) x 512, 2D=(8192,8192) x 512 Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 16384 bytes Total number of registers available per block: 16384 Warp size: 32 Maximum number of threads per multiprocessor: 1024 Maximum number of threads per block: 512 Maximum sizes of each dimension of a block: 512 x 512 x 64 Maximum sizes of each dimension of a grid: 65535 x 65535 x 1 Maximum memory pitch: 2147483647 bytes Texture alignment: 256 bytes Concurrent copy and kernel execution: Yes with 1 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): No Device PCI Bus ID / PCI location ID: 10 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 5.0, CUDA Runtime Version = 5.0, NumDevs = 2, Device0 = Tesla T10 Processor, Device1 = Tesla T10 Processor
Install CUDA from a base environement
Reservation and deployment
We will now install up to date NVIDIA drivers and up to date CUDA (5.5).
First reserve a node and then deploy wheezy-x64-base
.
Once the deployment is terminated, you should be able to connect to node and download CUDA 5.5 installer to /tmp.
![]() |
node :
|
http_proxy=http://proxy.
site .grid5000.fr:3128 wget http://developer.download.nvidia.com/compute/cuda/5_5/rel/installers/cuda_5.5.22_linux_64.run -P /tmp/ |
When download is over, you can see installation options from CUDA installer
root@adonis-2:/tmp# sh cuda_5.5.22_linux_64.run --help Options: -help : Print help message -driver : Install NVIDIA Display Driver -uninstall : Uninstall NVIDIA Display Driver -toolkit : Install CUDA 5.5 Toolkit (default: /usr/local/cuda-5.5) -toolkitpath=<PATH> : Specify a custom path for CUDA location -samples : Install CUDA 5.5 Samples (default: /usr/local/cuda-5.5/samples) -samplespath=<PATH> : Specify a custom path for Samples location -silent : Run in silent mode. Implies acceptance of the EULA -verbose : Run in verbose mode -extract=<PATH> : Extract individual installers from the .run file to PATH -optimus : Install driver support for Optimus -override : Overrides the installation checks (compiler, lib, etc) -kernel-source-path=<PATH> : Points to a non-default kernel source location -tmpdir <PATH> : Use <PATH> as temporary directory - useful when /tmp is noexec
We can extract all installers from this one :
This will extract 3 installers:
- NVIDIA-Linux-x86_64-319.37.run: NVIDIA drivers
- cuda-linux64-rel-5.5.22-16488124.run: CUDA installer
- cuda-samples-linux-5.5.22-16488124.run: CUDA samples installer
Each of them also have installation options :
Installation
We are ready to install NVIDIA drivers and CUDA 5.5. We will also use gcc-4.6 which is needed for this installation.
![]() |
node :
|
CC=/usr/bin/gcc-4.6
sh NVIDIA-Linux-x86_64-319.37.run --accept-license --silent --disable-nouveau -X --kernel-name=`uname -r` |
NVIDIA drivers and CUDA 5.5 are successfully installed, but some more configuration is needed such as add /usr/local/cuda-5.5/bin
to $PATH
and add /usr/local/cuda-5.5/lib64
and /usr/local/cuda-5.5/lib
to $LD_LIBRARY_PATH
.
But this is not permanent after a disconnection from your ssh session, this changes will be lost. To make this permanent, you have to edit /etc/profile
and create a file under /etc/ld.so.conf.d/
directory
![]() |
node :
|
echo -e "/usr/local/cuda-5.5/lib\n/usr/local/cuda-5.5/lib64" > /etc/ld.so.conf.d/cuda.conf && ldconfig |
We almost done the installation. The last step have to add a script which will enable GPU and execute it at start-up.
- Create a file named
/usr/local/bin/enable-gpu
and copy the following script in it.
#!/bin/bash
if /sbin/modprobe nvidia; then
if [ "$?" -eq 0 ]; then
# Count the number of NVIDIA controllers found.
NVDEVS=`lspci | grep -i NVIDIA`
N3D=`echo "$NVDEVS" | grep "3D controller" | wc -l`
NVGA=`echo "$NVDEVS" | grep "VGA compatible controller" | wc -l`
N=`expr $N3D + $NVGA - 1`
for i in `seq 0 $N`; do
mknod -m 666 /dev/nvidia$i c 195 $i
done
mknod -m 666 /dev/nvidiactl c 195 255
else
exit 1
fi
fi
- Give execution rights to the script file:
- Finally, add a line in the /etc/rc.local file to execute your script at boot time.
![]() |
node :
|
sed -e "s/exit 0//" -i /etc/rc.local; echo -e "#Enable GPUs at boot time\nsh /usr/local/bin/enable-gpu\n\nexit 0" >> /etc/rc.local |
Check installation
To check if NVIDIA drivers are correctly installed, you can use nvidia-smi tool.
This is an example of the result on adonis cluster:
root@adonis-2:~# nvidia-smi Wed Dec 4 14:42:08 2013 +------------------------------------------------------+ | NVIDIA-SMI 5.319.37 Driver Version: 319.37 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla T10 Proce... Off | 0000:0A:00.0 N/A | N/A | | N/A 36C N/A N/A / N/A | 3MB / 4095MB | N/A Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla T10 Proce... Off | 0000:0C:00.0 N/A | N/A | | N/A 36C N/A N/A / N/A | 3MB / 4095MB | N/A Default | +-------------------------------+----------------------+----------------------+
To check CUDA installation we will compile CUDA examples.
- Install dependencies
![]() |
node :
|
http_proxy=http://proxy.
site .grid5000.fr:3128 apt-get update && apt-get -y install freeglut3-dev libxmu-dev libxi-dev |
- Uncompress samples and compile them
You will be prompt to accept the EULA and for a installation path. In this example we will install examples in /usr/local/cuda-5.5/samples
- To compile CUDA samples, go to installation directory and then type make.
- Once compilation is over, you should be able to run some samples:
root@adonis-7:/tmp# /usr/local/cuda-5.5/samples/1_Utilities/deviceQuery/deviceQuery /usr/local/cuda-5.5/samples/1_Utilities/deviceQuery/deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 2 CUDA Capable device(s) Device 0: "Tesla T10 Processor" CUDA Driver Version / Runtime Version 5.5 / 5.5 CUDA Capability Major/Minor version number: 1.3 Total amount of global memory: 4096 MBytes (4294770688 bytes) (30) Multiprocessors, ( 8) CUDA Cores/MP: 240 CUDA Cores GPU Clock rate: 1296 MHz (1.30 GHz) Memory Clock rate: 800 Mhz Memory Bus Width: 512-bit Maximum Texture Dimension Size (x,y,z) 1D=(8192), 2D=(65536, 32768), 3D=(2048, 2048, 2048) Maximum Layered 1D Texture Size, (num) layers 1D=(8192), 512 layers Maximum Layered 2D Texture Size, (num) layers 2D=(8192, 8192), 512 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 16384 bytes Total number of registers available per block: 16384 Warp size: 32 Maximum number of threads per multiprocessor: 1024 Maximum number of threads per block: 512 Max dimension size of a thread block (x,y,z): (512, 512, 64) Max dimension size of a grid size (x,y,z): (65535, 65535, 1) Maximum memory pitch: 2147483647 bytes Texture alignment: 256 bytes Concurrent copy and kernel execution: Yes with 1 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): No Device PCI Bus ID / PCI location ID: 12 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > Device 1: "Tesla T10 Processor" CUDA Driver Version / Runtime Version 5.5 / 5.5 CUDA Capability Major/Minor version number: 1.3 Total amount of global memory: 4096 MBytes (4294770688 bytes) (30) Multiprocessors, ( 8) CUDA Cores/MP: 240 CUDA Cores GPU Clock rate: 1296 MHz (1.30 GHz) Memory Clock rate: 800 Mhz Memory Bus Width: 512-bit Maximum Texture Dimension Size (x,y,z) 1D=(8192), 2D=(65536, 32768), 3D=(2048, 2048, 2048) Maximum Layered 1D Texture Size, (num) layers 1D=(8192), 512 layers Maximum Layered 2D Texture Size, (num) layers 2D=(8192, 8192), 512 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 16384 bytes Total number of registers available per block: 16384 Warp size: 32 Maximum number of threads per multiprocessor: 1024 Maximum number of threads per block: 512 Max dimension size of a thread block (x,y,z): (512, 512, 64) Max dimension size of a grid size (x,y,z): (65535, 65535, 1) Maximum memory pitch: 2147483647 bytes Texture alignment: 256 bytes Concurrent copy and kernel execution: Yes with 1 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): No Device PCI Bus ID / PCI location ID: 10 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 5.5, CUDA Runtime Version = 5.5, NumDevs = 2, Device0 = Tesla T10 Processor, Device1 = Tesla T10 Processor Result = PASS
Please note that CUDA samples are described on the document named Samples.html located on samples installation directory.
Do not forget to backup your environment:
Using OpenCL
Intel MIC on Grid'5000
Reserve MIC on Nancy site
Configure Intel compiler
In order to run programs on Intel Xéon phi, you should compile with Intel compiler. An instance is available on /grid5000/compilers/icc13.2/ on Nancy's site. You must have a machine that can access to the token server and to the grid5000 nodes. If you have both, you should create licenses file (adapt with your server):
cat <<EOF >> ~/intel/licenses
SERVER license_server
ANY 28618
USE_SERVER
EOF
Then you should create a reverse:
With tow reverse command: |
Or by ssh configuration: Host accessuser Hostname access.grid5000.fr User Then connect to node: |
Note that port could be different: for example inria server is jetons.inria.fr and ports are 29030 and 34430.
Compile on host
Tutorial from: http://www.hpc.cineca.it/content/quick-guide-intel-mic-usage
with offload
Simple example can be find here: http://www.hpc.cineca.it/content/hellooffloadcpp
You can compile with:
Native
Simple example can be find here: http://www.hpc.cineca.it/content/hellonativecpp
You can compile with:
Login on mic:
Add LD_LIBRARY:
![]() |
grahite-mic0 :
|
export LD_LIBRARY_PATH=/grid5000/compilers/icc13.2/composer_xe_2013/mkl/lib/mic:${LD_LIBRARY_PATH} |
![]() |
grahite-mic0 :
|
export LD_LIBRARY_PATH=/grid5000/compilers/icc13.2/composer_xe_2013/tbb/lib/mic:${LD_LIBRARY_PATH} |
![]() |
grahite-mic0 :
|
export LD_LIBRARY_PATH=/grid5000/compilers/icc13.2/composer_xe_2013/lib/mic:${LD_LIBRARY_PATH} |
And execute: