GPUs on Grid5000: Difference between revisions
Ebertoncello (talk | contribs) |
Ebertoncello (talk | contribs) |
||
Line 192: | Line 192: | ||
== Check installation == | == Check installation == | ||
To check if NVIDIA drivers are correctly installed, you can use nvidia-smi tool. | |||
This is an exemple of the result on adonis cluster: | |||
root@adonis-4:~# nvidia-smi | |||
Wed Dec 4 14:42:08 2013 | |||
+------------------------------------------------------+ | |||
| NVIDIA-SMI 5.319.37 Driver Version: 319.37 | | |||
|-------------------------------+----------------------+----------------------+ | |||
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | |||
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | |||
|===============================+======================+======================| | |||
| 0 Tesla T10 Proce... Off | 0000:0A:00.0 N/A | N/A | | |||
| N/A 36C N/A N/A / N/A | 3MB / 4095MB | N/A Default | | |||
+-------------------------------+----------------------+----------------------+ | |||
| 1 Tesla T10 Proce... Off | 0000:0C:00.0 N/A | N/A | | |||
| N/A 36C N/A N/A / N/A | 3MB / 4095MB | N/A Default | | |||
+-------------------------------+----------------------+----------------------+ |
Revision as of 15:42, 4 December 2013
Purpose
This page presents how to use GPUs on Grid'5000 and how to install your own NVIDIA drivers and CUDA installation.
In this tutorial, we will first compile and use CUDA examples and in the second part, we will install NVIDIA drivers and compile CUDA 5 from a simple wheezy-x64-base environment.
Pre-requisite
- A basic knowledge of Grid'5000 is require, we suggest you to read Getting Started tutorial first.
- Information about hardware information and GPUs availability can be found on Special:G5KHardware.
Download and compile examples
GPU are available in:
- Grenoble (adonis)
- Lyon (orion)
- Lille (chirloute)
NVIDIA drivers 304.54 and CUDA 5.0 are installed by default on nodes.
You can reserve a node with GPU using OAR GPU property. For Grenoble and Lyon:
or for Lille:
![]() |
Warning |
---|---|
Please note that Lille GPU are shared between nodes and the oarsub command will differ from Grenoble or Lyon. When using GPUs at Lille, you may encounter some trouble, you can read more about Lille GPU on Lille:GPU. |
We will then download CUDA 5.0 samples and install them.
![]() |
node :
|
wget http://git.grid5000.fr/sources/cuda-samples_5.0.35_linux.run -P /tmp/ && cd /tmp |
You will be prompt to accept the EULA and for a installation path, we suggest you to use /tmp/samples/.
Then you can go to installation path, in our case /tmp/samples/
. If you list the directory, you will see a lot of folder starting with a number, these are CUDA examples. CUDA examples are describe in the document named Samples.html
. You might also want to have a look to doc
directory.
We will now compile examples, this will take a little time. From CUDA samples installation directory (/tmp/samples
), run make:
The process is complete when "Finished building CUDA samples" is printed. You should be able to run CUDA examples. We will try the one named Device Query
which is located in /tmp/samples/1_Utilities/deviceQuery/
. This sample enumerates the properties of the CUDA devices present in the system.
This is an example of the result on adonis cluster at Grenoble:
ebertoncello@adonis-2:/tmp/samples$ ./1_Utilities/deviceQuery/deviceQuery 1_Utilities/deviceQuery/deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 2 CUDA Capable device(s) Device 0: "Tesla T10 Processor" CUDA Driver Version / Runtime Version 5.0 / 5.0 CUDA Capability Major/Minor version number: 1.3 Total amount of global memory: 4096 MBytes (4294770688 bytes) (30) Multiprocessors x ( 8) CUDA Cores/MP: 240 CUDA Cores GPU Clock rate: 1296 MHz (1.30 GHz) Memory Clock rate: 800 Mhz Memory Bus Width: 512-bit Max Texture Dimension Size (x,y,z) 1D=(8192), 2D=(65536,32768), 3D=(2048,2048,2048) Max Layered Texture Size (dim) x layers 1D=(8192) x 512, 2D=(8192,8192) x 512 Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 16384 bytes Total number of registers available per block: 16384 Warp size: 32 Maximum number of threads per multiprocessor: 1024 Maximum number of threads per block: 512 Maximum sizes of each dimension of a block: 512 x 512 x 64 Maximum sizes of each dimension of a grid: 65535 x 65535 x 1 Maximum memory pitch: 2147483647 bytes Texture alignment: 256 bytes Concurrent copy and kernel execution: Yes with 1 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): No Device PCI Bus ID / PCI location ID: 12 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > Device 1: "Tesla T10 Processor" CUDA Driver Version / Runtime Version 5.0 / 5.0 CUDA Capability Major/Minor version number: 1.3 Total amount of global memory: 4096 MBytes (4294770688 bytes) (30) Multiprocessors x ( 8) CUDA Cores/MP: 240 CUDA Cores GPU Clock rate: 1296 MHz (1.30 GHz) Memory Clock rate: 800 Mhz Memory Bus Width: 512-bit Max Texture Dimension Size (x,y,z) 1D=(8192), 2D=(65536,32768), 3D=(2048,2048,2048) Max Layered Texture Size (dim) x layers 1D=(8192) x 512, 2D=(8192,8192) x 512 Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 16384 bytes Total number of registers available per block: 16384 Warp size: 32 Maximum number of threads per multiprocessor: 1024 Maximum number of threads per block: 512 Maximum sizes of each dimension of a block: 512 x 512 x 64 Maximum sizes of each dimension of a grid: 65535 x 65535 x 1 Maximum memory pitch: 2147483647 bytes Texture alignment: 256 bytes Concurrent copy and kernel execution: Yes with 1 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): No Device PCI Bus ID / PCI location ID: 10 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 5.0, CUDA Runtime Version = 5.0, NumDevs = 2, Device0 = Tesla T10 Processor, Device1 = Tesla T10 Processor
Install CUDA from a base environement
Reservation and deployment
We will now install up to date NVIDIA drivers and up to date CUDA (5.5).
First reserve a node and then deploy wheezy-x64-base
.
Once the deployment is terminated, you should be able to connect to node and download CUDA 5.5 installer to /tmp.
![]() |
node :
|
http_proxy=http://proxy.
site .grid5000.fr:3128 wget http://developer.download.nvidia.com/compute/cuda/5_5/rel/installers/cuda_5.5.22_linux_64.run -P /tmp/ |
When download is over, you can see installation options from CUDA installer
root@adonis-2:/tmp# sh cuda_5.5.22_linux_64.run --help Options: -help : Print help message -driver : Install NVIDIA Display Driver -uninstall : Uninstall NVIDIA Display Driver -toolkit : Install CUDA 5.5 Toolkit (default: /usr/local/cuda-5.5) -toolkitpath=<PATH> : Specify a custom path for CUDA location -samples : Install CUDA 5.5 Samples (default: /usr/local/cuda-5.5/samples) -samplespath=<PATH> : Specify a custom path for Samples location -silent : Run in silent mode. Implies acceptance of the EULA -verbose : Run in verbose mode -extract=<PATH> : Extract individual installers from the .run file to PATH -optimus : Install driver support for Optimus -override : Overrides the installation checks (compiler, lib, etc) -kernel-source-path=<PATH> : Points to a non-default kernel source location -tmpdir <PATH> : Use <PATH> as temporary directory - useful when /tmp is noexec
We can extract all installers from this one :
This will extract 3 installers:
- NVIDIA-Linux-x86_64-319.37.run: NVIDIA drivers
- cuda-linux64-rel-5.5.22-16488124.run: CUDA installer
- cuda-samples-linux-5.5.22-16488124.run: CUDA samples installer
Each of them also have installation options :
Installation
We are ready to install NVIDIA drivers and CUDA 5.5. We will also use gcc-4.6 which is needed for this installation.
![]() |
node :
|
CC=/usr/bin/gcc-4.6
sh NVIDIA-Linux-x86_64-319.37.run --accept-license --silent --disable-nouveau -X --kernel-name=`uname -r` |
NVIDIA drivers and CUDA 5.5 are successfully installed, but some more configuration is needed such as add /usr/local/cuda-5.5/bin
to $PATH
and add /usr/local/cuda-5.5/lib64
and /usr/local/cuda-5.5/lib
to $LD_LIBRARY_PATH
.
But this is not permanent after a disconnection from your ssh session, this changes will be lost. To make this permanent, you have to edit /etc/profile
and create a file under /etc/ld.so.conf.d/
directory
![]() |
node :
|
echo -e "/usr/local/cuda-5.0/lib\n/usr/local/cuda-5.0/lib64" > /etc/ld.so.conf.d/cuda.conf && ldconfig |
We almost done the installation. The last step have to add a script which will enable GPU and execute it at start-up.
- Create a file named
/usr/local/bin/enable-gpu
and copy the following script in it.
#!/bin/bash
if /sbin/modprobe nvidia; then
if [ "$?" -eq 0 ]; then
# Count the number of NVIDIA controllers found.
NVDEVS=`lspci | grep -i NVIDIA`
N3D=`echo "$NVDEVS" | grep "3D controller" | wc -l`
NVGA=`echo "$NVDEVS" | grep "VGA compatible controller" | wc -l`
N=`expr $N3D + $NVGA - 1`
for i in `seq 0 $N`; do
mknod -m 666 /dev/nvidia$i c 195 $i
done
mknod -m 666 /dev/nvidiactl c 195 255
else
exit 1
fi
fi
- Give execution rights to the script file:
- Finally, add a line in the /etc/rc.local file to execute your script at boot time.
![]() |
node :
|
sed -e "s/exit 0//" -i /etc/rc.local; echo -e "#Enable GPUs at boot time\nsh /usr/local/bin/enable-gpu\n\nexit 0" >> /etc/rc.local |
Check installation
To check if NVIDIA drivers are correctly installed, you can use nvidia-smi tool.
This is an exemple of the result on adonis cluster:
root@adonis-4:~# nvidia-smi Wed Dec 4 14:42:08 2013 +------------------------------------------------------+ | NVIDIA-SMI 5.319.37 Driver Version: 319.37 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla T10 Proce... Off | 0000:0A:00.0 N/A | N/A | | N/A 36C N/A N/A / N/A | 3MB / 4095MB | N/A Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla T10 Proce... Off | 0000:0C:00.0 N/A | N/A | | N/A 36C N/A N/A / N/A | 3MB / 4095MB | N/A Default | +-------------------------------+----------------------+----------------------+