![[PukiWiki]](image/bc_logo.png "[PukiWiki]"){kind=logo}
This article explains how to install CentOS 6.4 and CUDA5 on a PC, which is equipped with Tesla K20.
The specification of hardware is shown below
- CPU: Core i7 3770(3.4GHz,4core/8thread)
- Memory: 32GB(DDR3-12800 8GBx4)
- HDD: 1TB(SATA,7200rpm)
- GPU: ETSK20-5GER(nvidia Tesla K20c)
Since ETSK20-5GER does not have terminals of display outputs, such as D-SUB, DVI, HDMI, and so on, use the terminals of the mother board are used. To use the terminals, at the configuration of UEFI, the Primary Display is On Board.
The settings of installing CentOS 6.4 are shown below.
- Language & Keyboard: Japanese
- Package: Standard Desktop Install
- HDD Partition: Using whole disk, the standard configuration
- Network: DHCP
To install CentOS, there is no customization (modification) on packages.
Please follow the instruction that the installer of CnetOS provides.
Both Firewall and SELinux are configured as Disable.
This part describes how to prepare for installing CUDA after the installation of CentOS is completed.
The Kernel version of CentOS, which has been just installed, is shown below.
$ uname -a Linux localhost.localdomain 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
To update Kernel of CentOS, please apply to yum command.
# yum update
After updating Kernel is completed, please reboot the OS and make sure that the updated Kernel works.
# reboot
After rebooting the CnetOS, please check the newer version of Kernel is installed. The newer version of Kernel is shown below.
$ uname -a Linux localhost.localdomain 2.6.32-358.6.2.el6.x86_64 #1 SMP Thu May 16 20:59:36 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Create a new user. Then, add the new user to sudo account.
# useradd beat # passwd beat
# /usr/sbin/visudo ...<Omitted> ## user MACHINE=COMMANDS ## ## The COMMANDS section may have other options added to it. ## ## Allow root to run any commands anywhere root ALL=(ALL) ALL beat ALL=(ALL) ALL
The addition of packages and modification of configuration are not directly related to install CUDA.
However, to make installing process easier, install ntpd and openssh.
Install ntpd.
# yum install ntp
Add ntp server.
# vi /etc/ntp/step-tickers # List of servers used for initial synchronization. ntp.nict.jp ntp.jst.mfeed.ad.jp ntp.ring.gr.jp
Boot ntpd.
# /etc/init.d/ntpd start
ntpd is added to the service.
# chkconfig --level 3 ntpd on # chkconfig --level 5 ntpd on
Install openssh.
If openssh is already installed, please skip this part of the instruction.
# yum install openssh
PermitRootLogin and PermitEmptyPasswords, which are located at the inside of the directory /etc/ssh/sshd_config, are set as No.
# vi /etc/ssh/sshd_config ... <Omitted> # Authentication: #LoginGraceTime 2m PermitRootLogin no #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 ... <Omitted> # To disable tunneled clear text passwords, change to no here! #PasswordAuthentication yes PermitEmptyPasswords no PasswordAuthentication yes ...
Boot sshd.
# /etc/init.d/sshd start
Add service to sshd.
# chkconfig --level 5 sshd on # chkconfig --level 3 sshd on
This section describes how to install the driver for Tesla K20c.
To install the deriver for Tesla K20c, the Developer Tools and packages of CentOS are installed. Then, these packages are configured if they are necessary.
# yum groupinstall 'Development tools'
An error will occur if X is booted during the diver for Tesla K20c and CUDA5, so runlevel is changed to runlevel 3. Then, it is rebooted.
# vi /etc/inittab ...<Omitted> # Default runlevel. The runlevels used are: # 0 - halt (Do NOT set initdefault to this) # 1 - Single user mode # 2 - Multiuser, without NFS (The same as 3, if you do not have networking) # 3 - Full multiuser mode # 4 - unused # 5 - X11 # 6 - reboot (Do NOT set initdefault to this) # id:3:initdefault: # reboot
After rebooting, NVIDIA-Linux-x86_64-319.23.run is downloaded from the URL shown below.
http://www.nvidia.co.jp/object/linux-display-amd64-319.23-driver-jp.html
Execute NVIDIA-Linux-x86_64-319.23.run, and if you agree to the license agreement, then the instillation process of the driver for Tesla K20c is completed. After the installation is finished, the version file is created at the directory of /proc/driver/nvidea/.
# chmod +x NVIDIA-Linux-x86_64-319.23.run # ./NVIDIA-Linux-x86_64-319.23.run # cat /proc/driver/nvidia/version NVRM version: NVIDIA UNIX x86_64 Kernel Module 304.54 Sat Sep 29 00:05:49 PDT 2012 GCC version: gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC)
Caution:
The driver is needed to be re-built after the kernel update and reboot process are done.
The re-build process is required since the updated kernel cannot read the nvidea driver. Without updating the driver, the CUDA5 Samples are not be executed.
For the reference, the version of the new driver and the directory for the new nvidia driver are listed below.
# uname -a Linux tesla.beatcraft.com 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28 17:19:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux # cat /proc/driver/nvidia/version NVRM version: NVIDIA UNIX x86_64 Kernel Module 319.23 Thu May 16 19:36:02 PDT 2013 GCC version: gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC)
To read the driver, create a file at /etc/init.d/nvidea, and let runlevel3 and 5 be executed as the system is booted. For more details of the startup script, please visit the URL shown below.
http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/index.html#install-cuda-software
# vi /etc/init.d/nvidia
#!/bin/bash
/sbin/modprobe nvidia
if [ "$?" -eq 0 ]; then
# Count the number of NVIDIA controllers found.
NVDEVS=`lspci | grep -i NVIDIA`
N3D=`echo "$NVDEVS" | grep "3D controller" | wc -l`
NVGA=`echo "$NVDEVS" | grep "VGA compatible controller" | wc -l`
N=`expr $N3D + $NVGA - 1`
for i in `seq 0 $N`; do
mknod -m 666 /dev/nvidia$i c 195 $i
done
mknod -m 666 /dev/nvidiactl c 195 255
else
exit 1
fi
# cd /etc/rc5.d/
# ln -s S99nvidia ../init.d/nvidia
# cd /etc/rc3.d/
# ln -s S99nvidia ../init.d/nvidia
This section shows how to install CUDA5 and executes a sample CUDA program.
Download CUDA for CentOS6 64bit version (cuda_5.0.35_linux_64_rhel6.x-1.run) from URL below.
https://developer.nvidia.com/cuda-downloads
Add a PATH for CUDA5 in .bashrc. This is a precautionary measure. If the sample programs can not find the PATH, the samples cannot be installed correctly.
$ cd /home/beat
$ vi .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific aliases and functions
export PATH=/usr/local/cuda-5.0/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-5.0/lib64:/usr/local/cuda-5.0/lib:$LD_LIBRARY_PATH
# source .bashrc
To build the samples, these packages listed below are required.
# yum install freeglut-devel.x86_64 libXi-devel.x86_64 mpich2-devel.x86_64 libXmu-devel.x86_64
Execute cuda_5.0.35_linux_64_rhel6.x-1.run. Specify the directory that the sample program is installed at the default directory, which is /usr/local/cuda-05.
# chmod +x cuda_5.0.35_linux_64_rhel6.x-1.run # ./cuda_5.0.35_linux_64_rhel6.x-1.run
Sample programs are installed at the directories of /usr/local/cuda-5.0/samples and /root/NVUDA-5.0_Samples.
To let a root user build the samples, add ld.so.conf.d/cuda.conf, then, execute ldconfig.
# cat /etc/ld.so.conf.d/cuda.conf /usr/local/cuda-5.0/lib /usr/local/cuda-5.0/lib64 # ldconfig
Let a user build and execute NVIDIA_CUDA-5.0_Samples.
Copy the file NVIDIA_CUDA-5.0_Samples from /root/NVIDIA_CUDA-5.0_Samples. Change the owner of the file NVIDIA_CUDA-5.0_Samples to a user. Then, execute make.
# cp -afr /root/NVIDIA_CUDA-5.0_Samples /home/beat/. # cd /home/beat # chown -R beat.beat NVIDIA_CUDA-5.0_Samples $ cd ~/NVIDIA_CUDA-5.0_Samples $ make
After the build process is completed, Change the directory to /NVIDIA_CUDA-5.0_Samples/bin/linux/release/. Then, execute sample programs, which are listed on Running the Binaries of the NVIDIA CUDA Getting Started Guide for Linux. The URL of this page is show below. The names of sample programs, which are going to be executed are deviceQuery and bandwidthTest.
http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/index.html#running-binaries
To look at the outputs shown below, both deviceQuery and bandwidthTest recognize Tesla K20c.
$ cd ~/NVIDIA_CUDA-5.0_Samples/bin/linux/release/
$ ./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "Tesla K20c"
CUDA Driver Version / Runtime Version 5.0 / 5.0
CUDA Capability Major/Minor version number: 3.5
Total amount of global memory: 4800 MBytes (5032706048 bytes)
(13) Multiprocessors x (192) CUDA Cores/MP: 2496 CUDA Cores
GPU Clock rate: 706 MHz (0.71 GHz)
Memory Clock rate: 2600 Mhz
Memory Bus Width: 320-bit
L2 Cache Size: 1310720 bytes
Max Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536,65536), 3D=(4096,4096,4096)
Max Layered Texture Size (dim) x layers 1D=(16384) x 2048, 2D=(16384,16384) x 2048
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Maximum sizes of each dimension of a block: 1024 x 1024 x 64
Maximum sizes of each dimension of a grid: 2147483647 x 65535 x 65535
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Enabled
Device supports Unified Addressing (UVA): Yes
Device PCI Bus ID / PCI location ID: 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 5.0, CUDA Runtime Version = 5.0, NumDevs = 1, Device0 = Tesla K20c
$ ./bandwidthTest
[CUDA Bandwidth Test] - Starting...
Running on...
Device 0: Tesla K20c
Quick Mode
Host to Device Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 6431.7
Device to Host Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 6396.1
Device to Device Bandwidth, 1 Device(s)
PINNED Memory Transfers
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 143752.5
After the install and configuration of CUDA, the remaining Hard Disk space is shown below.
# df -h
Filesystem Size Used Avail Use% マウント位置
/dev/mapper/vg_beattesla-lv_root
50G 4.3G 45G 9% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 485M 66M 394M 15% /boot
/dev/mapper/vg_beattesla-lv_home
852G 1.4G 808G 1% /home
- 2013/06/04 This article is initially uploaded.
- 2013/09/26 The tittle of the section is changed from "To download Tesla K20c driver" to "To download and install Tesla K20c driver." Caution is added at the section.
