Hi, I am using slurm 17.11.3-2 version on a small ROCKS 7 cluster. I have two gpu nodes with nvidia driver 384.111 and opencl library installed. Moreover in /etc/OpenCL/vendors directory there are two files (nvidia.icd and intel.icd). Files are attached bellow. When I am submiting slurm script I get this output file (see output_intel_and_nvidia.icd.log). However when I remove nvidia.icd file to another location and leave only intel.icd file in /etc/OpenCL/vendors directory everything works fine. Also, what is interesting that my code with OpenCL library works also fine when I am running code directly from gpu node and every time I am not even using gpu memory. In addition to, you can see my slurm script attached to this message. Can someone explain how slurm interacts with .icd files and why code can not be completed (it stops at ~10 GB of RAM allocated instead of ~16-20 GB) when nvidia.icd persist?
nvidia.icd
Description: Binary data
An Vas 12 09:28:51 EET 2019 gpu-0-1.local /home/edvinas/JOB_SCRIPTS Number of platforms 1 Platform Name Intel(R) OpenCL Platform Vendor Intel(R) Corporation Platform Version OpenCL 1.2 Platform Profile FULL_PROFILE Platform Extensions cl_khr_icd cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_depth_images cl_khr_3d_image_writes cl_intel_exec_by_local_thread cl_khr_spir cl_khr_fp64 Platform Extensions function suffix INTEL
Platform Name Intel(R) OpenCL
Number of devices 1
Device Name Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Device Vendor Intel(R) Corporation
Device Vendor ID 0x8086
Device Version OpenCL 1.2 (Build 475)
Driver Version 1.2.0.475
Device OpenCL C Version OpenCL C 1.2
Device Type CPU
Device Available Yes
Device Profile FULL_PROFILE
Max compute units 8
Max clock frequency 3600MHz
Device Partition (core)
Max number of sub-devices 8
Supported partition types by counts, equally, by names (Intel)
Max work item dimensions 3
Max work item sizes 8192x8192x8192
Max work group size 8192
Compiler Available Yes
Linker Available Yes
Preferred work group size multiple 128
Preferred / native vector sizes
char 1 / 32
short 1 / 16
int 1 / 8
long 1 / 4
half 0 / 0 (n/a)
float 1 / 8
double 1 / 4 (cl_khr_fp64)
Half-precision Floating-point support (n/a)
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero No
Round to infinity No
IEEE754-2008 fused multiply-add No
Support is emulated in software No
Correctly-rounded divide and sqrt operations No
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations No
Address bits 64, Little-Endian
Global memory size 33447714816 (31.15GiB)
Error Correction support No
Max memory allocation 8361928704 (7.788GiB)
Unified memory for Host and Device Yes
Minimum alignment for any data type 128 bytes
Alignment of base address 1024 bits (128 bytes)
Global Memory cache type Read/Write
Global Memory cache size 262144 (256KiB)
Global Memory cache line 64 bytes
Image support Yes
Max number of samplers per kernel 480
Max size for 1D images from buffer 522620544 pixels
Max 1D or 2D image array size 2048 images
Max 2D image size 16384x16384 pixels
Max 3D image size 2048x2048x2048 pixels
Max number of read image args 480
Max number of write image args 480
Local memory type Global
Local memory size 32768 (32KiB)
Max constant buffer size 131072 (128KiB)
Max number of constant args 480
Max size of kernel argument 3840 (3.75KiB)
Queue properties
Out-of-order execution Yes
Profiling Yes
Local thread execution (Intel) Yes
Prefer user sync for interop No
Profiling timer resolution 1ns
Execution capabilities
Run OpenCL kernels Yes
Run native kernels Yes
SPIR versions 1.2
printf() buffer size 1048576 (1024KiB)
Built-in kernels
Device Extensions cl_khr_icd cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_depth_images cl_khr_3d_image_writes cl_intel_exec_by_local_thread cl_khr_spir cl_khr_fp64
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) No platform
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) No platform
clCreateContext(NULL, ...) [default] No platform
clCreateContext(NULL, ...) [other] Success [INTEL]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127504
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 31457280
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) 34603008
file locks (-x) unlimited
Platform = Intel(R) OpenCL
type = Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Using device = Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Iskiriame masyva i=0
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=1
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=2
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=3
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=4
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=5
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=6
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=7
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=8
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=9
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=10
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=11
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=12
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=13
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=14
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=15
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=16
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=17
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=18
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=19
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=20
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=21
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=22
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=23
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=24
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=25
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=26
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=27
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=28
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=29
Pradedame isskirti
Isskyrimas baigesi
Hello World!
double4(1, 2, 3, 4)BAIGEME DARBA!
An Vas 12 09:31:24 EET 2019
gpu-0-1.local
/home/edvinas/JOB_SCRIPTS
An Vas 12 09:46:30 EET 2019
gpu-0-1.local
/home/edvinas/JOB_SCRIPTS
Number of platforms 2
Platform Name NVIDIA CUDA
Platform Vendor NVIDIA Corporation
Platform Version OpenCL 1.2 CUDA 9.0.282
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_copy_opts cl_khr_gl_event cl_nv_create_buffer
Platform Extensions function suffix NV
Platform Name Intel(R) OpenCL
Platform Vendor Intel(R) Corporation
Platform Version OpenCL 1.2
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_icd cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_depth_images cl_khr_3d_image_writes cl_intel_exec_by_local_thread cl_khr_spir cl_khr_fp64
Platform Extensions function suffix INTEL
Platform Name NVIDIA CUDA
Number of devices 1
Device Name Quadro K5000
Device Vendor NVIDIA Corporation
Device Vendor ID 0x10de
Device Version OpenCL 1.2 CUDA
Driver Version 384.111
Device OpenCL C Version OpenCL C 1.2
Device Type GPU
Device Available Yes
Device Profile FULL_PROFILE
Device Topology (NV) PCI-E, 01:00.0
Max compute units 8
Max clock frequency 705MHz
Compute Capability (NV) 3.0
Device Partition (core)
Max number of sub-devices 1
Supported partition types None
Max work item dimensions 3
Max work item sizes 1024x1024x64
Max work group size 1024
Compiler Available Yes
Linker Available Yes
Preferred work group size multiple 32
Warp size (NV) 32
Preferred / native vector sizes
char 1 / 1
short 1 / 1
int 1 / 1
long 1 / 1
half 0 / 0 (n/a)
float 1 / 1
double 1 / 1 (cl_khr_fp64)
Half-precision Floating-point support (n/a)
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations Yes
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations No
Address bits 64, Little-Endian
Global memory size 3698524160 (3.445GiB)
Error Correction support Yes
Max memory allocation 924631040 (881.8MiB)
Unified memory for Host and Device No
Integrated memory (NV) No
Minimum alignment for any data type 128 bytes
Alignment of base address 4096 bits (512 bytes)
Global Memory cache type Read/Write
Global Memory cache size 131072 (128KiB)
Global Memory cache line 128 bytes
Image support Yes
Max number of samplers per kernel 32
Max size for 1D images from buffer 134217728 pixels
Max 1D or 2D image array size 2048 images
Max 2D image size 16384x16384 pixels
Max 3D image size 4096x4096x4096 pixels
Max number of read image args 256
Max number of write image args 16
Local memory type Local
Local memory size 49152 (48KiB)
Registers per block (NV) 65536
Max constant buffer size 65536 (64KiB)
Max number of constant args 9
Max size of kernel argument 4352 (4.25KiB)
Queue properties
Out-of-order execution Yes
Profiling Yes
Prefer user sync for interop No
Profiling timer resolution 1000ns
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
Kernel execution timeout (NV) No
Concurrent copy and kernel execution (NV) Yes
Number of async copy engines 2
printf() buffer size 1048576 (1024KiB)
Built-in kernels
Device Extensions cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_copy_opts cl_khr_gl_event cl_nv_create_buffer
Platform Name Intel(R) OpenCL
Number of devices 1
Device Name Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Device Vendor Intel(R) Corporation
Device Vendor ID 0x8086
Device Version OpenCL 1.2 (Build 475)
Driver Version 1.2.0.475
Device OpenCL C Version OpenCL C 1.2
Device Type CPU
Device Available Yes
Device Profile FULL_PROFILE
Max compute units 8
Max clock frequency 3600MHz
Device Partition (core)
Max number of sub-devices 8
Supported partition types by counts, equally, by names (Intel)
Max work item dimensions 3
Max work item sizes 8192x8192x8192
Max work group size 8192
Compiler Available Yes
Linker Available Yes
Preferred work group size multiple 128
Preferred / native vector sizes
char 1 / 32
short 1 / 16
int 1 / 8
long 1 / 4
half 0 / 0 (n/a)
float 1 / 8
double 1 / 4 (cl_khr_fp64)
Half-precision Floating-point support (n/a)
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero No
Round to infinity No
IEEE754-2008 fused multiply-add No
Support is emulated in software No
Correctly-rounded divide and sqrt operations No
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations No
Address bits 64, Little-Endian
Global memory size 33447714816 (31.15GiB)
Error Correction support No
Max memory allocation 8361928704 (7.788GiB)
Unified memory for Host and Device Yes
Minimum alignment for any data type 128 bytes
Alignment of base address 1024 bits (128 bytes)
Global Memory cache type Read/Write
Global Memory cache size 262144 (256KiB)
Global Memory cache line 64 bytes
Image support Yes
Max number of samplers per kernel 480
Max size for 1D images from buffer 522620544 pixels
Max 1D or 2D image array size 2048 images
Max 2D image size 16384x16384 pixels
Max 3D image size 2048x2048x2048 pixels
Max number of read image args 480
Max number of write image args 480
Local memory type Global
Local memory size 32768 (32KiB)
Max constant buffer size 131072 (128KiB)
Max number of constant args 480
Max size of kernel argument 3840 (3.75KiB)
Queue properties
Out-of-order execution Yes
Profiling Yes
Local thread execution (Intel) Yes
Prefer user sync for interop No
Profiling timer resolution 1ns
Execution capabilities
Run OpenCL kernels Yes
Run native kernels Yes
SPIR versions 1.2
printf() buffer size 1048576 (1024KiB)
Built-in kernels
Device Extensions cl_khr_icd cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_depth_images cl_khr_3d_image_writes cl_intel_exec_by_local_thread cl_khr_spir cl_khr_fp64
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) No platform
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) No platform
clCreateContext(NULL, ...) [default] No platform
clCreateContext(NULL, ...) [other] Success [NV]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127504
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 31457280
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) 34603008
file locks (-x) unlimited
Platform = NVIDIA CUDA
type = Quadro K5000
Platform = Intel(R) OpenCL
type = Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Using device = Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Iskiriame masyva i=0
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=1
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=2
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=3
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=4
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=5
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=6
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=7
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=8
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=9
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=10
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=11
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=12
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=13
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=14
Pradedame isskirti
Isskyrimas baigesi
Hello World!
Iskiriame masyva i=15
Pradedame isskirti
Isskyrimas baigesi
Hello World!
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::compute::opencl_error> >'
what(): Out of Host Memory
Iskiriame masyva i=16
/var/spool/slurmd/job11121/slurm_script: line 15: 5682 Aborted /home/edvinas/GPU-MEMORY-BUILD/GPU-MEMORY 20000000
BAIGEME DARBA!
An Vas 12 09:47:55 EET 2019
gpu-0-1.local
/home/edvinas/JOB_SCRIPTS
intel.icd
Description: Binary data
CPU-K5000_EDD.sh
Description: application/shellscript
