That was a login node. Here is a compute node on SUMMIT: 12:45 130 adams/feature-mat-cuda *= ~/petsc/src/mat/tutorials$ nvidia-smi Sat Sep 26 12:45:28 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.116.00 Driver Version: 418.116.00 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000004:04:00.0 Off | 0 | | N/A 36C P0 36W / 300W | 0MiB / 16130MiB | 0% E. Process | +-------------------------------+----------------------+----------------------+ | 1 Tesla V100-SXM2... On | 00000004:05:00.0 Off | 0 | | N/A 41C P0 36W / 300W | 0MiB / 16130MiB | 0% E. Process | +-------------------------------+----------------------+----------------------+ | 2 Tesla V100-SXM2... On | 00000035:03:00.0 Off | 0 | | N/A 34C P0 36W / 300W | 0MiB / 16130MiB | 0% E. Process | +-------------------------------+----------------------+----------------------+ | 3 Tesla V100-SXM2... On | 00000035:04:00.0 Off | 0 | | N/A 42C P0 37W / 300W | 0MiB / 16130MiB | 0% E. Process | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ On Sat, Sep 26, 2020 at 12:44 PM Mark Adams <[email protected]> wrote: > 0:30 adams/feature-mat-cuda *= ~/petsc$ nvidia-smi > Sat Sep 26 12:42:37 2020 > > +-----------------------------------------------------------------------------+ > | NVIDIA-SMI 418.116.00 Driver Version: 418.116.00 CUDA Version: 10.1 > | > > |-------------------------------+----------------------+----------------------+ > | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. > ECC | > | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute > M. | > > |===============================+======================+======================| > | 0 Tesla V100-SXM2... On | 00000004:05:00.0 Off | > 0 | > | N/A 38C P0 36W / 300W | 0MiB / 16130MiB | 0% E. > Process | > > +-------------------------------+----------------------+----------------------+ > | 1 Tesla V100-SXM2... On | 00000035:04:00.0 Off | > 0 | > | N/A 40C P0 38W / 300W | 0MiB / 16130MiB | 0% E. > Process | > > +-------------------------------+----------------------+----------------------+ > > > +-----------------------------------------------------------------------------+ > | Processes: GPU > Memory | > | GPU PID Type Process name Usage > | > > |=============================================================================| > | No running processes found > | > > +-----------------------------------------------------------------------------+ > > On Fri, Sep 25, 2020 at 9:36 PM Jacob Faibussowitsch <[email protected]> > wrote: > >> Configure by default should find out the available GPU and build for >> that sm_* it should not require the user to set this (how the heck is the >> user going to know what to set?) If I remember correctly there is a >> utility available that gives this information. >> >> For CUDA I believe the tool is nvidia-smi. Should make sure this >> automatic detection works when configuring —with-batch though since login >> nodes might have different arch than compute. >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: (312) 694-3391 >> >> On Sep 25, 2020, at 21:09, Barry Smith <[email protected]> wrote: >> >> >> Configure by default should find out the available GPU and build for >> that sm_* it should not require the user to set this (how the heck is the >> user going to know what to set?) If I remember correctly there is a >> utility available that gives this information. >> >> For generic builds like in package distributions I don't know how it >> should work, ideally all the possibilities would be available in the >> library and at run time the correct one will be utilized. >> >> Barry >> >> >> On Sep 25, 2020, at 5:49 PM, Mark Adams <[email protected]> wrote: >> >> '--CUDAFLAGS=-arch=sm_70', >> >> seems to fix this. >> >> On Fri, Sep 25, 2020 at 6:31 PM Mark Adams <[email protected]> wrote: >> >>> I see kokkos and hyper have a sm_70 flag, but I don't see one for PETSc. >>> >>> It looks like you have to specify this to get modern atomics to work in >>> Cuda. I get: >>> >>> /ccs/home/adams/petsc/include/petscaijdevice.h(99): error: no instance >>> of overloaded function "atomicAdd" matches the argument list >>> argument types are: (double *, double) >>> >>> I tried using a Kokkos configuration, thinking I could get these sm_70 >>> flags, but that did not work. >>> >>> Any ideas? >>> >>> Mark >>> >> >> >>
