Hi,
Yes, RTK only uses one GPU. We haven't implemented multi-GPU capability.
Any contribution in that direction is welcome!
If you really have 32 GB available for GPU0, then I don't understand what
is the issue but you should be able to find out.
Simon
On Fri, Jun 16, 2023 at 6:32 PM Rahman, Obaid <rahm...@ornl.gov> wrote:
> Thank you, Simon and Nils, for the suggestions.
>
> This is what I am doing:
> a. I read a (Xray CT) projection data (~1.5 GB)
> b. Then perform FDK reconstruction (~10 GB)
> c. Then forward project that reconstruction
>
> I am using the cuda version of itk-rtk. Steps a and b work fine and the
> reconstruction looks as expected.
> Then these are the lines of code I use for forward projection:
>
> ForwardProj = rtk.CudaForwardProjectionImageFilter[CudaImageType].New()
> ForwardProj.SetGeometry( geometry )
> ForwardProj.SetInput(0, constantImageSource.GetOutput())
> ForwardProj.SetInput(1, recon_image.GetOutput())
> ForwardProj.Update()
>
> I get this error:
> Traceback (most recent call last):
> File "projection_test.py", line 223, in <module>
> ForwardProj.Update()
> RuntimeError: /work/src/rtkCudaUtilities.cu:115:
> ITK ERROR: CUDA ERROR: out of memory
>
> I also try to delete the variables I don’t need using (del variable), but
> I still get that error.
> The reconstruction is supposed to be ~10GB, projection is supposed to be
> 1.5 GB. GPU0 memory is 32 GB.
>
> I can see that before I get the error, GPU0 memory reaches about its
> capacity.
> The weird thing is that GPU1, GPU2, GPU3 that have the same memory
> capacity are basically unused (~4MB each).
> Is the forward projection not performed over all 4 GPUs?
>
> To try to force the code to use all 4 GPUs, I have also included these
> lines in the code:
> os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
> os.environ["CUDA_VISIBLE_DEVICES"]="0,1,2,3" # specify which GPU(s) to be
> used
>
> But I still get that error, and only GPU0 gets used.
>
>
> Best,
> Obaid
>
> On Jun 16, 2023, at 8:58 AM, krah <nils.k...@creatis.insa-lyon.fr> wrote:
>
> Just to add to Simon's answer:
> Objects are deleted by python's garbage collection mechanism once there
> are no references left to the object. (keyword: reference counting).
> The
> del variable
> statement Simon mention deletes the reference variable, but the memory is
> freed only when python runs a garbage collection. That might immediately
> afterwards, or slightly later (scheduled). Usually, you do not need to
> worry about that and deleting the reference as Simon said is sufficient. If
> for some reason you need to be absolutely sure that the object is garbage
> collected immediately, you can trigger a garbage collection by
>
> import gc
> gc.collect()
>
> Again, that should not be necessary, but it is useful to bear in mind how
> the memory is actually handled in python.
>
> Cheers,
> Nils
>
> On Jun 16 2023, at 9:10 am, Simon Rit <simon....@creatis.insa-lyon.fr>
> wrote:
>
>> Hi,
>> If you have a Python variable holding memory, you can simply call
>> del variable_name
>> If you want to clear the GPU memory for an image, you can access the CPU
>> buffer pointer to have it moved to the computer RAM. Examples below.
>> Simon
>>
>> >>> import os
>> >>> os.system('nvidia-smi')
>> Fri Jun 16 08:49:38 2023
>>
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6
>> |
>>
>> |-------------------------------+----------------------+----------------------+
>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
>> ECC |
>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>> Compute M. |
>> | | |
>> MIG M. |
>>
>> |===============================+======================+======================|
>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>> N/A |
>> | N/A 52C P8 N/A / N/A | 0MiB / 4096MiB | 0%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>>
>>
>>
>> +-----------------------------------------------------------------------------+
>> | Processes:
>> |
>> | GPU GI CI PID Type Process name GPU
>> Memory |
>> | ID ID Usage
>> |
>>
>> |=============================================================================|
>> | No running processes found
>> |
>>
>> +-----------------------------------------------------------------------------+
>> 0
>> >>> import itk
>> >>> from itk import RTK as rtk
>> >>> img=itk.CudaImage[itk.F, 3].New()
>> >>> img.SetRegions([500]*3)
>> >>> img.Allocate()
>> >>> os.system('nvidia-smi')
>> Fri Jun 16 08:51:28 2023
>>
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6
>> |
>>
>> |-------------------------------+----------------------+----------------------+
>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
>> ECC |
>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>> Compute M. |
>> | | |
>> MIG M. |
>>
>> |===============================+======================+======================|
>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>> N/A |
>> | N/A 58C P8 N/A / N/A | 45MiB / 4096MiB | 0%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>>
>>
>>
>> +-----------------------------------------------------------------------------+
>> | Processes:
>> |
>> | GPU GI CI PID Type Process name GPU
>> Memory |
>> | ID ID Usage
>> |
>>
>> |=============================================================================|
>> | 0 N/A N/A 10890 C python
>> 43MiB |
>>
>> +-----------------------------------------------------------------------------+
>> 0
>> >>> img.GetCudaDataManager().GetGPUBufferPointer()
>> <Swig Object of type 'void *' at 0x7f20d3327f60>
>> >>> os.system('nvidia-smi')
>> Fri Jun 16 08:52:05 2023
>>
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6
>> |
>>
>> |-------------------------------+----------------------+----------------------+
>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
>> ECC |
>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>> Compute M. |
>> | | |
>> MIG M. |
>>
>> |===============================+======================+======================|
>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>> N/A |
>> | N/A 54C P0 N/A / N/A | 525MiB / 4096MiB | 0%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>>
>>
>>
>> +-----------------------------------------------------------------------------+
>> | Processes:
>> |
>> | GPU GI CI PID Type Process name GPU
>> Memory |
>> | ID ID Usage
>> |
>>
>> |=============================================================================|
>> | 0 N/A N/A 10890 C python
>> 523MiB |
>>
>> +-----------------------------------------------------------------------------+
>> 0
>> >>> del img
>> >>> os.system('nvidia-smi')
>> Fri Jun 16 08:52:21 2023
>>
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6
>> |
>>
>> |-------------------------------+----------------------+----------------------+
>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
>> ECC |
>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>> Compute M. |
>> | | |
>> MIG M. |
>>
>> |===============================+======================+======================|
>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>> N/A |
>> | N/A 56C P0 N/A / N/A | 2MiB / 4096MiB | 0%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>>
>>
>>
>> +-----------------------------------------------------------------------------+
>> | Processes:
>> |
>> | GPU GI CI PID Type Process name GPU
>> Memory |
>> | ID ID Usage
>> |
>>
>> |=============================================================================|
>> | No running processes found
>> |
>>
>> +-----------------------------------------------------------------------------+
>> 0
>> >>> img=itk.CudaImage[itk.F, 3].New()
>> >>> img.SetRegions([500]*3)
>> >>> img.GetCudaDataManager().GetGPUBufferPointer()
>> <Swig Object of type 'void *' at 0x7f20b2c30870>
>> >>> os.system('nvidia-smi')
>> Fri Jun 16 08:54:10 2023
>>
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6
>> |
>>
>> |-------------------------------+----------------------+----------------------+
>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
>> ECC |
>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>> Compute M. |
>> | | |
>> MIG M. |
>>
>> |===============================+======================+======================|
>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>> N/A |
>> | N/A 58C P0 N/A / N/A | 525MiB / 4096MiB | 0%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>>
>>
>>
>> +-----------------------------------------------------------------------------+
>> | Processes:
>> |
>> | GPU GI CI PID Type Process name GPU
>> Memory |
>> | ID ID Usage
>> |
>>
>> |=============================================================================|
>> | 0 N/A N/A 10890 C python
>> 523MiB |
>>
>> +-----------------------------------------------------------------------------+
>> 0
>> >>> img.GetCudaDataManager().GetCPUBufferPointer()
>> >>> os.system('nvidia-smi')
>> Fri Jun 16 08:54:20 2023
>>
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6
>> |
>>
>> |-------------------------------+----------------------+----------------------+
>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
>> ECC |
>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>> Compute M. |
>> | | |
>> MIG M. |
>>
>> |===============================+======================+======================|
>> | 0 Quadro P2000 Off | 00000000:01:00.0 Off |
>> N/A |
>> | N/A 58C P0 N/A / N/A | 47MiB / 4096MiB | 0%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>>
>>
>>
>> +-----------------------------------------------------------------------------+
>> | Processes:
>> |
>> | GPU GI CI PID Type Process name GPU
>> Memory |
>> | ID ID Usage
>> |
>>
>> |=============================================================================|
>> | 0 N/A N/A 10890 C python
>> 45MiB |
>>
>> +-----------------------------------------------------------------------------+
>> 0
>>
>> On Thu, Jun 15, 2023 at 5:33 PM Rahman, Obaid <rahm...@ornl.gov> wrote:
>>
>>> Hi,
>>>
>>> I am using ink-rtk-cuda116 with *python*.
>>> I have too many cuda variables (images and filters).
>>> I would like to clear some of these variables.
>>>
>>> I am getting the following error:
>>> ITK ERROR: CUDA ERROR: out of memory
>>>
>>> Does anyone know how to clear Cuda variables in python?
>>> Thanks.
>>>
>>> Best,
>>> Obaidullah Rahman
>>> Oak Ridge National Laboratory, TN, United States
>>> _______________________________________________
>>> Rtk-users mailing list
>>> rtk-us...@openrtk.org
>>> https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users
>>> <https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=CChHfQz9-I6wvdDiw7oSBgscNCDgHdPren14tU6VDTrIDrHB3k0WFG2Xrm2AWEU0&s=SJB2aIBPO7DWRc1JOmqQjztdRTfjN2hb5Sa4vrWUVfE&e=>
>>>
>> _______________________________________________
>> Rtk-users mailing list
>> rtk-us...@openrtk.org
>> https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users
>> <https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwQFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=CChHfQz9-I6wvdDiw7oSBgscNCDgHdPren14tU6VDTrIDrHB3k0WFG2Xrm2AWEU0&s=SJB2aIBPO7DWRc1JOmqQjztdRTfjN2hb5Sa4vrWUVfE&e=>
>>
>
>
_______________________________________________
Rtk-users mailing list
rtk-us...@openrtk.org
https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users
