External Email - Use Caution
Hi,
I increased the memory, and it ran partially to create the segmented input
images, however I reached this error that I don't quite understand:
2023-05-10 17:16:15.526359: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could
not find cuda drivers on your machine, GPU will not be used.
2023-05-10 17:16:15.609260: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could
not find cuda drivers on your machine, GPU will not be used.
2023-05-10 17:16:18.907335: W tensorflow/compiler/
tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
using 1 thread
Segmentation of reference image already exists; reading from disk
Segmentation of floating image already exists; reading from disk
Linear registration
Computing centroids and estimating affine transform
Traceback (most recent call last):
File "/scratch.global/tpengo/freesurfer/python/scripts/mri_easyreg", line
1809, in <module>
main()
File "/scratch.global/tpengo/freesurfer/python/scripts/mri_easyreg", line
194, in main
Mflo = getM(atlasCOG[:, ok > 0], floCOG[:, ok > 0])
File "/scratch.global/tpengo/freesurfer/python/scripts/mri_easyreg", line
1198, in getM
x = np.matmul(np.linalg.inv(np.matmul(np.transpose(A), A)),
np.matmul(np.transpose(A), b))
File "<__array_function__ internals>", line 180, in inv
File "/scratch.global/tpengo/freesurfer/python/lib/python3.
8/site-packages/numpy/linalg/linalg.py", line 552, in inv
ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)
File "/scratch.global/tpengo/freesurfer/python/lib/python3.
8/site-packages/numpy/linalg/linalg.py", line 89, in
_raise_linalgerror_singular
raise LinAlgError("Singular matrix")
numpy.linalg.LinAlgError: Singular matrix
Nadia Kane
Junior Imaging Specialist
University Imaging Centers | University of Minnesota
https://secure-web.cisco.com/1CgfJ293M5EIHLM0zgv8CWqjm5qLHzRSUoMULMp0eirEbBVXIuV654IFCZC7RoAZSn9awmMZcF4muPxvNhjNMqZGH9RNuq5x1VM1aj1SsTkuyEdkMwBfpm6A9ovVG7t2QZYuLCmmcM3kbGk3eWmUvi1i8ktYu8Qz0DnnIbFbXk5qOqF-Fwds8GBOBAGOW0sxqwUxEtjpg0xqe6zQzHfXCLhc7WoxW3xj-8e1TTorb-hzym4-JOhFHYpz_pzcVMA-EjLMXRQ2rG781UGroEfWMHuqnajC8mxUOTiC-Y0yhdAR-v1itFaj6AakAI8EBJzYE/https%3A%2F%2Fmed.umn.edu%2Fuic
On Tue, May 9, 2023 at 12:38 PM Nadia Kane <kanex...@umn.edu> wrote:
> Hi,
> I'm working on applying deep learning in image registration for a master's
> project. The overarching goal was to use real tissue clearing lightsheet
> data to perform the image registration, but first created some
> synthetically deformed brains based on the Allen Brain Atlas and Lab2im
> algorithm
> <https://secure-web.cisco.com/1LleH4XmcSc3vbFU8r_dMj1epTIr9qgP95cyHn0T9IwgC_jKBRSbIoA35pkuM5EoQY2In_inv57KtPNQ3PCOYNOPF6z2WVrt6FVldJPAv0DopfHXQL64mg8znPA2eMAjJCWsjJvZazuVoELOnU15wPSvhyFMDemzB0L8yHimGi-JN_e9qBhw4sZkc5-HV9Pgc8DUp06Zamcj6tU2klSA8IMTUS5bBaDoaSbRf0uuX-X7QFVcuoyJJSz-NvzReh7SJ6k9LUhQKeCDMKzgBvgUCKrI2Tsj95aDs_bwqMlZIBtpMdLO2OCx72QkAz487kz6s/https%3A%2F%2Fgithub.com%2FBBillot%2Flab2im>.
> However, I'm having some
> issues implementing the command in an Centos Linux 7 (Core) HPC
> environment. The command gets Killed, I'm wondering if you could provide
> some more insight? Thank you!
>
> I've set up my environment as follows:
>
> module load mamba
> mamba create -n freesurf # I named the environment freesurf
> conda activate freesurf
> mamba install tensorflow-gpu cuda-version=11.8
>
>
> I've attached the easyreg.sh shell script that I run. Here's the output:
>
> (freesurfer) kanex161@agc02 [~/Documents/Thesis/code] % bash easyreg.sh
> 2023-05-09 11:19:32.374206: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could
> not find cuda drivers on your machine, GPU will not be used.
> 2023-05-09 11:19:32.405672: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could
> not find cuda drivers on your machine, GPU will not be used.
> 2023-05-09 11:19:32.405981: I
> tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary
> is optimized to use available CPU instructions in performance-critical
> operations.
> To enable the following instructions: AVX2 FMA, in other operations,
> rebuild TensorFlow with the appropriate compiler flags.
> 2023-05-09 11:19:33.258226: W
> tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could
> not find TensorRT
> [W interface.cpp:47] Warning: Loading nvfuser library failed with: Error
> in dlopen: libnvfuser_codegen.so: cannot open shared object file: No such
> file or directory (function LoadingNvfuserLibrary)
> using 1 thread
> Segmenting reference image
> Reading reference image
> Setting up segmentation net
> 2023-05-09 11:19:38.489093: E
> tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:266] failed
> call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
> 2023-05-09 11:19:38.489166: I
> tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168]
> retrieving CUDA diagnostic information for host: agc02
> 2023-05-09 11:19:38.489202: I
> tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175]
> hostname: agc02
> 2023-05-09 11:19:38.489460: I
> tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199]
> libcuda reported version is: 470.103.1
> 2023-05-09 11:19:38.489502: I
> tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203]
> kernel reported version is: 470.103.1
> 2023-05-09 11:19:38.489530: I
> tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:309]
> kernel version seems to match DSO: 470.103.1
> Inference / segmentation
> easyreg.sh: line 8: 1108392 Killed ./mri_easyreg --ref
> /home/umii/kanex161/Documents/Thesis/data/outputs/atlas/atlas.nii.gz --flo
> /home/umii/kanex161/Documents/Thesis/data/outputs/lab2im/brains/brain_00.nii.gz
> --ref_seg
> /home/umii/kanex161/Documents/Thesis/data/outputs/easyreg/ref_seg.nii.gz
> --flo_seg
> /home/umii/kanex161/Documents/Thesis/data/outputs/easyreg/flo_seg.nii.gz
> --flo_reg
> /home/umii/kanex161/Documents/Thesis/data/outputs/easyreg/flo_reg.nii.gz
> --fwd_field
> /home/umii/kanex161/Documents/Thesis/data/outputs/easyreg/fwd_field.nii.gz
>
_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Mass General Brigham
Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline
<https://www.massgeneralbrigham.org/complianceline> .
Please note that this e-mail is not secure (encrypted). If you do not wish to
continue communication over unencrypted e-mail, please notify the sender of
this message immediately. Continuing to send or respond to e-mail after
receiving this message means you understand and accept this risk and wish to
continue to communicate over unencrypted e-mail.
