External Email - Use Caution
Can you try it on a computer with more ram?
From: freesurfer-boun...@nmr.mgh.harvard.edu
<freesurfer-boun...@nmr.mgh.harvard.edu> On Behalf Of Matthew Lynch
Sent: Thursday, July 13, 2023 1:55 PM
To: freesurfer@nmr.mgh.harvard.edu
Subject: [Freesurfer] recon-all-clinical out of memory error
External Email - Use Caution
I am having trouble running the new recon-all-clinical module. I input 5mm T1
axial DICOM images and it fails in SynthSeg with what appears to be an out of
memory error. I am running FreeSurfer 7.4.0 in WSL with Ubuntu 22 on a
computer with 16 Gb RAM. Can anyone provide any thoughts on how to correct
this? Thanks in advance!
Here is the command:
recon-all-clinical.sh ./T1AX5MM/119 fsclin 1
here is the error output:
SynthSeg-robust 2.0
using CPU, hiding all CUDA_VISIBLE_DEVICES
using 1 thread
predicting 1/1
the following problem occured with image
/mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz :
Traceback (most recent call last):
File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in
predict
post_patch_segmentation, post_patch_parcellation, qc_score =
net.predict([image, shape_input])
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py",
line 52, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution
error:
Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most recent
call last):
File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 2642,
in <module>
main()
File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 162,
in main
predict(
File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315,
in predict
post_patch_segmentation, post_patch_parcellation, qc_score =
net.predict([image, shape_input])
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2382, in predict
tmp_batch_outputs = self.predict_function(iterator)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2169, in predict_function
return step_function(self, iterator)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2155, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2143, in run_step
outputs = model.predict_step(data)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 2111, in predict_step
return self(x, training=False)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
line 558, in __call__
return super().__call__(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
outputs = call_fn(inputs, *args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py",
line 512, in call
return self._run_internal_graph(inputs, training=training, mask=mask)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py",
line 669, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
outputs = call_fn(inputs, *args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
return fn(*args, **kwargs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py",
line 196, in call
return self._merge_function(inputs)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py",
line 134, in _merge_function
return backend.concatenate(inputs, axis=self.axis)
File
"/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py",
line 3581, in concatenate
return tf.concat([to_dense(x) for x in tensors], axis)
Node: 'model_5/unet_parc_merge_8/concat'
OOM when allocating tensor with shape[1,256,256,160,72] and type float on
/job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu
[[{{node model_5/unet_parc_merge_8/concat}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add
report_tensor_allocations_upon_oom to RunOptions for current allocation info.
This isn't available when running in Eager mode.
[Op:__inference_predict_function_10473]
resuming program execution
_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Mass General Brigham
Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline
<https://www.massgeneralbrigham.org/complianceline> .
Please note that this e-mail is not secure (encrypted). If you do not wish to
continue communication over unencrypted e-mail, please notify the sender of
this message immediately. Continuing to send or respond to e-mail after
receiving this message means you understand and accept this risk and wish to
continue to communicate over unencrypted e-mail.
