External Email - Use Caution
I do not have access to a computer with more RAM. I can run recon-all with
1mm T1 images without any problems. How much RAM would recon-all-clinical
be expected to need? Does it require less RAM if it uses the GPU instead
of CPU?
On Thu, Jul 13, 2023 at 1:58 PM Fischl, Bruce R.,PHD <
bfis...@mgh.harvard.edu> wrote:
> External Email - Use Caution
>
> Can you try it on a computer with more ram?
>
>
>
> *From:* freesurfer-boun...@nmr.mgh.harvard.edu <
> freesurfer-boun...@nmr.mgh.harvard.edu> *On Behalf Of *Matthew Lynch
> *Sent:* Thursday, July 13, 2023 1:55 PM
> *To:* freesurfer@nmr.mgh.harvard.edu
> *Subject:* [Freesurfer] recon-all-clinical out of memory error
>
>
>
> * External Email - Use Caution *
>
> I am having trouble running the new recon-all-clinical module. I input
> 5mm T1 axial DICOM images and it fails in SynthSeg with what appears to be
> an out of memory error. I am running FreeSurfer 7.4.0 in WSL with Ubuntu
> 22 on a computer with 16 Gb RAM. Can anyone provide any thoughts on how
> to correct this? Thanks in advance!
>
>
>
> Here is the command:
>
>
>
> recon-all-clinical.sh ./T1AX5MM/119 fsclin 1
>
>
>
>
>
> here is the error output:
>
>
>
> SynthSeg-robust 2.0
> using CPU, hiding all CUDA_VISIBLE_DEVICES
> using 1 thread
> predicting 1/1
>
> the following problem occured with image
> /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz :
> Traceback (most recent call last):
> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
> 315, in predict
> post_patch_segmentation, post_patch_parcellation, qc_score =
> net.predict([image, shape_input])
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
> line 70, in error_handler
> raise e.with_traceback(filtered_tb) from None
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py",
> line 52, in quick_execute
> tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
> tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph
> execution error:
>
> Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most
> recent call last):
> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
> 2642, in <module>
> main()
> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
> 162, in main
> predict(
> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line
> 315, in predict
> post_patch_segmentation, post_patch_parcellation, qc_score =
> net.predict([image, shape_input])
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
> line 65, in error_handler
> return fn(*args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
> line 2382, in predict
> tmp_batch_outputs = self.predict_function(iterator)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
> line 2169, in predict_function
> return step_function(self, iterator)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
> line 2155, in step_function
> outputs = model.distribute_strategy.run(run_step, args=(data,))
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
> line 2143, in run_step
> outputs = model.predict_step(data)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
> line 2111, in predict_step
> return self(x, training=False)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
> line 65, in error_handler
> return fn(*args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py",
> line 558, in __call__
> return super().__call__(*args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
> line 65, in error_handler
> return fn(*args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py",
> line 1145, in __call__
> outputs = call_fn(inputs, *args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
> line 96, in error_handler
> return fn(*args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py",
> line 512, in call
> return self._run_internal_graph(inputs, training=training, mask=mask)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py",
> line 669, in _run_internal_graph
> outputs = node.layer(*args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
> line 65, in error_handler
> return fn(*args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py",
> line 1145, in __call__
> outputs = call_fn(inputs, *args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py",
> line 96, in error_handler
> return fn(*args, **kwargs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py",
> line 196, in call
> return self._merge_function(inputs)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py",
> line 134, in _merge_function
> return backend.concatenate(inputs, axis=self.axis)
> File
> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py",
> line 3581, in concatenate
> return tf.concat([to_dense(x) for x in tensors], axis)
> Node: 'model_5/unet_parc_merge_8/concat'
> OOM when allocating tensor with shape[1,256,256,160,72] and type float on
> /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu
> [[{{node model_5/unet_parc_merge_8/concat}}]]
> Hint: If you want to see a list of allocated tensors when OOM happens, add
> report_tensor_allocations_upon_oom to RunOptions for current allocation
> info. This isn't available when running in Eager mode.
> [Op:__inference_predict_function_10473]
>
> resuming program execution
>
>
> _______________________________________________
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://secure-web.cisco.com/1cgiw0QpeuIZHhfIZEvHRuSB2y_kwffJua3EEuK4AZSxIkPlisXBPGc96CC_jh1Z_mXQcqiDE9wgrUZxUC0O2Ks9yHi5UgBHUQOHkYYeQFVg3XxYFfhWZGgMfrZ1smYoc7M2_ore_PCAWQIUlpfZFoa7_PvNU0oeRp2AdfscOtGo7o1RqkruqzzdefASm--Suy_KSrwFpwq3bswq8sXLAKxNhvmgjG41xFQY68BWNKG2idcUPrAxtrnBZaUL8MMlVj210BxUGOsudCYS9qDV8AFYQAaYWHucMwecxCQGgkoVO5547ucAMIofYRlVv-hTiiqnxeL0WC_0TnYmiMn1tBQ/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer
> The information in this e-mail is intended only for the person to whom it
> is addressed. If you believe this e-mail was sent to you in error and the
> e-mail contains patient information, please contact the Mass General
> Brigham Compliance HelpLine at
> https://secure-web.cisco.com/1OHxlgWD90YT7mlV-NqvsHcbQvZrB7oYQInUUJVdImP_oVX3PJrAp6BumYG-qFweR7ad7Dd_M-xyjPiJBSZ5YTd7H5u3GqIIcVDpvk8TNguVnml7_lbqoZz3LJCq_zMxlQrun6kaFOJ5owntVf2ofc-gxKiQFOzrHvlE-KTMQqA1IBCTtvlQUtqfJbRb7Ujp5-DvEOEUu0gDkYibdIQZd098Jom3dMChwcbS1eW-p_TQPKTJgIrLf13jMSd6Z7VX1mvWwzfYlQBl-leYgQnH3Z4ZkzApYcxDDRhaSfEEmbCNQdEMMxfyrNbgLbytM21WO11vfE-u2VB67yGkk_0VmVQ/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline
> <
> https://secure-web.cisco.com/1OHxlgWD90YT7mlV-NqvsHcbQvZrB7oYQInUUJVdImP_oVX3PJrAp6BumYG-qFweR7ad7Dd_M-xyjPiJBSZ5YTd7H5u3GqIIcVDpvk8TNguVnml7_lbqoZz3LJCq_zMxlQrun6kaFOJ5owntVf2ofc-gxKiQFOzrHvlE-KTMQqA1IBCTtvlQUtqfJbRb7Ujp5-DvEOEUu0gDkYibdIQZd098Jom3dMChwcbS1eW-p_TQPKTJgIrLf13jMSd6Z7VX1mvWwzfYlQBl-leYgQnH3Z4ZkzApYcxDDRhaSfEEmbCNQdEMMxfyrNbgLbytM21WO11vfE-u2VB67yGkk_0VmVQ/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline>
> .
>
_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Mass General Brigham
Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline
<https://www.massgeneralbrigham.org/complianceline> .
Please note that this e-mail is not secure (encrypted). If you do not wish to
continue communication over unencrypted e-mail, please notify the sender of
this message immediately. Continuing to send or respond to e-mail after
receiving this message means you understand and accept this risk and wish to
continue to communicate over unencrypted e-mail.
