External Email - Use Caution I was able to get recon-all-clinical working with 16 Gb computer.
I thought the problem might be Windows hogging memory, so I created a live persistent USB install of lubuntu, using lubuntu 22 as a low resource distro in order to minimize memory usage, and installed freesurfer 7.4.0. Even with initial memory usage on 1Gb on lubuntu, synthseg still failed with out of memory error with 8 Gb swap file. I watched the swap memory fill up and then process was killed. I added 16Gb swap using the following instructions, for a total swap space of 24 Gb + 16 Gb memory: https://secure-web.cisco.com/1iFbhXQHpEFIJXXJqO_gzEOq97K25g0DNoKMVRh6pXcfEcG45FjIHKXyZ-aP4IALU-nPOQLDiOSqu0cFmHUgQutpHNu0uDVykcyNMml-zVextcU0zdKZS40VUf8z_9D7dIRdwaA3dnuN6rKOop5o_IBd0Xhm4gQmczEdE5SepDZMz9a0BGgeAbf09f3ZzPyuMDZXLwyERiQRVQPyJMHuGClyrhtUC7d4NJuP_9KRBoqeR9BA4fwwkEHxZ3pS9vdU4gongLDl_1Q8VBZYKrm2q6TtMZRzrvRcidoPwmB4YdcPTSMGaOV0N2NodjXNNXXpmYqkMsO42oxmBpIBrh1lWHA/https%3A%2F%2Faskubuntu.com%2Fquestions%2F178712%2Fhow-to-increase-swap-space This worked. On Thu, Jul 13, 2023 at 3:28 PM Matthew Lynch <matthewl...@gmail.com> wrote: > I do not have access to a computer with more RAM. I can run recon-all > with 1mm T1 images without any problems. How much RAM would > recon-all-clinical be expected to need? Does it require less RAM if it > uses the GPU instead of CPU? > > On Thu, Jul 13, 2023 at 1:58 PM Fischl, Bruce R.,PHD < > bfis...@mgh.harvard.edu> wrote: > >> External Email - Use Caution >> >> Can you try it on a computer with more ram? >> >> >> >> *From:* freesurfer-boun...@nmr.mgh.harvard.edu < >> freesurfer-boun...@nmr.mgh.harvard.edu> *On Behalf Of *Matthew Lynch >> *Sent:* Thursday, July 13, 2023 1:55 PM >> *To:* freesurfer@nmr.mgh.harvard.edu >> *Subject:* [Freesurfer] recon-all-clinical out of memory error >> >> >> >> * External Email - Use Caution * >> >> I am having trouble running the new recon-all-clinical module. I input >> 5mm T1 axial DICOM images and it fails in SynthSeg with what appears to be >> an out of memory error. I am running FreeSurfer 7.4.0 in WSL with Ubuntu >> 22 on a computer with 16 Gb RAM. Can anyone provide any thoughts on how >> to correct this? Thanks in advance! >> >> >> >> Here is the command: >> >> >> >> recon-all-clinical.sh ./T1AX5MM/119 fsclin 1 >> >> >> >> >> >> here is the error output: >> >> >> >> SynthSeg-robust 2.0 >> using CPU, hiding all CUDA_VISIBLE_DEVICES >> using 1 thread >> predicting 1/1 >> >> the following problem occured with image >> /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz : >> Traceback (most recent call last): >> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line >> 315, in predict >> post_patch_segmentation, post_patch_parcellation, qc_score = >> net.predict([image, shape_input]) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >> line 70, in error_handler >> raise e.with_traceback(filtered_tb) from None >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", >> line 52, in quick_execute >> tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, >> tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph >> execution error: >> >> Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most >> recent call last): >> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line >> 2642, in <module> >> main() >> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line >> 162, in main >> predict( >> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line >> 315, in predict >> post_patch_segmentation, post_patch_parcellation, qc_score = >> net.predict([image, shape_input]) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >> line 65, in error_handler >> return fn(*args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >> line 2382, in predict >> tmp_batch_outputs = self.predict_function(iterator) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >> line 2169, in predict_function >> return step_function(self, iterator) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >> line 2155, in step_function >> outputs = model.distribute_strategy.run(run_step, args=(data,)) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >> line 2143, in run_step >> outputs = model.predict_step(data) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >> line 2111, in predict_step >> return self(x, training=False) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >> line 65, in error_handler >> return fn(*args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >> line 558, in __call__ >> return super().__call__(*args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >> line 65, in error_handler >> return fn(*args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", >> line 1145, in __call__ >> outputs = call_fn(inputs, *args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >> line 96, in error_handler >> return fn(*args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", >> line 512, in call >> return self._run_internal_graph(inputs, training=training, >> mask=mask) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", >> line 669, in _run_internal_graph >> outputs = node.layer(*args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >> line 65, in error_handler >> return fn(*args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", >> line 1145, in __call__ >> outputs = call_fn(inputs, *args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >> line 96, in error_handler >> return fn(*args, **kwargs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py", >> line 196, in call >> return self._merge_function(inputs) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py", >> line 134, in _merge_function >> return backend.concatenate(inputs, axis=self.axis) >> File >> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py", >> line 3581, in concatenate >> return tf.concat([to_dense(x) for x in tensors], axis) >> Node: 'model_5/unet_parc_merge_8/concat' >> OOM when allocating tensor with shape[1,256,256,160,72] and type float on >> /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu >> [[{{node model_5/unet_parc_merge_8/concat}}]] >> Hint: If you want to see a list of allocated tensors when OOM happens, >> add report_tensor_allocations_upon_oom to RunOptions for current allocation >> info. This isn't available when running in Eager mode. >> [Op:__inference_predict_function_10473] >> >> resuming program execution >> >> >> _______________________________________________ >> Freesurfer mailing list >> Freesurfer@nmr.mgh.harvard.edu >> https://secure-web.cisco.com/1ZpY_Fk6Q4ePI0Sh15uR7SD5mRjOqVDsP72J6nSh5V5mTUrsjgKRbye9RKgTxrOGkTTH-UIvIh6VpC1EwCc8pIeDezFO4h4Q8R2eDcNxTprGQ-stFz5pr-WOYsh4_NnUi4tvMSmzuLrkdvCcDohPPbfQdC34IwEJfmsjKfNkcDSIGH2pAatOu7gWLfKFvSO3gZweTk4QOjxGopZDQfJrKfA8452Tq_3Tv2qp7dnnhJppGpLwiCA0I4gSRPx7zeK8jRqFKOC4Ye50v78wl3TyJEMhLUpJtsUJOKU9xmIZbeA7zWEA9B968Rxg6Qs0pTZG91PQXPzN3a4BnGPxRweQYMA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer >> The information in this e-mail is intended only for the person to whom it >> is addressed. If you believe this e-mail was sent to you in error and the >> e-mail contains patient information, please contact the Mass General >> Brigham Compliance HelpLine at >> https://secure-web.cisco.com/10c39fPfTU9WxSfxiVo7hkfTFoQqC9QZxoxSaHBk5w1QOCAuqdkNjb8e_m16UnS5q1EeF7irRmSH5MpCTG-l2i8LcBm08bIfA4x90VEx7JZFAVFzROisPMy5js8PwqBY5tDw0_pE83l52M8MgVIGoMImcOS2d8UD636usrj9mCQSEXmIyf7nJWa6DWnFS_cwotlwLsnQBWspKLjJ9XEuYd1PIYrMiAZOP4ov8UyHUR7gF1sTGZMkrC2VsApESY-Qp7O5uCzovC2t3TxIxL3JJzY7fhFltossoQbXok4A5HnVoNDIl2q4EgfAUH0L8t12vMruhGkNAebXgIVwb_10n0g/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline >> < >> https://secure-web.cisco.com/10c39fPfTU9WxSfxiVo7hkfTFoQqC9QZxoxSaHBk5w1QOCAuqdkNjb8e_m16UnS5q1EeF7irRmSH5MpCTG-l2i8LcBm08bIfA4x90VEx7JZFAVFzROisPMy5js8PwqBY5tDw0_pE83l52M8MgVIGoMImcOS2d8UD636usrj9mCQSEXmIyf7nJWa6DWnFS_cwotlwLsnQBWspKLjJ9XEuYd1PIYrMiAZOP4ov8UyHUR7gF1sTGZMkrC2VsApESY-Qp7O5uCzovC2t3TxIxL3JJzY7fhFltossoQbXok4A5HnVoNDIl2q4EgfAUH0L8t12vMruhGkNAebXgIVwb_10n0g/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline> >> . >> >
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
