External Email - Use Caution Hello Matthew, Thanks for this info. If it is not too much work, just wondering if you could make a rough estimate or compute an average for the *total*memory* in use = (size of physical RAM in use by synthseg + size of all swap files resident on disk used by synthseg). - R. On Aug 10, 2023, at 16:08, Matthew Lynch <matthewl...@gmail.com> wrote: External Email - Use Caution Here is some additional follow up. I was able to get recon-all-clinical working in Ubuntu in WSL on a 16 Gb machine by increasing the swap and memory allowed in WSL. This is done through the .wslconfig file which is located in the %USERPROFILE% folder (e.g. C:\Users\username\). I had to specify a swap size of 32 Gb and also specify that WSL is allowed to use 32 Gb (even though the system only has 16 Gb -- it will use swap space).Here is the .wslconfig file I used:-----[wsl2]# Limits VM memory to use no more than x GB, this can be set as whole numbers using GB or MB# This must be set to 32 Gb or so to accomodate recon-all-clinical# If you set a 32 Gb swap but not 32 Gb memory allowance, WSL will stop at the memory limit and not use swapmemory=32GB # Sets amount of swap storage space to 8GB, default is 25% of available RAMswap=32GB# Sets swapfile path location, default is %USERPROFILE%\AppData\Local\Temp\swap.vhdx# I believe the double backslash is necessaryswapfile=P:\\Users\\username\\WSL\\wsl-swap.vhdx------For information on the .wslconfig file, see: MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/1qV7qOd8MV8Me_H9OrBCSVXR30wzEoGDkBeHd3EGrM14KqAV3QgSA3NPSYDCQmZEjI3IBScWH0I4l3nd8okmiZxGDigkkrYLyO8vmtch63i9fBH9ADb6hgH46SuT2KDYJspHszp9mhrrl0AMgFv5b1aEUW-VSCZ5X5JL69gRnhCNObH2-vbK7kUXnwAoGnffh0nYmeZQ5Re955E7SWL6oJgkZwZaIyX5xF1dwo2Nud50cb69HJdnhAqFYounJ-eTXA4LdozmcTGz5xfnVs77f1JZNiDIvS7xIoHFrG8ttVhPjrCp84LTSa2LvsDKU4eXHgYCPpzg2pX6gA5WC_PsqWA/https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fwindows%2Fwsl%2Fwsl-configOn Sat, Jul 29, 2023 at 6:27 AM Matthew Lynch <matthewl...@gmail.com> wrote:I was able to get recon-all-clinical working with 16 Gb computer.I thought the problem might be Windows hogging memory, so I created a live persistent USB install of lubuntu, using lubuntu 22 as a low resource distro in order to minimize memory usage, and installed freesurfer 7.4.0. Even with initial memory usage on 1Gb on lubuntu, synthseg still failed with out of memory error with 8 Gb swap file. I watched the swap memory fill up and then process was killed.I added 16Gb swap using the following instructions, for a total swap space of 24 Gb + 16 Gb memory:MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/1eJ0r45kveGk81B9YYbQBd38RPcxAhW6yy1xCpY2JgWABNj1jDLYcbEOyK_mTjZs1jNOU9o8FFfx9J2BZHUQVHx5x-vTaCnVyovnc-nEXs5V3qZ30_38Dg0p9cACa5w1_lKVc9YpaDJACf-4L0hBx6QzEc11mvu9_ygulTf4O-h7dGx8IpranJ1pdCgtC0BBGATBsIsn1MLnoFMOtCXnGww8HwOTQOzCc1i2H__pxiCkp2tkzZLl_J3MCxPJYhNGotULIBJAy9k6io9RAiTaF41ibYapFMgPVpdzJgvqfgx7fns_Q3LIcKTEtM8lsc1yGd2_Tfr1j6EXuSsoOYb3PWg/https%3A%2F%2Faskubuntu.com%2Fquestions%2F178712%2Fhow-to-increase-swap-spaceThis worked.On Thu, Jul 13, 2023 at 3:28 PM Matthew Lynch <matthewl...@gmail.com> wrote:I do not have access to a computer with more RAM. I can run recon-all with 1mm T1 images without any problems. How much RAM would recon-all-clinical be expected to need? Does it require less RAM if it uses the GPU instead of CPU? On Thu, Jul 13, 2023 at 1:58 PM Fischl, Bruce R.,PHD <bfis...@mgh.harvard.edu> wrote: External Email - Use Caution Can you try it on a computer with more ram? From: freesurfer-boun...@nmr.mgh.harvard.edu <freesurfer-boun...@nmr.mgh.harvard.edu> On Behalf Of Matthew LynchSent: Thursday, July 13, 2023 1:55 PMTo: freesur...@nmr.mgh.harvard.eduSubject: [Freesurfer] recon-all-clinical out of memory error External Email - Use Caution I am having trouble running the new recon-all-clinical module. I input 5mm T1 axial DICOM images and it fails in SynthSeg with what appears to be an out of memory error. I am running FreeSurfer 7.4.0 in WSL with Ubuntu 22 on a computer with 16 Gb RAM. Can anyone provide any thoughts on how to correct this? Thanks in advance! Here is the command: recon-all-clinical.sh ./T1AX5MM/119 fsclin 1 here is the error output: SynthSeg-robust 2.0using CPU, hiding all CUDA_VISIBLE_DEVICESusing 1 threadpredicting 1/1the following problem occured with image /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz :Traceback (most recent call last): File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input]) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler raise e.with_traceback(filtered_tb) from None File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most recent call last): File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 2642, in <module> main() File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 162, in main predict( File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line 315, in predict post_patch_segmentation, post_patch_parcellation, qc_score = net.predict([image, shape_input]) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2382, in predict tmp_batch_outputs = self.predict_function(iterator) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2169, in predict_function return step_function(self, iterator) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2155, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2143, in run_step outputs = model.predict_step(data) File "/usr/local/freesurfer/7..4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 2111, in predict_step return self(x, training=False) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", line 558, in __call__ return super().__call__(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 512, in call return self._run_internal_graph(inputs, training=training, mask=mask) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph outputs = node.layer(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1145, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils..py", line 96, in error_handler return fn(*args, **kwargs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py", line 196, in call return self._merge_function(inputs) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py", line 134, in _merge_function return backend.concatenate(inputs, axis=self.axis) File "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py", line 3581, in concatenate return tf.concat([to_dense(x) for x in tensors], axis)Node: 'model_5/unet_parc_merge_8/concat'OOM when allocating tensor with shape[1,256,256,160,72] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu [[{{node model_5/unet_parc_merge_8/concat}}]]Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [Op:__inference_predict_function_10473]resuming program execution _______________________________________________Freesurfer mailing listfreesur...@nmr.mgh.harvard.eduMailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/1sGGeEELL_ptsmtxu6jXuGs6vRitR3LycVsPzvStr8FHqFTsoBuDsU-3Kxfz3Ty6gv_bn2o_5Fou_l6MRmi8qhb_J_jBp0JFMF8BIst9Hh2dZSOCq7FJWAfm41-7iKE3mrB8DlVEXEnee1v_z65b38EYsjalUP-2Ju2tq46vBoozGSuLsui7-P-VyCsHe5W9tZ26QBIRcRfyYht7h5kGi3XaqgM2o8-ZjWnSN5BqclmHMIhQz_uEZnHs0BRNAINPOsd2XDsGbHTKPF_N5dCNVpOGLOOzhd2HWmMNxuDs8UujXERmaUWYsZMas6KYly-XJIG828bNELUp-c2sCEuUKoA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2FfreesurferThe information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/1GGJKJcSPmrVWsCmkwwqr3lYR6Sm50Ruyj3cm__lnXdLgW36kayZKh0Xw2dn-mmBEvFsuS42poez7lJow8WvJQP307l5PykKVPLMrhSIGL4jfGJVU33ECRSuarXO4vXxhd2euaXCb_VuKI5YKGE9DWE9xl0krzuyv3-53vM_JlLqNc6U06poRV0JIHf3kPGvd5a5Zn0N070kewoksMXggd6HRFHxX211S3q6Om1v0WGzittbuhUe1tUb8-qnW59kh-7wl6Fhf2ee-VeWSaobXXddDYlZ-LjcEwbzBqTYt4LMPVhTuPBmiHueQxz_p23PBYAm1pIPwaZ11LDfL_LukSw/https%3A%2F%2Fwww.massgeneralbrigham..org%2Fcomplianceline%26nbsp;<MailScanner has detected a possible fraud attempt from "secure-web.cisco.com" claiming to be https://secure-web.cisco.com/1cKDAfCLvXBlonZ3USIM_Kd00dKqRCnAU4rBkjU9tiySP4NeO-gF8_r-EqVhizQ5vc5DPadVbRWNoAyqrQtkCtZHWqjrzRgSYLNZcHlY4vKwuwgFoxLy6yaad2biNgtej8msuCKYZrCIyzIVh4y_WaxMuoGF75cDe4y8pEx1jJ1v-spL0igEtbdyAxQ8XpiwEqtLEIZTStDbTppuF220hIfUorDZwLe3BOTximbeBakfOQkDD3Gr7k7jVFfU3eBDXnqgmiEhBKktsHLFmUBdv6tVMfn6yEzBFFNqIoQ__4OJoygQRuK2l6UCySRCoUKv9M6OSY2J4r0YY-aC4ddGgzA/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline%26gt; ._______________________________________________Freesurfer mailing listfreesur...@nmr.mgh.harvard.eduhttps://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
