External Email - Use Caution The method I mentioned before does not always work. It worked for all but about 3 scans out of approximately 30. There are some scans for which the "committed memory" caps at about 23 Gb in the task manager and then either the out of memory error occurs or the WSL session is killed. Increasing the .wslconfig swap and memory beyond 32 Gb didn't work. Increasing the Windows pagefile size to 8 Gb didn't work. The output of free -h during the synthseg run was 0Gb used, 40Gb free, which seems to suggest it is not using the swap. I suspect this is a matter of somehow getting WSL to properly use large amounts of swap space.
On Thu, Aug 10, 2023 at 5:52 PM fsbuild <fsbu...@contbay.com> wrote: > Hello Matthew, > > Thanks for this info. > > If it is not too much work, just wondering if you could make a rough > estimate or compute an average for the *total*memory* in use = (size of > physical RAM in use by synthseg + size of all swap files resident on disk > used by synthseg). > > - R. > > On Aug 10, 2023, at 16:08, Matthew Lynch <matthewl...@gmail.com> wrote: > > External Email - Use Caution > > Here is some additional follow up. I was able to get recon-all-clinical > working in Ubuntu in WSL on a 16 Gb machine by increasing the swap and > memory allowed in WSL.. This is done through the .wslconfig file which is > located in the %USERPROFILE% folder (e.g. C:\Users\username\). I had to > specify a swap size of 32 Gb and also specify that WSL is allowed to use 32 > Gb (even though the system only has 16 Gb -- it will use swap space). > > Here is the .wslconfig file I used: > ----- > [wsl2] > > # Limits VM memory to use no more than x GB, this can be set as whole > numbers using GB or MB > # This must be set to 32 Gb or so to accomodate recon-all-clinical > # If you set a 32 Gb swap but not 32 Gb memory allowance, WSL will stop at > the memory limit and not use swap > memory=32GB > > # Sets amount of swap storage space to 8GB, default is 25% of available RAM > swap=32GB > > # Sets swapfile path location, default is > %USERPROFILE%\AppData\Local\Temp\swap.vhdx > # I believe the double backslash is necessary > swapfile=P:\\Users\\username\\WSL\\wsl-swap.vhdx > ------ > > For information on the .wslconfig file, see: *MailScanner has detected a > possible fraud attempt from "secure-web.cisco.com" claiming to be* > https://secure-web.cisco.com/1R9bbKcMD8m9Iba-TRQ62pSq0J_Ddg5C94ajqq7BXd_fbVgs6VGqnuPkYaI6ZDcu9NLaPE7lMDHy9lAzCrcphr6bsZLRFbQZzF4MWvRoR_PbphX012Jdo4VVUkclqw2EfcQDB717ZmZbfWgXfNRW-GPiUYqAakbNRgXJqa3wLU7OKD9gNyZXmuW_aISvDh2flGnHFq8HkqADXyypyekvureiTVE2TpURtac1BdduxYo18blHKpbaxn1kRWyPgpQATILi6AtheAZypd_rf4zg3tgSbdEC1qncpVovCwkRrQR3w-F4nTtVVflGTcBc_stqmABe_OjcVw0_B-uhAlpA92g/https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fwindows%2Fwsl%2Fwsl-config > <https://secure-web.cisco.com/1LNayjHy-1B5c1F-Uvbc7S2iZdU0Jv7SuKk5LkXsvEotEDyVYpDly5OU0naxoFLKh8oycxM1HnrZB2D3QAe7kjZO4VwUj0CV2ia_Ur93dhLdStO9YGvnPp1T8KYoYZ8A7SU_zjyQB9q9ZEi7XL01sOv12HVA3pEuB6kd3orUtCrIvisUeSzPIvYDMeUvcpq4A_uG1HjIF-fnvmVg9A33nNuNpSvIdykJz91NN-f-4LC9Bt6ltTemfRGXXDxenb-4_MKCs8ol-gZjauJ7RkwcbkGLcSMi3C9yY2H1aGvyG3PPWD7Nmgl0oaVF8XgRWUAjud_IN5AuL3R33m2S9TDJoYQ/https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fwindows%2Fwsl%2Fwsl-config> > > > On Sat, Jul 29, 2023 at 6:27 AM Matthew Lynch <matthewl...@gmail.com> > wrote: > >> >> I was able to get recon-all-clinical working with 16 Gb computer. >> >> I thought the problem might be Windows hogging memory, so I created a >> live persistent USB install of lubuntu, using lubuntu 22 as a low resource >> distro in order to minimize memory usage, and installed freesurfer 7.4.0. >> Even with initial memory usage on 1Gb on lubuntu, synthseg still failed >> with out of memory error with 8 Gb swap file. I watched the swap memory >> fill up and then process was killed. >> >> I added 16Gb swap using the following instructions, for a total swap >> space of 24 Gb + 16 Gb memory: >> >> *MailScanner has detected a possible fraud attempt from >> "secure-web.cisco.com" claiming to be* >> https://secure-web.cisco.com/1jwI6G6GqqL8t7VhAFnki1fqPD46CaZzoct10KHE5JEIzVb2KnNMEPKOO9-sktJaM2kkvSg8-MY9svnsqbNsXpr444upYwiJRVoQzqgYy-595-LhU9OxZSjNEhUFkLqAnMWh_a-jpyL2Gfk86f34D45SbFUidGfjzbt2UpNnfTuOToHnTtU0Ul5OCA_IjUTF-49U2YHD8C2iVUpz0eJqPsfsJd-4NfX-ZcQUgcWroZxUMF2lJNze7OdaMrIEq6wnLn8SbWEfBxOYZ7LBfFO6CFvViSl72mH-i1gj2chNaoohZ6Dty90dFscxWgTS9mg8dWdBd-43B3VBQVsbQ4LTSCA/https%3A%2F%2Faskubuntu.com%2Fquestions%2F178712%2Fhow-to-increase-swap-space >> <https://secure-web.cisco.com/1ATUFZLp99G3zWsjWUI2HJaukRqMFcIimcs9TpbaIY-04Afcv6AnAqhXtv_tgDukEBDaKwCk0vi98Yo_CGHy_K0ixuTpGpvNyTNrWtDzAn3HiIrwLCM0XRljKu-f0ZBRxrw6K2OD_pOvEiIQtQD3oWtGq6wYRyF-4IBg22YWXBPz8JFai2hLSUOT0905dDFEps0_6H-G9XMI2XWSouiBCiL6-EkfX5ep2GT4dJKuI3x4U4yWDynhWh-LHvyQ4yqlzSO8lpYOVJyF4LdbcOLYGjP2u3NwKvCObnw3m3o59ENYo-LjZX3jSA241KQ_KYXiyiwemGYB_YOXHMZ0rBN_e_g/https%3A%2F%2Faskubuntu.com%2Fquestions%2F178712%2Fhow-to-increase-swap-space> >> >> This worked. >> >> >> >> >> >> On Thu, Jul 13, 2023 at 3:28 PM Matthew Lynch <matthewl...@gmail.com> >> wrote: >> >>> I do not have access to a computer with more RAM. I can run recon-all >>> with 1mm T1 images without any problems. How much RAM would >>> recon-all-clinical be expected to need? Does it require less RAM if it >>> uses the GPU instead of CPU? >>> >>> On Thu, Jul 13, 2023 at 1:58 PM Fischl, Bruce R.,PHD < >>> bfis...@mgh.harvard.edu> wrote: >>> >>>> External Email - Use Caution >>>> >>>> Can you try it on a computer with more ram? >>>> >>>> >>>> >>>> *From:* freesurfer-boun...@nmr..mgh.harvard.edu >>>> <freesurfer-boun...@nmr.mgh.harvard.edu> < >>>> freesurfer-boun...@nmr.mgh.harvard.edu> *On Behalf Of *Matthew Lynch >>>> *Sent:* Thursday, July 13, 2023 1:55 PM >>>> *To:* freesurfer@nmr.mgh.harvard.edu >>>> *Subject:* [Freesurfer] recon-all-clinical out of memory error >>>> >>>> >>>> >>>> * External Email - Use Caution * >>>> >>>> I am having trouble running the new recon-all-clinical module. I input >>>> 5mm T1 axial DICOM images and it fails in SynthSeg with what appears to be >>>> an out of memory error. I am running FreeSurfer 7.4.0 in WSL with Ubuntu >>>> 22 on a computer with 16 Gb RAM. Can anyone provide any thoughts on how >>>> to correct this? Thanks in advance! >>>> >>>> >>>> >>>> Here is the command: >>>> >>>> >>>> >>>> recon-all-clinical.sh ./T1AX5MM/119 fsclin 1 >>>> >>>> >>>> >>>> >>>> >>>> here is the error output: >>>> >>>> >>>> >>>> SynthSeg-robust 2.0 >>>> using CPU, hiding all CUDA_VISIBLE_DEVICES >>>> using 1 thread >>>> predicting 1/1 >>>> >>>> the following problem occured with image >>>> /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz : >>>> Traceback (most recent call last): >>>> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", line >>>> 315, in predict >>>> post_patch_segmentation, post_patch_parcellation, qc_score = >>>> net.predict([image, shape_input]) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>> line 70, in error_handler >>>> raise e.with_traceback(filtered_tb) from None >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", >>>> line 52, in quick_execute >>>> tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, >>>> op_name, >>>> tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph >>>> execution error: >>>> >>>> Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most >>>> recent call last): >>>> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", >>>> line 2642, in <module> >>>> main() >>>> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", >>>> line 162, in main >>>> predict( >>>> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", >>>> line 315, in predict >>>> post_patch_segmentation, post_patch_parcellation, qc_score = >>>> net.predict([image, shape_input]) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>> line 65, in error_handler >>>> return fn(*args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>> line 2382, in predict >>>> tmp_batch_outputs = self.predict_function(iterator) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>> line 2169, in predict_function >>>> return step_function(self, iterator) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>> line 2155, in step_function >>>> outputs = model.distribute_strategy.run(run_step, args=(data,)) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>> line 2143, in run_step >>>> outputs = model.predict_step(data) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>> line 2111, in predict_step >>>> return self(x, training=False) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>> line 65, in error_handler >>>> return fn(*args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>> line 558, in __call__ >>>> return super().__call__(*args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>> line 65, in error_handler >>>> return fn(*args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", >>>> line 1145, in __call__ >>>> outputs = call_fn(inputs, *args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>> line 96, in error_handler >>>> return fn(*args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional..py", >>>> line 512, in call >>>> return self._run_internal_graph(inputs, training=training, >>>> mask=mask) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", >>>> line 669, in _run_internal_graph >>>> outputs = node.layer(*args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>> line 65, in error_handler >>>> return fn(*args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", >>>> line 1145, in __call__ >>>> outputs = call_fn(inputs, *args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>> line 96, in error_handler >>>> return fn(*args, **kwargs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py", >>>> line 196, in call >>>> return self._merge_function(inputs) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py", >>>> line 134, in _merge_function >>>> return backend.concatenate(inputs, axis=self..axis) >>>> File >>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py", >>>> line 3581, in concatenate >>>> return tf.concat([to_dense(x) for x in tensors], axis) >>>> Node: 'model_5/unet_parc_merge_8/concat' >>>> OOM when allocating tensor with shape[1,256,256,160,72] and type float >>>> on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu >>>> [[{{node model_5/unet_parc_merge_8/concat}}]] >>>> Hint: If you want to see a list of allocated tensors when OOM happens, >>>> add report_tensor_allocations_upon_oom to RunOptions for current allocation >>>> info. This isn't available when running in Eager mode. >>>> [Op:__inference_predict_function_10473] >>>> >>>> resuming program execution >>>> >>>> >>>> _______________________________________________ >>>> Freesurfer mailing list >>>> Freesurfer@nmr.mgh.harvard.edu >>>> *MailScanner has detected a possible fraud attempt from >>>> "secure-web.cisco.com" claiming to be* >>>> https://secure-web.cisco.com/1wiBjrpf8z7Msz3loto_C4ERTEmEXQYbJvVAXHqgtRlWo1avZbDT7GY8qTivZHYkE_dwh3VfSIwrlorQdh1LcmhSr4Jvi_7w-UBtd2w4qmfAz1PLVu-B34JU1VCd0gJFLKKeR4tsjg7ltm0O3LuBqCQL2TU99ElpyJCSKcOitQpcikF4ZALogBnP16Kw9DHwCCAzH3RdgfJ8Smv1aBPrRyHZbX4BiuVOgvwe0jRgrJ-OxXAN1rilO29XScv1TVXSy8LSBjD9kdy2n0K73yJ6HV-MBkyqxagLMBS1yYI90roAQpUSTocxUCd45tXIINHg8UJr0MHuH1428C1gpvOmddA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer >>>> <https://secure-web.cisco.com/1u0nl-K2x_1OFm48c9dzdc0w67ql_Fuoe1wDdrGZUXsP41rs-0R8LoUxRB6x928sxV45Ykeqgnb7pPrIl6E0CGK3gZIVGKlx33l3S-Iy7FPHtyKGlW9CfiKxurLrYJccTOPVpNziCA0d11V_n0M7dKMrRZiHWMMbMEQzSkFIx5Q1tg8XTqj-vsALEm6hqYpC4vj1hN5OLvQeEx1rQ-v9isUFyxkBzpZmiqpQfDxkAcwoxJcI01dHs_6rR4NuR908kLjKXBVvZC1r-tiJUVp-vr_Mjhnq1D-BCe8y06KvxlX8sQTzp0ZtoyovZybr_EQIcF9rA8irOx_IDxErbPqim3A/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer> >>>> The information in this e-mail is intended only for the person to whom >>>> it is addressed. If you believe this e-mail was sent to you in error and >>>> the e-mail contains patient information, please contact the Mass General >>>> Brigham Compliance HelpLine at *MailScanner has detected a possible >>>> fraud attempt from "secure-web.cisco.com" claiming to be* >>>> https://secure-web.cisco.com/17syISX6IHwnQPZyJ6dw--ARhb3l4gtgc0dDGDP33-fnKAScnJdMNyHv6dPTVvbQxoTklBK1SZKwc4Nnfb25flSopHdcufkf1vwbF1JMnG8VFl0GmU_T3xs4xHgx-wi0OzJv1UU6PLxmn1-aUDUbrfnA1LG7Ae9RhUxGtWtNi38Uh86aSyNusoHKtoBo9rJYOTidbkqAQE9ow6q3ceyxpuchL6X56WHND8I_CvythiRl8KLGci6FIOjVR_fWMigApjUi18R6Rn1W_oL2GPWswrOFnTcJR6_skB_ck9ssZzNA1MRTrx8YM_ZrlaYD1bbkTbk_GpKWNfqkoQvPvY8J-TQ/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline >>>> <https://secure-web.cisco.com/1oxN4Iv0FCNsxwCwS_hC9E1W9wfOKeQy2WR3sYrrTLusSFJNz9gq0Ghz6aOkgIHDO4a3bLBgc6vH2c0SsCepg57Y5__IOEYLbrrQ9tkxww_NfST1kprVLlN8Tg1-gLpddtEeKMKwZ5ZHaf_P5IQlMjoDP3MsPXjuGGJDuPJ7jOs4MSNou62vDAoyzJT-mwDdW_XPsqpGSOY1Y760dIrXgqHqSKEJbH1vNIJiMTFwILJ4KBQCj28uFBZx_1lX6qE5LLpJqP7KuN4r_Gmzg5yocfr0FfWV2-fe0_dHL0uQFfcmU7tkjXbQN9eYmgbWPxmQVsJM-bmQtSHOnyc8WUSvS8g/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline> >>>> <*MailScanner has detected a possible fraud attempt from >>>> "secure-web..cisco.com" claiming to be* >>>> https://secure-web.cisco.com/17syISX6IHwnQPZyJ6dw--ARhb3l4gtgc0dDGDP33-fnKAScnJdMNyHv6dPTVvbQxoTklBK1SZKwc4Nnfb25flSopHdcufkf1vwbF1JMnG8VFl0GmU_T3xs4xHgx-wi0OzJv1UU6PLxmn1-aUDUbrfnA1LG7Ae9RhUxGtWtNi38Uh86aSyNusoHKtoBo9rJYOTidbkqAQE9ow6q3ceyxpuchL6X56WHND8I_CvythiRl8KLGci6FIOjVR_fWMigApjUi18R6Rn1W_oL2GPWswrOFnTcJR6_skB_ck9ssZzNA1MRTrx8YM_ZrlaYD1bbkTbk_GpKWNfqkoQvPvY8J-TQ/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline >>>> <https://secure-web.cisco.com/1oxN4Iv0FCNsxwCwS_hC9E1W9wfOKeQy2WR3sYrrTLusSFJNz9gq0Ghz6aOkgIHDO4a3bLBgc6vH2c0SsCepg57Y5__IOEYLbrrQ9tkxww_NfST1kprVLlN8Tg1-gLpddtEeKMKwZ5ZHaf_P5IQlMjoDP3MsPXjuGGJDuPJ7jOs4MSNou62vDAoyzJT-mwDdW_XPsqpGSOY1Y760dIrXgqHqSKEJbH1vNIJiMTFwILJ4KBQCj28uFBZx_1lX6qE5LLpJqP7KuN4r_Gmzg5yocfr0FfWV2-fe0_dHL0uQFfcmU7tkjXbQN9eYmgbWPxmQVsJM-bmQtSHOnyc8WUSvS8g/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline>> >>>> . >>>> >>> _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://secure-web.cisco.com/1wiBjrpf8z7Msz3loto_C4ERTEmEXQYbJvVAXHqgtRlWo1avZbDT7GY8qTivZHYkE_dwh3VfSIwrlorQdh1LcmhSr4Jvi_7w-UBtd2w4qmfAz1PLVu-B34JU1VCd0gJFLKKeR4tsjg7ltm0O3LuBqCQL2TU99ElpyJCSKcOitQpcikF4ZALogBnP16Kw9DHwCCAzH3RdgfJ8Smv1aBPrRyHZbX4BiuVOgvwe0jRgrJ-OxXAN1rilO29XScv1TVXSy8LSBjD9kdy2n0K73yJ6HV-MBkyqxagLMBS1yYI90roAQpUSTocxUCd45tXIINHg8UJr0MHuH1428C1gpvOmddA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer > > >
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
