External Email - Use Caution Recon-all-clinical worked consistently with 32 Gb RAM. I used WSL Ubuntu with 26Gb allocated to WSL before it would use swap. I watched one run, and it used total physical and swap memory of about 28 Gb.
With 16Gb RAM it worked on about 75% of scans. I'm not sure why this didn't work on some scans with 16Gb RAM and 100Gb swap space. It would crash around 23 Gb total memory usage. I still wonder if there was some configuration that would have allowed it to use more disk space as swap. On Sat, Aug 19, 2023 at 10:45 PM fsbuild <fsbu...@contbay.com> wrote: > > Thanks for the updates. Not that it will likely help with the memory > underrun, but it should be straightforward to update Freesurfer from 7.2.0 > to 7.4.1 in the VM from the command line. > > You can use the Firefox browser in the VM to download the 7.4.1 installer > with the link, > > > https://secure-web.cisco.com/1qh9GqcyCJRYw9IPCV3zcxtF4nlkFXaGgUncXjaORpmG4FlI7I_HcpDNWptdiWcKMa25sAYzjBSikFGbJfaxjbE-WZ-vU1DmLifnHRpPyKe6cCk4hmu3rB2mWSJwaP45WRBwi6nHlkowPSy53LToGBR-vWBjPHwtpglCIrkdODGHkOKVjpOXIw3ulEc8sU06GOV4HG2tpFMAv8-OSD3ve-7d3Q0pNHnWnzQFKS5DbgpGEnbBOaeUH5jbMCMXJ96REnDA-OPmP-F72Fe3CuUfGFfXlD_2hzjVWmdxss1ViXxy4eFA_6uuvS203JGrzEChncGfyPjZbFXBs1FKygBxHFA/https%3A%2F%2Fsurfer.nmr.mgh.harvard.edu%2Fpub%2Fdist%2Ffreesurfer%2F7.4.1%2Ffreesurfer_ubuntu18-7.4..1_amd64.deb > > > In a terminal window, remove the existing freesurfer 7.2.0 installation, > $ sudo apt remove freesurfer > > Then go to where the 7.4.1 installer was downloaded and run, > $ sudo apt install ./freesurfer_ubuntu18-7.4.1_amd64.deb > > There was likely a soft link to the previous freesurfer distribution in > developer's home directory, so to update that, > > $ cd $HOME > $ rm -f freesurfer > $ ln -s /usr/local/freesurfer/7.4.1 freesurfer > $ ls freesurfer > <output deleted) > > ... check you have 7.4.1 ... > > $ cat freesurfer/build-stamp.txt > freesurfer-linux-ubuntu18_x86_64-7.4.1-20230614-7eb8460 > > > - R. > > On Aug 19, 2023, at 08:00, Matthew Lynch <matthewl...@gmail.com> wrote: > > External Email - Use Caution > > Thank you fsbuild for the suggestions. Unfortunately it still does not > work. I did not try the virtual machine, because I could only find version > 7.2.0 ova file on the freesurfer website. It appears the 7.4.0 version > does not yet have a virtual machine? I am trying to use > recon-all-clinical, which did not appear until 7.4.0. The standard > recon-all works fine on 1mm scans on my 16 Gb machine under WSL. But I am > trying to explore use of recon-all-clinical. > > It appeared as if WSL was not using any of the available swap space, so I > tried a few different swap and memory configurations in .wslconfig. Then I > tried to create a cgroup that would limit physical memory and force > freesurfer to use swap. This did cause freesurfer to use swap space, but > it still crashed with out of memory error when the combined swap and > physical memory got to about 23 Gb (out of a total of 100 Gb virtual > memory). > > I then created a total Linux version using Porteus Linux live USB and > unpacking the freesurfer 7.4.0 tar.gz and finding a few minor > dependencies. This is a fairly lightweight Linux distro that runs from USB > so I didn't have to create a dual boot. The base memory usage was about > 1.3 Gb before running recon-all-clinical. I also activated a 16 Gb swap > file with swapon. It didn't work either, even in a pure Linux > environment. At maximum memory usage, it used only about 7-9% of the 16 Gb > swap space and I think it also crashed at the same place around 21 Gb total > memory usage with an OOM error. > > So it appears to be complaining of out of memory even when there is ample > memory available in the form of swap space. I have tried various options > for vm.overcommit and vm.swappiness. > > This seems to occur only for a few selected scans. For most of them it > works. There is a long error chain but this seems to be the relevant line: > > OOM when allocating tensor with shape[1,288,288,192,72] and type float on > /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu > [[{{node model_5/unet_parc_merge_8/concat}}]] > > It fails there every time. > > I bit the bullet and bought an additional 16 Gb physical RAM for a total > of 32 Gb. I'll let you know if that works. > > On Sun, Aug 13, 2023 at 10:11 PM fsbuild <fsbu...@contbay.com> wrote: > >> You might be better off with swapping inside the Linux VM instead of on >> the windows host. Or I don’t see anything preventing the Windows host from >> trying to swap out the entire VM process - which I would not expect to be >> efficient. You only need the application running *inside*the*Linux*VM* to >> be swapped out if necessary (using the Linux OS to do that). >> >> This should be straightforward to setup using VirtualBox to run the >> Ubuntu Linux VM we provide with Freesurfer already installed, i.e., >> download from >> >> *MailScanner has detected a possible fraud attempt from >> "secure-web.cisco.com" claiming to be* >> https://secure-web.cisco.com/1hVotqbPRUPT7sZtYDt0KRWz0I2ev0gP_dZMNndcQqh9zQwH3b214M7A5S3skb3jS20CuZflrfVFQJdKSTYhOaelOEwxVcwsLCbsYQ5v5xnI_LBYJvomLsJzdvqVwiEN38lJ3MSwK2GR9Mlc93__ZniogUpVyaXQFcAWvLw7t3b2hO3tzneENUjzwo5blz9pa54nECtjSbWeCNWHz1OuvkjXWz4K_kYoifKPrpBaVjD4w784C0Nejn9mEHn0CDsq4ax5oMDaEPpxlqh4qupDn_xSJIunZP6OY60ZaxNib0IoXhgddVgzf14k4lpSWKFEPtaXW-2jvu0OBz6ZbrKPFgw/https%3A%2F%2Fsurfer.nmr.mgh.harvard.edu%2Ffswiki%2FVM_67 >> <https://secure-web.cisco.com/1QjFp0WYmYLEMcfBOnXqbkQWZMeUHZaYYlKuZbGX26V8Vnk2ey_dREmr4sJ-WzYsL75GZjH-ZV4Oyk9pw_e5SHF4KorfUOcpPUi5hd8dgRWJrqz3KoWxCAC8Rj53KB1IZ9HkS-RBrl4YNlDQJofiV64p1CCKGaHFWQft2MxsVPSm27WyIvjTCzjO9ZkFYRXfd8lNO0US58QoJqtbxvKAbSp7zG4eV7PN01fZtOVLAw5m0ihpFXdKULLW-KPD-Xd1DR9-RgYjp4-IzqIj_aq859teaqgV0nLcfsnWLoQ9qSNHvCf7muxSmbzSzqdrbD7vk6LBOcvRIfWWTWRIbc2nA9w/https%3A%2F%2Fsurfer.nmr.mgh.harvard.edu%2Ffswiki%2FVM_67> >> >> >> In the screenshot below, the FreeSurfer VM starts with 4G allocated to >> it. So correspondingly, inside the VM the “free” command reports there is >> ~4G available of which only 1G is being used since nothing consequential is >> running. If you stopped the VM, increased the allocated memory in the >> VirtualBox application, and re-started the VM, you would see the total >> available memory scale up accordingly. >> >> For available swap space, you would look at the boot image on /dev/sda1. >> That image is made on purpose as small as possible so it can grow and/or >> space can otherwise be used. The “df” commands shows that out of the box, >> the VM takes up 22G of the max allowed size of 170G. That leaves >> potentially 170-22=148G to be used for other purposes. You certainly would >> not want to allocate all that for swap space, but there is space to use for >> a good chunk of swap space apart from space you will consume doing work >> that adds files into the VM. Here is 1 of many articles about setting up >> swapping in VirtualBox (which we do not setup by default), >> >> *MailScanner has detected a possible fraud attempt from >> "secure-web.cisco.com" claiming to be* >> https://secure-web.cisco.com/1CYP-Hz4Gra97p75REougwUpKAkAw6VAiHiVnWmrgNRItZzN5DJtDOKXDhz5RSJp7DoyL06noGZgT5ZysGZFqeMKVXNsTn6y_SOSeYo7xbE_XczQM0GlOLMTpZKT7rIA3Jixo2Z6p0IX4hS_mgk4OzN44qwLTJiuqja6p0IQSlfJaZP_j82ZbV1vP0npZY9t4XT_f09UusRpXfQbv4IDNuCANCEqz6x-VDbb6oQ0mTVOxmTUe6c4V0x05yVbVun_4zeuIfSQTVjU5iAlmK0SZaoFwAiwoftv6UBfBvZNZPhLeTs4wOL1mEgqB55LPqFpisrj3Mj8wb5JR0FstNPgRGA/https%3A%2F%2Fsciencetechvideos.wordpress.com%2F2019%2F01%2F09%2Fhow-to-add-swap-partition-or-swap-space-to-ubuntu-virtual-machine%2F >> <https://secure-web.cisco.com/1Q1M0ib1_obbWnUcE7Yr3awOIT-xkSxOF12MaXqwf5IIVtKXmy1WU9rMWC7GSXKQH2QEU0pJah9mpOKavuCNN4FuaflV410TS0w6_MqWJtt5Y-9rb1d6RRh-LsJKKoZkvUWpSwps1bwxQV5xpvapLwTd55X_s-DzyFok8qzxUPDCENCBnXNv8qCxZ0EUtrx1CF_nywimYKHCENTDFc_HTN7gNv1V7EGAUPKOKIAA6D38NO_zyvvNvopHdFbfME6HvgyN76YEymD-yNytqRuhY82x_nBhf9h1HGuCzL9lZ4TEl1JE4Im3XhG20-j--vHa_RIbkMzVK56-ZzEFptDGSlQ/https%3A%2F%2Fsciencetechvideos.wordpress.com%2F2019%2F01%2F09%2Fhow-to-add-swap-partition-or-swap-space-to-ubuntu-virtual-machine%2F> >> >> >> The efficiency of swapping will be limited by your system, e.g.., how >> fast the disks are, their bus interface, etc. There’s no guarantee even >> the most efficient swap configuration won’t cause the machine to slow down >> to a crawl depending upon what you are processing. It’s not a substitute >> for having RAM to allocate to the VM. We have discussed increasing the >> published memory system requirements for Freesurfer. >> >> - R. >> >> >> On Aug 13, 2023, at 06:39, Matthew Lynch <matthewl...@gmail.com> wrote: >> >> External Email - Use Caution >> >> The method I mentioned before does not always work. It worked for all >> but about 3 scans out of approximately 30. There are some scans for which >> the "committed memory" caps at about 23 Gb in the task manager and then >> either the out of memory error occurs or the WSL session is killed. >> Increasing the .wslconfig swap and memory beyond 32 Gb didn't work. >> Increasing the Windows pagefile size to 8 Gb didn't work. The output of >> free -h during the synthseg run was 0Gb used, 40Gb free, which seems to >> suggest it is not using the swap. I suspect this is a matter of somehow >> getting WSL to properly use large amounts of swap space. >> >> On Thu, Aug 10, 2023 at 5:52 PM fsbuild <fsbu...@contbay.com> wrote: >> >>> Hello Matthew, >>> >>> Thanks for this info. >>> >>> If it is not too much work, just wondering if you could make a rough >>> estimate or compute an average for the *total*memory* in use = (size of >>> physical RAM in use by synthseg + size of all swap files resident on disk >>> used by synthseg). >>> >>> - R. >>> >>> On Aug 10, 2023, at 16:08, Matthew Lynch <matthewl...@gmail.com> wrote: >>> >>> External Email - Use Caution >>> >>> Here is some additional follow up. I was able to get recon-all-clinical >>> working in Ubuntu in WSL on a 16 Gb machine by increasing the swap and >>> memory allowed in WSL.. This is done through the .wslconfig file which is >>> located in the %USERPROFILE% folder (e.g. C:\Users\username\). I had to >>> specify a swap size of 32 Gb and also specify that WSL is allowed to use 32 >>> Gb (even though the system only has 16 Gb -- it will use swap space). >>> >>> Here is the .wslconfig file I used: >>> ----- >>> [wsl2] >>> >>> # Limits VM memory to use no more than x GB, this can be set as whole >>> numbers using GB or MB >>> # This must be set to 32 Gb or so to accomodate recon-all-clinical >>> # If you set a 32 Gb swap but not 32 Gb memory allowance, WSL will stop >>> at the memory limit and not use swap >>> memory=32GB >>> >>> # Sets amount of swap storage space to 8GB, default is 25% of available >>> RAM >>> swap=32GB >>> >>> # Sets swapfile path location, default is >>> %USERPROFILE%\AppData\Local\Temp\swap.vhdx >>> # I believe the double backslash is necessary >>> swapfile=P:\\Users\\username\\WSL\\wsl-swap.vhdx >>> ------ >>> >>> For information on the .wslconfig file, see: *MailScanner has detected >>> a possible fraud attempt from "secure-web.cisco.com" claiming to be* >>> https://secure-web.cisco.com/1pCHT2pj4jex1saQgxl7WjYue9bVlKFec2rIwxsTJHj_qWisnpzgiov_NXlPPBq6lzohU81fg3Dw60sOIUgveXBOHFJFGxMCgDgU_NRfQxCYmxfeOTWbE7wtNB_LHM84lJidXIdBCDom4-NJlw21gtYWbPTzIiOwzZeDv2Y8_tVdG3iiGzE7Aw2HQrxLYjJ9_Iu4p5pDYkLAYIivrn77quAVYh-qh8MDyDZ9mjx5rqnQJLefiM9goiXdjSEJERCFsd7jB6fWoL5OY_r1JSU25MR8AtY4oH967myi7gUs5KxMJOCGFuyNKqKQQe1CRNL2Pe8BskYJWt5jfDOgaRbeZUw/https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fwindows%2Fwsl%2Fwsl-config >>> <https://secure-web.cisco.com/1LNayjHy-1B5c1F-Uvbc7S2iZdU0Jv7SuKk5LkXsvEotEDyVYpDly5OU0naxoFLKh8oycxM1HnrZB2D3QAe7kjZO4VwUj0CV2ia_Ur93dhLdStO9YGvnPp1T8KYoYZ8A7SU_zjyQB9q9ZEi7XL01sOv12HVA3pEuB6kd3orUtCrIvisUeSzPIvYDMeUvcpq4A_uG1HjIF-fnvmVg9A33nNuNpSvIdykJz91NN-f-4LC9Bt6ltTemfRGXXDxenb-4_MKCs8ol-gZjauJ7RkwcbkGLcSMi3C9yY2H1aGvyG3PPWD7Nmgl0oaVF8XgRWUAjud_IN5AuL3R33m2S9TDJoYQ/https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fwindows%2Fwsl%2Fwsl-config> >>> >>> >>> On Sat, Jul 29, 2023 at 6:27 AM Matthew Lynch <matthewl...@gmail.com> >>> wrote: >>> >>>> >>>> I was able to get recon-all-clinical working with 16 Gb computer. >>>> >>>> I thought the problem might be Windows hogging memory, so I created a >>>> live persistent USB install of lubuntu, using lubuntu 22 as a low resource >>>> distro in order to minimize memory usage, and installed freesurfer 7.4.0. >>>> Even with initial memory usage on 1Gb on lubuntu, synthseg still failed >>>> with out of memory error with 8 Gb swap file. I watched the swap memory >>>> fill up and then process was killed. >>>> >>>> I added 16Gb swap using the following instructions, for a total swap >>>> space of 24 Gb + 16 Gb memory: >>>> >>>> *MailScanner has detected a possible fraud attempt from >>>> "secure-web..cisco.com" claiming to be* >>>> https://secure-web.cisco.com/1LTUBGWKpYkeGaqxoG7dPiVBZtStz4SsqKuBjncl5kVz9i7_Ms-aMHRZTqA1sQrEZIc3-Kc_u1Fn8hVVswcJOQbl-52hSuuQL-av0hwOwY_Z95bpdTPPMyUV63eh5V-r39LuOBw6cI3QUthBZM6gUlw209v8gxO-kvtwdS91ZhwakThVxcssArdjan0pI44dQAmZRqGWdVsEjLZHtJ0DUS5BfwbrrwNtaVLv7roOHkNexs4bZOBxXPgf2eEVe9PuHqoYXleq74p438PuJCc_rgyaq2eHyYVi5Cn6dBAZj4zN7usrr6awcyPSdS5Ap1BG_tMXvI4CyNrh8GGZJF6VryQ/https%3A%2F%2Faskubuntu.com%2Fquestions%2F178712%2Fhow-to-increase-swap-space >>>> <https://secure-web.cisco.com/1ATUFZLp99G3zWsjWUI2HJaukRqMFcIimcs9TpbaIY-04Afcv6AnAqhXtv_tgDukEBDaKwCk0vi98Yo_CGHy_K0ixuTpGpvNyTNrWtDzAn3HiIrwLCM0XRljKu-f0ZBRxrw6K2OD_pOvEiIQtQD3oWtGq6wYRyF-4IBg22YWXBPz8JFai2hLSUOT0905dDFEps0_6H-G9XMI2XWSouiBCiL6-EkfX5ep2GT4dJKuI3x4U4yWDynhWh-LHvyQ4yqlzSO8lpYOVJyF4LdbcOLYGjP2u3NwKvCObnw3m3o59ENYo-LjZX3jSA241KQ_KYXiyiwemGYB_YOXHMZ0rBN_e_g/https%3A%2F%2Faskubuntu.com%2Fquestions%2F178712%2Fhow-to-increase-swap-space> >>>> >>>> This worked. >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Jul 13, 2023 at 3:28 PM Matthew Lynch <matthewl...@gmail.com> >>>> wrote: >>>> >>>>> I do not have access to a computer with more RAM. I can run recon-all >>>>> with 1mm T1 images without any problems. How much RAM would >>>>> recon-all-clinical be expected to need? Does it require less RAM if it >>>>> uses the GPU instead of CPU? >>>>> >>>>> On Thu, Jul 13, 2023 at 1:58 PM Fischl, Bruce R.,PHD < >>>>> bfis...@mgh.harvard.edu> wrote: >>>>> >>>>>> External Email - Use Caution >>>>>> >>>>>> Can you try it on a computer with more ram? >>>>>> >>>>>> >>>>>> >>>>>> *From:* freesurfer-boun...@nmr..mgh.harvard.edu >>>>>> <freesurfer-boun...@nmr.mgh.harvard.edu> < >>>>>> freesurfer-boun...@nmr.mgh.harvard.edu> *On Behalf Of *Matthew Lynch >>>>>> *Sent:* Thursday, July 13, 2023 1:55 PM >>>>>> *To:* freesurfer@nmr.mgh.harvard.edu >>>>>> *Subject:* [Freesurfer] recon-all-clinical out of memory error >>>>>> >>>>>> >>>>>> >>>>>> * External Email - Use Caution * >>>>>> >>>>>> I am having trouble running the new recon-all-clinical module. I >>>>>> input 5mm T1 axial DICOM images and it fails in SynthSeg with what >>>>>> appears >>>>>> to be an out of memory error.. I am running FreeSurfer 7.4.0 in WSL with >>>>>> Ubuntu 22 on a computer with 16 Gb RAM. Can anyone provide any thoughts >>>>>> on how to correct this? Thanks in advance! >>>>>> >>>>>> >>>>>> >>>>>> Here is the command: >>>>>> >>>>>> >>>>>> >>>>>> recon-all-clinical.sh ./T1AX5MM/119 fsclin 1 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> here is the error output: >>>>>> >>>>>> >>>>>> >>>>>> SynthSeg-robust 2.0 >>>>>> using CPU, hiding all CUDA_VISIBLE_DEVICES >>>>>> using 1 thread >>>>>> predicting 1/1 >>>>>> >>>>>> the following problem occured with image >>>>>> /mnt/p/FreeSurfer/5MM/WM20120718/fsclin/mri/native.mgz : >>>>>> Traceback (most recent call last): >>>>>> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", >>>>>> line 315, in predict >>>>>> post_patch_segmentation, post_patch_parcellation, qc_score = >>>>>> net.predict([image, shape_input]) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>>>> line 70, in error_handler >>>>>> raise e.with_traceback(filtered_tb) from None >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", >>>>>> line 52, in quick_execute >>>>>> tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, >>>>>> op_name, >>>>>> tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph >>>>>> execution error: >>>>>> >>>>>> Detected at node 'model_5/unet_parc_merge_8/concat' defined at (most >>>>>> recent call last): >>>>>> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", >>>>>> line 2642, in <module> >>>>>> main() >>>>>> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", >>>>>> line 162, in main >>>>>> predict( >>>>>> File "/usr/local/freesurfer/7.4.0/python/scripts/mri_synthseg", >>>>>> line 315, in predict >>>>>> post_patch_segmentation, post_patch_parcellation, qc_score = >>>>>> net.predict([image, shape_input]) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>>>> line 65, in error_handler >>>>>> return fn(*args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>>>> line 2382, in predict >>>>>> tmp_batch_outputs = self.predict_function(iterator) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3..8/site-packages/keras/engine/training.py", >>>>>> line 2169, in predict_function >>>>>> return step_function(self, iterator) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>>>> line 2155, in step_function >>>>>> outputs = model.distribute_strategy.run(run_step, args=(data,)) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>>>> line 2143, in run_step >>>>>> outputs = model.predict_step(data) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>>>> line 2111, in predict_step >>>>>> return self(x, training=False) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>>>> line 65, in error_handler >>>>>> return fn(*args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/training.py", >>>>>> line 558, in __call__ >>>>>> return super().__call__(*args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>>>> line 65, in error_handler >>>>>> return fn(*args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", >>>>>> line 1145, in __call__ >>>>>> outputs = call_fn(inputs, *args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>>>> line 96, in error_handler >>>>>> return fn(*args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3..8/site-packages/keras/engine/functional..py", >>>>>> line 512, in call >>>>>> return self._run_internal_graph(inputs, training=training, >>>>>> mask=mask) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/functional.py", >>>>>> line 669, in _run_internal_graph >>>>>> outputs = node.layer(*args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>>>> line 65, in error_handler >>>>>> return fn(*args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/engine/base_layer.py", >>>>>> line 1145, in __call__ >>>>>> outputs = call_fn(inputs, *args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/utils/traceback_utils.py", >>>>>> line 96, in error_handler >>>>>> return fn(*args, **kwargs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/base_merge.py", >>>>>> line 196, in call >>>>>> return self._merge_function(inputs) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/layers/merging/concatenate.py", >>>>>> line 134, in _merge_function >>>>>> return backend.concatenate(inputs, axis=self..axis) >>>>>> File >>>>>> "/usr/local/freesurfer/7.4.0/python/lib/python3.8/site-packages/keras/backend.py", >>>>>> line 3581, in concatenate >>>>>> return tf.concat([to_dense(x) for x in tensors], axis) >>>>>> Node: 'model_5/unet_parc_merge_8/concat' >>>>>> OOM when allocating tensor with shape[1,256,256,160,72] and type >>>>>> float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu >>>>>> [[{{node model_5/unet_parc_merge_8/concat}}]] >>>>>> Hint: If you want to see a list of allocated tensors when OOM >>>>>> happens, add report_tensor_allocations_upon_oom to RunOptions for current >>>>>> allocation info.. This isn't available when running in Eager mode. >>>>>> [Op:__inference_predict_function_10473] >>>>>> >>>>>> resuming program execution >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Freesurfer mailing list >>>>>> Freesurfer@nmr.mgh.harvard.edu >>>>>> *MailScanner has detected a possible fraud attempt from >>>>>> "secure-web.cisco.com" claiming to be* >>>>>> https://secure-web.cisco.com/1E7QpLd_NCJ5_32e0sU87SlSqE5keIZTezGhVoz7vOnPYiMU8SU8Jfxe8JloFA8LPjdTq0IaEtrGbHx484PUoDyDPtNBoQq8aVZqHVyzdMaLNY2gKSNLP_uSNxnAfi0W0rZyzPesOA8-FRE3eJM0vik9HUM-0f6cII16kHN27YPtyACMIxshpQ470CyaSlcGQHDTa6pvjNYpTRLuT3eD3b8pCwAVUnSKSd_aWe5R-AJPxvKcYeppIXGOdEHi7OKaR9OnqQXUpWPV3UfSACJfs167ghoZ6PizU69YWdUjWN15q71H0S4FJyJiYGb633UwwUlM4Hls2dtDwA-0AY9daJA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer >>>>>> <https://secure-web.cisco.com/1u0nl-K2x_1OFm48c9dzdc0w67ql_Fuoe1wDdrGZUXsP41rs-0R8LoUxRB6x928sxV45Ykeqgnb7pPrIl6E0CGK3gZIVGKlx33l3S-Iy7FPHtyKGlW9CfiKxurLrYJccTOPVpNziCA0d11V_n0M7dKMrRZiHWMMbMEQzSkFIx5Q1tg8XTqj-vsALEm6hqYpC4vj1hN5OLvQeEx1rQ-v9isUFyxkBzpZmiqpQfDxkAcwoxJcI01dHs_6rR4NuR908kLjKXBVvZC1r-tiJUVp-vr_Mjhnq1D-BCe8y06KvxlX8sQTzp0ZtoyovZybr_EQIcF9rA8irOx_IDxErbPqim3A/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer> >>>>>> The information in this e-mail is intended only for the person to >>>>>> whom it is addressed. If you believe this e-mail was sent to you in >>>>>> error >>>>>> and the e-mail contains patient information, please contact the Mass >>>>>> General Brigham Compliance HelpLine at *MailScanner has detected a >>>>>> possible fraud attempt from "secure-web.cisco.com" claiming to be* >>>>>> https://secure-web.cisco.com/1rYo_5FtCJMTIk7vCkFN3DvniVH-Rx96ReZnUnEXbS0F3yo6k7AawtoaNl_Q78epM2zSdq_f16xQaw58NI978LFmm7HSA_l9Clpbn2ptAiZ-TVY0SUAyKP-dtIREXoxnSM8DWFLPbwO1L4njz0EN95OQY4ddlGr5rcUA-8D0u4sbHL-ikigwdXQ9h0CNNIzIeFRmXhE5QeWbFaDwjbb-arFBidEpX_tLjP-iDFZ9Sx0JjFcGiYnXmjhsn1XOK7C2iH4o-3iMhqjKdb-KeRsk211AoaVvK9Ekl81y7SMq1rofuiKeX1OAgziWlo6WjouzdPByt1-1MICqF7QkNFy0qLA/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline >>>>>> <https://secure-web.cisco.com/1oxN4Iv0FCNsxwCwS_hC9E1W9wfOKeQy2WR3sYrrTLusSFJNz9gq0Ghz6aOkgIHDO4a3bLBgc6vH2c0SsCepg57Y5__IOEYLbrrQ9tkxww_NfST1kprVLlN8Tg1-gLpddtEeKMKwZ5ZHaf_P5IQlMjoDP3MsPXjuGGJDuPJ7jOs4MSNou62vDAoyzJT-mwDdW_XPsqpGSOY1Y760dIrXgqHqSKEJbH1vNIJiMTFwILJ4KBQCj28uFBZx_1lX6qE5LLpJqP7KuN4r_Gmzg5yocfr0FfWV2-fe0_dHL0uQFfcmU7tkjXbQN9eYmgbWPxmQVsJM-bmQtSHOnyc8WUSvS8g/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline> >>>>>> <*MailScanner has detected a possible fraud attempt from >>>>>> "secure-web..cisco.com" claiming to be* >>>>>> https://secure-web.cisco.com/1rYo_5FtCJMTIk7vCkFN3DvniVH-Rx96ReZnUnEXbS0F3yo6k7AawtoaNl_Q78epM2zSdq_f16xQaw58NI978LFmm7HSA_l9Clpbn2ptAiZ-TVY0SUAyKP-dtIREXoxnSM8DWFLPbwO1L4njz0EN95OQY4ddlGr5rcUA-8D0u4sbHL-ikigwdXQ9h0CNNIzIeFRmXhE5QeWbFaDwjbb-arFBidEpX_tLjP-iDFZ9Sx0JjFcGiYnXmjhsn1XOK7C2iH4o-3iMhqjKdb-KeRsk211AoaVvK9Ekl81y7SMq1rofuiKeX1OAgziWlo6WjouzdPByt1-1MICqF7QkNFy0qLA/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline >>>>>> <https://secure-web.cisco.com/1oxN4Iv0FCNsxwCwS_hC9E1W9wfOKeQy2WR3sYrrTLusSFJNz9gq0Ghz6aOkgIHDO4a3bLBgc6vH2c0SsCepg57Y5__IOEYLbrrQ9tkxww_NfST1kprVLlN8Tg1-gLpddtEeKMKwZ5ZHaf_P5IQlMjoDP3MsPXjuGGJDuPJ7jOs4MSNou62vDAoyzJT-mwDdW_XPsqpGSOY1Y760dIrXgqHqSKEJbH1vNIJiMTFwILJ4KBQCj28uFBZx_1lX6qE5LLpJqP7KuN4r_Gmzg5yocfr0FfWV2-fe0_dHL0uQFfcmU7tkjXbQN9eYmgbWPxmQVsJM-bmQtSHOnyc8WUSvS8g/https%3A%2F%2Fwww.massgeneralbrigham.org%2Fcomplianceline>> >>>>>> . >>>>>> >>>>> _______________________________________________ >>> Freesurfer mailing list >>> Freesurfer@nmr.mgh.harvard.edu >>> *MailScanner has detected a possible fraud attempt from >>> "secure-web.cisco.com" claiming to be* >>> https://secure-web.cisco.com/1E7QpLd_NCJ5_32e0sU87SlSqE5keIZTezGhVoz7vOnPYiMU8SU8Jfxe8JloFA8LPjdTq0IaEtrGbHx484PUoDyDPtNBoQq8aVZqHVyzdMaLNY2gKSNLP_uSNxnAfi0W0rZyzPesOA8-FRE3eJM0vik9HUM-0f6cII16kHN27YPtyACMIxshpQ470CyaSlcGQHDTa6pvjNYpTRLuT3eD3b8pCwAVUnSKSd_aWe5R-AJPxvKcYeppIXGOdEHi7OKaR9OnqQXUpWPV3UfSACJfs167ghoZ6PizU69YWdUjWN15q71H0S4FJyJiYGb633UwwUlM4Hls2dtDwA-0AY9daJA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer >>> <https://secure-web.cisco.com/1wiBjrpf8z7Msz3loto_C4ERTEmEXQYbJvVAXHqgtRlWo1avZbDT7GY8qTivZHYkE_dwh3VfSIwrlorQdh1LcmhSr4Jvi_7w-UBtd2w4qmfAz1PLVu-B34JU1VCd0gJFLKKeR4tsjg7ltm0O3LuBqCQL2TU99ElpyJCSKcOitQpcikF4ZALogBnP16Kw9DHwCCAzH3RdgfJ8Smv1aBPrRyHZbX4BiuVOgvwe0jRgrJ-OxXAN1rilO29XScv1TVXSy8LSBjD9kdy2n0K73yJ6HV-MBkyqxagLMBS1yYI90roAQpUSTocxUCd45tXIINHg8UJr0MHuH1428C1gpvOmddA/https%3A%2F%2Fmail..nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer> >>> >>> >>> _______________________________________________ >> Freesurfer mailing list >> Freesurfer@nmr.mgh.harvard.edu >> *MailScanner has detected a possible fraud attempt from >> "secure-web.cisco.com" claiming to be* >> https://secure-web.cisco.com/1E7QpLd_NCJ5_32e0sU87SlSqE5keIZTezGhVoz7vOnPYiMU8SU8Jfxe8JloFA8LPjdTq0IaEtrGbHx484PUoDyDPtNBoQq8aVZqHVyzdMaLNY2gKSNLP_uSNxnAfi0W0rZyzPesOA8-FRE3eJM0vik9HUM-0f6cII16kHN27YPtyACMIxshpQ470CyaSlcGQHDTa6pvjNYpTRLuT3eD3b8pCwAVUnSKSd_aWe5R-AJPxvKcYeppIXGOdEHi7OKaR9OnqQXUpWPV3UfSACJfs167ghoZ6PizU69YWdUjWN15q71H0S4FJyJiYGb633UwwUlM4Hls2dtDwA-0AY9daJA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer >> <https://secure-web.cisco.com/1sLDdPM9j6VYHvq6Rcgsth6peXMNTaCQH4qwUDjx6QrNja7j7duh9Ry3p8EV24vNY9XGJX1zD2h-HsbMnj9qwhg2xJoaBj7p1cOiLvImlgZkBfcU_TbQpNQ68WaVF2EPPsJCvoiC_CTzEAsxc4VMSnLNgNrgadLQkQ2Wwlonf6wyAkVR6lI4GCdQWWrTw8lo0Nr700EXUK7oTbnNJEa15ms1s0nD6Prq8_tWDP-FAEyZNQ6zVqNkD88JsnqTO0J4PF_iZarsmO9pUsCAcO4ftaOF6XkxmUg8kMmIS-hrH5iD-BLSCNn6vG49NZTPOvJG2XzTQXCl22qMnmeG_wXryuQ/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer> >> >> >> _______________________________________________ > Freesurfer mailing list > Freesurfer@nmr.mgh.harvard.edu > https://secure-web.cisco.com/1E7QpLd_NCJ5_32e0sU87SlSqE5keIZTezGhVoz7vOnPYiMU8SU8Jfxe8JloFA8LPjdTq0IaEtrGbHx484PUoDyDPtNBoQq8aVZqHVyzdMaLNY2gKSNLP_uSNxnAfi0W0rZyzPesOA8-FRE3eJM0vik9HUM-0f6cII16kHN27YPtyACMIxshpQ470CyaSlcGQHDTa6pvjNYpTRLuT3eD3b8pCwAVUnSKSd_aWe5R-AJPxvKcYeppIXGOdEHi7OKaR9OnqQXUpWPV3UfSACJfs167ghoZ6PizU69YWdUjWN15q71H0S4FJyJiYGb633UwwUlM4Hls2dtDwA-0AY9daJA/https%3A%2F%2Fmail.nmr.mgh.harvard.edu%2Fmailman%2Flistinfo%2Ffreesurfer > > >
_______________________________________________ Freesurfer mailing list Freesurfer@nmr.mgh.harvard.edu https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer Please note that this e-mail is not secure (encrypted). If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately. Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.
