External Email - Use Caution
Michael, I can't thank you enough.
Knowing that it writes to /tmp made all the difference.
In my case I mounted an external directory on /tmp instead of the
sessiondir approach, but it now runs fine.
Jasper
On Fri, 5 Oct 2018 at 23:47, Michael Krause <kra...@mpib-berlin.mpg.de>
wrote:
> Dear *,
>
> we also run Freesurfer on a cluster and are starting to use singularity, so
> naturally I'm curious about this issue.
>
> I was able to reproduce the error with a clean singularity run (-c -e).
> Then, I rebuilt a container to start nu_correct with "strace -f -e
> open,write" and I believe I found the problem:
>
> [pid 35994] open("/subs/testsub/mri/tmp.mri_nu_correct.mni.35410/nu1.imp",
> O_RDONLY) = 3
> [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
> [pid 35994] open("/tmp/minc-qKdMD4", O_RDWR|O_CREAT|O_TRUNC, 0666) = 5
> [pid 35994] write(5, "\0\0\0\0\0\0\0\0", 8) = 8
> [pid 35994] write(5,
> "CDF\1\0\0\0\0\0\0\0\n\0\0\0\3\0\0\0\6xspace\0\0\0\0\1\0"..., 8192) = 8192
> [pid 35994] write(5,
>
> "|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"...,
> 8192) = 8192
> [...]
> [pid 35994] write(5,
>
> "|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0|\360\0\0"...,
> 8192) = -1 ENOSPC (No space left on device)
> [pid 35994] write(2, "ncendef: ", 9ncendef: ) = 9
> [pid 35994] write(2, "ncid 5", 6ncid 5) = 6
> [pid 35994] write(2, ": No space left on device", 25: No space left on
> device) = 25
>
>
> So there are two issues here I think:
>
> 1) nu_correct is saving things to /tmp when it could just use ./tmp
> 2) the singularity default for "sessiondir max size" is 16MB, which
> includes
> /tmp, is too small for the minc-tmp files
>
> I changed the value from 16 to 1024 and it passed the nu_correct step.
>
> cheers,
> Michael
>
> On 10/5/18 5:57 PM, Greve, Douglas N.,Ph.D. wrote:
> > Not that I know of, but this is not a native FS script, it comes from the
> > MNI, so it could be doing something we don't understand. Can you do an
> > experiment for us? Can you run it outside of the container using reprozip
> > (or something equivalent) to see all the files it touches?
> >
> > On 10/5/18 5:31 AM, Jasper van den Bosch wrote:
> >>
> >> External Email - Use Caution
> >>
> >> We are trying to run recon_all inside an fmriprep singularity container,
> >> but are running into an issue with nu_correct. It says it cannot write
> to
> >> disk, however there is plenty of space on the tmpdir and output dir.
> Does
> >> this script try to write to any other locations? Because this would fail
> >> inside the container, where each writable location has to be 'mounted'.
> >>
> >> Error message:
> >>
> >> ncendef: ncid 5: No space left on device
> >> Error outputting volume: possibly disk full?
> >> nu_evaluate: crashed while running evaluate_field (termination
> status=139)
> >> nu_correct: crashed while running nu_evaluate (termination status=65280)
> >> ERROR: nu_correct
> >>
> >> Command ran:
> >>
> >> nu_correct -clobber ./tmp.mri_nu_correct.mni.1121/nu0.mnc
> >> ./tmp.mri_nu_correct.mni.1121/nu1.mnc -tmpdir
> >> ./tmp.mri_nu_correct.mni.1121/0/ -iterations 1000 -distance 50
> >> [xxx@yyyy.cluster:/data/BIDS/derivatives/freesurfer/sub-1/mri/]
> >> [2018-10-04 14:59:29] running:
> >> /opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen
> 0.15
> >> 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify
> -b_spline
> >> 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir
> >> ./tmp.mri_nu_correct.mni.1121/0/ ./tmp.mri_nu_correct.mni.1121/nu0.mnc
> >> ./tmp.mri_nu_correct.mni.1121/nu1.imp
> >>
> >>
> >> this issue at the fmriprep
> >> repo: https://github.com/poldracklab/fmriprep/issues/1308
> >> possibly related: https://github.com/freesurfer/freesurfer/issues/462
> >>
> >> Many thanks for any advice,
> >> Jasper van den Bosch
> >>
> >>
> >>
> >> _______________________________________________
> >> Freesurfer mailing list
> >> Freesurfer@nmr.mgh.harvard.edu
> >> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
> >
> >
> > _______________________________________________
> > Freesurfer mailing list
> > Freesurfer@nmr.mgh.harvard.edu
> > https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
> >
>
> External Email - Use Caution
>
> _______________________________________________
> Freesurfer mailing list
> Freesurfer@nmr.mgh.harvard.edu
> https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
_______________________________________________
Freesurfer mailing list
Freesurfer@nmr.mgh.harvard.edu
https://mail.nmr.mgh.harvard.edu/mailman/listinfo/freesurfer
