Dear all, I’m trying to understand how bplapply works with memory and what could be causing the following error - Error: 'bplapply' receive data failed: error reading from connection
When I get this error the system (ubuntu linux 16.04) seems to have filled up it’s memory then severs the connection to the slaves but the slaves stay alive and zombied. Unfortunately the way that the algorithm works it is hard to estimate the amount of memory required by each core prior to running. I have a small example below that seems to show the same behavior except that the zombies are later killed, freeing up the system incase the command was processed with a try except. So my question is - Is there a memory safe fashion to run bplapply or at least stop it from zombifying its slaves? Best regards, Paul PS: quick google also shows the issue here - https://github.com/YosefLab/scone/issues/69 without a fix. Attempt to get system to recreate issue on small example ---- [varangian ~] :) ulimit 20 [varangian ~] :) R R version 3.4.4 (2018-03-15) -- "Someone to Lean On” …. > library(BiocParallel) > options(MulticoreParam=quote(MulticoreParam(workers=4))) > lmat<-lapply(sample(1:100, 50), function(z){matrix(rnorm(100000, sd=z), > ncol=10000)}) > foo<-bplapply(lmat, cor, BPPARAM=SerialParam()) #### run in serial Error: cannot allocate vector of size 762.9 Mb > gc() …. > foo<-bplapply(lmat, cor, BPPARAM=MulticoreParam(4)) #### run in parallel on > reduced 4 cores Error: 'bplapply' receive data failed: cannot allocate vector of size 762.9 Mb > Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal From actual XCMS code error > xdata <- findChromPeaks(raw_data, param = cwp, BPPARAM=MulticoreParam(4)) Error: 'bplapply' receive data failed: error reading from connection > sessionInfo() Error in system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE) : cannot popen '/usr/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory' > system('free -m') Warning message: In system("free -m") : system call failed: Cannot allocate memory > q() ….. [varangian ~] :) free -m total used free shared buff/cache available Mem: 7983 7774 125 1 82 17 Swap: 16384 16186 198 [varangian ~] :) kill -9 $(pgrep R) [varangian ~] :) R > sessionInfo() ... R version 3.4.4 (2018-03-15) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 16.04.4 LTS Matrix products: default BLAS: /usr/lib/libblas/libblas.so.3.6.0 LAPACK: /usr/lib/lapack/liblapack.so.3.6.0 locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] parallel stats graphics grDevices utils datasets methods [8] base other attached packages: [1] xcms_3.1.1 MSnbase_2.4.2 ProtGenerics_1.10.0 [4] mzR_2.13.5 Rcpp_0.12.16 BiocParallel_1.12.0 [7] Biobase_2.38.0 BiocGenerics_0.24.0 loaded via a namespace (and not attached): [1] pillar_1.2.1 compiler_3.4.4 BiocInstaller_1.28.0 [4] RColorBrewer_1.1-2 plyr_1.8.4 tools_3.4.4 [7] iterators_1.0.9 zlibbioc_1.24.0 MALDIquant_1.17 [10] digest_0.6.15 preprocessCore_1.40.0 tibble_1.4.2 [13] gtable_0.2.0 lattice_0.20-35 rlang_0.2.0 [16] Matrix_1.2-14 foreach_1.4.4 S4Vectors_0.16.0 [19] IRanges_2.12.0 stats4_3.4.4 multtest_2.34.0 [22] grid_3.4.4 impute_1.52.0 survival_2.41-3 [25] XML_3.98-1.10 RANN_2.5.1 limma_3.34.9 [28] ggplot2_2.2.1 MASS_7.3-49 splines_3.4.4 [31] scales_0.5.0 pcaMethods_1.70.0 codetools_0.2-15 [34] MassSpecWavelet_1.44.0 mzID_1.16.0 colorspace_1.3-2 [37] affy_1.56.0 lazyeval_0.2.1 munsell_0.4.3 [40] doParallel_1.0.11 vsn_3.46.0 affyio_1.48.0 [[alternative HTML version deleted]] _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
