The issue with data.table in R 3.5 and R 3.6 is a race condition in data.table's use of multiple threads via OpenMP. Running with the environment variable OMP_NUM_THREADS=1 makes the memory problem go away.
Calling DATAPTR for ALTREP objects, like compact arithmetic sequences, temporarily disables the GC. Calling DATAPTR from multiple threads without synchronization creates a race condition that can leave the GC disabled, so memory use grows. It's not a leak: signaling a user interrupt, or anything that triggers a jump to top level, re-enables the GC. The issue exists in R 3.5 as well but is a little less apparent since, due to an oversight, byte compiled code often does not produce compact arithmetic sequences. This will hopefully be addressed in R 3.5.1. Best, luke On Mon, 14 May 2018, Bailey, Paul wrote:
A package I'm submitting depends on data.table, which currently has a memory leak on Windows and so I'm getting rejected because of "memory exhausted (limit reached?)". I'm unsure how to proceed. It seems the issue is with R3.6.0 and data.table, so I don't think users would have an immediate issue if my package were updated. I could limit the tests that use data.table to keep under a memory size, but how much memory does CRAN allocate on Windows, and how do I enforce that? You can see the memory leak issue on GitHub at https://github.com/Rdatatable/data.table/issues/2866 Best, Paul ______________________________________________ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
-- Luke Tierney Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics and Fax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke-tier...@uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu ______________________________________________ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
