Thanks guys for your kind help! I would follow your tips and try to fix these issues.
It's indeed quite hard to maintain it, luckily in past two years I made it. It's indeed the right time to cut down some unnecessary packages and make it easier to be maintained. Thanks for your help! ^_^ Best Yuan Tian On Thu, Mar 22, 2018 at 12:14 AM Hervé Pagès <hpa...@fredhutch.org> wrote: > Hi, > > On 03/21/2018 07:32 AM, Martin Morgan wrote: > > > > > > On 03/21/2018 10:14 AM, Henrik Bengtsson wrote: > >> A few quick comments: > >> > >> * I don't think the limit was hardened in R 3.4.0; maybe you started to > >> experience it sound then because more dependent packages started to > >> rely on > >> more DLLs? > >> > >> * The limit is OS/platform specific so it's not that R core is > >> unwilling to > >> fix this - I think they havejust tried to find a safe limit that work > >> everywhere. Having said this... > >> > >> * It'll be less restrictive by default in R 3.5.0: from it's NEWS "The > >> maximum number of DLLs that can be loaded into R e.g. via dyn.load() has > >> been increased up to 614 when the OS limit on the number of open files > >> allows" > >> > >> * R.utils::gcDLLs() will attempt to unregister possibly stray DLLs > >> remaining after unloading packages. So unloading packages and gcDLLs() > >> may > >> provide you with a workaround until R 3.5.0 is in place (possibly also > >> even > >> after). > >> > >> See > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_HenrikBengtsson_Wishlist-2Dfor-2DR_issues_29&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=7UNVEDPfEm1j53ksGc7naLgnieltNzOnbv4lPUQh4Jk&e= > >> for > >> "random" notes and references to this problem. > >> > >> > >> On Wed, Mar 21, 2018, 04:10 Yuan Tian <tianyuan1991...@gmail.com> > wrote: > >> > >>> Hello Guys: > >>> > >>> I encountered an error like "maximal number of dlls reached...". I am > >>> maintaining ChAMP package now, which integrated many other packages > >>> in my > >>> research field. I have not updated this package in past 2 months but > >>> suddenly this error happens. > >>> > >>> Currently, I think I know the reason is since R 3.4.X, numbers of > >>> DLLs in > >>> default R session was set 100. I have tested that using a newly > >>> started R > >>> 3.4.4 to install and load ChAMP package, *it works smoothly*, after > >>> loading, I checked DLL loaded with function getLoadedDLLs(), then I see > >>> ChAMP used 95 Dlls. I know it's a lot, but some of them are loaded by > >>> ChAMP's relying packages but itself. *However, ChAMP cannot pass > >>> Bioconductor check, thus I suspect Bioconductor checking system does > NOT > >>> start a new R session for each package right? Which means it's not > >>> 100 DLLs > >>> allocated for each package?* > > > > ChAMP is passing BiocCheck in release, where R is 3.4.x ? > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__bioconductor.org_checkResults_release_bioc-2DLATEST_ChAMP_malbec1-2Dchecksrc.html&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=_B73-e5Y2XkRdvs7ssHGb6DAdSnMmG30eE_Ywu1W7iQ&e= > > mmmh, this is a stale page. Not sure what it's doing here. Will > investigate. > > Today's "true" results are here: > > http://bioconductor.org/checkResults/release/bioc-LATEST/ChAMP/ > > Note that you can't get to the stale page by following links from > the report's main page: > > http://bioconductor.org/checkResults/release/bioc-LATEST/ > > > > > > > Each package is built and tested in a separate process. Vignettes are > > actually built in a single process (by R, not Bioconductor) so multiple > > vignettes could load a higher number of DLLs. > > Also just to clarify: the build machines don't use any special settings > to limit the nb of DLLs. They use whatever R uses by default. > > > > >>> > >>> Currently, I have told my users to modify R environment to allow their > R > >>> session with more DLLs. It works but only on users computer, not > >>> Bioconductor checking system. > > Note that, in release (with R 3.4.4), the max DLL error happens at > the end of an install: > > > biocLite("ChAMP") > BioC_mirror: https://bioconductor.org > Using Bioconductor 3.6 (BiocInstaller 1.28.0), R 3.4.3 (2017-11-30). > Installing package(s) ‘ChAMP’ > trying URL > ' > https://bioconductor.org/packages/3.6/bioc/src/contrib/ChAMP_2.9.10.tar.gz > ' > Content type 'application/x-gzip' length 15513863 bytes (14.8 MB) > ================================================== > downloaded 14.8 MB > > * installing *source* package ‘ChAMP’ ... > ** R > ** inst > ** preparing package for lazy loading > Warning: replacing previous import ‘igraph::edges’ by ‘graph::edges’ > when loading ‘FEM’ > Warning: replacing previous import ‘igraph::intersection’ by > ‘graph::intersection’ when loading ‘FEM’ > Warning: replacing previous import ‘igraph::degree’ by ‘graph::degree’ > when loading ‘FEM’ > Warning: replacing previous import ‘igraph::union’ by ‘graph::union’ > when loading ‘FEM’ > Warning: replacing previous import ‘limma::plotMA’ by > ‘BiocGenerics::plotMA’ when loading ‘FEM’ > Warning: replacing previous import ‘Matrix::colSums’ by > ‘BiocGenerics::colSums’ when loading ‘FEM’ > Warning: replacing previous import ‘Matrix::colMeans’ by > ‘BiocGenerics::colMeans’ when loading ‘FEM’ > Warning: replacing previous import ‘Matrix::rowMeans’ by > ‘BiocGenerics::rowMeans’ when loading ‘FEM’ > Warning: replacing previous import ‘Matrix::rowSums’ by > ‘BiocGenerics::rowSums’ when loading ‘FEM’ > Warning: replacing previous import ‘Matrix::which’ by > ‘BiocGenerics::which’ when loading ‘FEM’ > Warning: replacing previous import ‘igraph::normalize’ by > ‘BiocGenerics::normalize’ when loading ‘FEM’ > Warning: replacing previous import 'plyr::summarise' by > 'plotly::summarise' when loading 'ChAMP' > Warning: replacing previous import 'plyr::rename' by 'plotly::rename' > when loading 'ChAMP' > Warning: replacing previous import 'plyr::arrange' by 'plotly::arrange' > when loading 'ChAMP' > Warning: replacing previous import 'plyr::mutate' by 'plotly::mutate' > when loading 'ChAMP' > Error in dyn.load(file, DLLpath = DLLpath, ...) : > unable to load shared object > '/home/hpages/R/R-3.4.3/library/robustbase/libs/robustbase.so': > `maximal number of DLLs reached... > ERROR: lazy loading failed for package 'ChAMP' > * removing '/home/hpages/R/R-3.4.3/library/ChAMP' > > The downloaded source packages are in > ‘/tmp/Rtmp87p9oq/downloaded_packages’ > Updating HTML index of packages in '.Library' > Making 'packages.html' ... done > > So even if ChAMP's documentation says something about increasing the > max number of DLLs, most users are probably going to find this situation > frustrating and complain about it. > > >>> Thus I think I have to reduce dlls used by > >>> current package right? Like removing some relying packages or > >>> function. *My > >>> another question is how many DLLs is allowed by Bioconductor check? I > >>> think > >>> it's less than 100. But I don't know I should cut it into 80 or 60 or > >>> even > >>> 50 dlls used.* > >>> > >>> It's really disappointing that I need to modify quite a lot of code, > and > >>> even could hurt some key functionality of the package. Thus here I am > >>> seeking your help and suggestions here. > > > > Frankly, I think a package with so many dependencies cannot be > > maintained -- a change in any one of those packages could break your > > package (e.g., by changing their own dependencies to include additional > > DLLs!), and it must be virtually impossible to get sufficient test > > coverage to be confident that these problems will be detected before the > > package is made available to the user. It is time to consider a more > > modular design focusing on essential features. > > The first thing you could try to do is move some deps from Depends or > Imports to Suggests. Right now, after doing library(ChAMP), > sessionInfo() reports 36 packages on the search path and another 160 > packages loaded via a namespace! Couldn't some of those packages be > moved to Suggests without hurting the usability of ChAMP? > > One last thing: ChAMP is at version 2.9.10 in release and 2.9.11 in > devel. This is not good. The y part of x.y.z should always be even > in release. Generally speaking, you should only bump the z part of the > version when you make changes to your package. We take care of bumping > the y part for you at release time. See: > > https://bioconductor.org/developers/how-to/version-numbering/ > > Too late to fix in release though. Just be aware that when we release > BioC 3.7, we'll bump ChAMP version to 2.10.0 in the new release and > to 2.11.0 in the new devel (i.e. BioC 3.8). > > Cheers, > H. > > > > > Martin > > > >>> > >>> Best > >>> Yuan Tian > >>> > >>> [[alternative HTML version deleted]] > >>> > >>> _______________________________________________ > >>> Bioc-devel@r-project.org mailing list > >>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=Rs9gi9iOQtOG4zUXSOO07D7t_smZ_h6OPyISr1DHf8c&e= > >>> > >>> > >> > >> [[alternative HTML version deleted]] > >> > >> _______________________________________________ > >> Bioc-devel@r-project.org mailing list > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=Rs9gi9iOQtOG4zUXSOO07D7t_smZ_h6OPyISr1DHf8c&e= > >> > >> > > > > > > This email message may contain legally privileged and/or...{{dropped:2}} > > > > _______________________________________________ > > Bioc-devel@r-project.org mailing list > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=Rs9gi9iOQtOG4zUXSOO07D7t_smZ_h6OPyISr1DHf8c&e= > > > > -- > Hervé Pagès > > Program in Computational Biology > Division of Public Health Sciences > Fred Hutchinson Cancer Research Center > 1100 Fairview Ave. N, M1-B514 > P.O. Box 19024 > Seattle, WA 98109-1024 > > E-mail: hpa...@fredhutch.org > Phone: (206) 667-5791 > Fax: (206) 667-1319 > [[alternative HTML version deleted]] _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
