Re: [Rd] Inconsistency in median()
> Gustavo Zapata Wainberg > on Mon, 3 May 2021 20:48:49 +0200 writes: > Hi! > I'm wrinting this post because there is an inconsistency > when median() is calculated for even or odd vectors. For > odd vectors, attributes (such as labels added with Hmisc) > are kept after running median(), but this is not the case > if the vector is even, in this last case attributes are > lost. > I know that this is due to median() using mean() to obtain > the result when the vector is even, and mean() always > takes attributes off vectors. Yes, and this has been the design of median() for ever : If n := length(x) is odd, the median is "the middle" observation, and should equal to x[j] for j = (n+1)/2 and hence e.g., is well defined for an ordered factor. When n is even however, median() must be the mean of "the two middle" observations, which is e.g., not even *defined* for an ordered factor. We *could* talk of the so called lo-median or hi-median (terms probably coined by John W. Tukey) because (IIRC), these are equal to each other and to the median for odd n, but are equal to x[j] and x[j+1] j=n/2 for even n *and* are still "of the same kind" as x[] itself. Interestingly, for the mad() { = the median absolute deviation from the median} we *do* allow to specify logical 'low' and 'high', but that for the "outer" median in MAD's definition, not the inner one. ## From /src/library/stats/R/mad.R : mad <- function(x, center = median(x), constant = 1.4826, na.rm = FALSE, low = FALSE, high = FALSE) { if(na.rm) x <- x[!is.na(x)] n <- length(x) constant * if((low || high) && n%%2 == 0) { if(low && high) stop("'low' and 'high' cannot be both TRUE") n2 <- n %/% 2 + as.integer(high) sort(abs(x - center), partial = n2)[n2] } else median(abs(x - center)) } > Don't you think that attributes should be kept in both > cases? well, not all attributes can be kept. Note that for *named* vectors x, x[j] can (and does) keep the name, but there's definitely no sensible name to give to (x[j] + x[j+1])/2 I'm willing to collaborate with some, considering to extend median.default() making hi-median and lo-median available to the user. Both of these will always return x[j] for some j and hence keep all (sensible!) attributes (well, if the `[`-method for the corresponding class has been defined correctly; I've encountered quite a few cases where people created vector-like classes but did not provide a "correct" subsetting method (typically you should make sure both a `[[` and `[` method works!). Best regards, Martin Martin Maechler ETH Zurich and R Core team > And, going further, shouldn't mean() keep > attributes as well? I have looked in R's Bugzilla and I > didn't find an entry related to this issue. > Please, let me know if you consider that this issue should > be posted in R's bugzilla. > Here is an example with code. > rndvar <- rnorm(n = 100) > Hmisc::label(rndvar) <- "A label for RNDVAR" > str(median(rndvar[-c(1,2)])) > Returns: "num 0.0368" > str(median(rndvar[-1])) > Returns: 'labelled' num 0.0322 - attr(*, "label")= chr "A > label for RNDVAR" > Thanks in advance! > Gustavo Zapata-Wainberg > [[alternative HTML version deleted]] > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Testing R build when using --without-recommended-packages?
I'm on Linux (Ubuntu 18.04). How do I check an R build when using --without-recommended-packages? 'make check' assumes 'recommended' packages are installed, so that fails without them available. DETAILS: When I build R from source without 'recommended' packages: curl -O https://cran.r-project.org/src/base-prerelease/R-latest.tar.gz tar xvfz R-latest.tar.gz cd R-beta ./configure --enable-memory-profiling --enable-R-shlib --prefix="$PREFIX" make I cannot figure out how to validate the build. Following Section 'Installation' of 'R Installation and Administration', I run: make check results in: Testing examples for package ‘stats’ Error: testing 'stats' failed Execution halted This is because those tests assume 'MASS' is installed; $ cat /path/to/tests/Examples/stats-Ex.Rout.fail > utils::data(muscle, package = "MASS") Error in find.package(package, lib.loc, verbose = verbose) : there is no package called ‘MASS’ Calls: -> find.package Execution halted BTW, isn't this a bug? Shouldn't this example run conditionally on 'MASS' being installed, because 'MASS' is a suggested package here; Package: stats Version: 4.1.0 ... Imports: utils, grDevices, graphics Suggests: MASS, Matrix, SuppDists, methods, stats4 /Henrik __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
On 4 May 2021 at 09:31, Henrik Bengtsson wrote: | I'm on Linux (Ubuntu 18.04). How do I check an R build when using | --without-recommended-packages? 'make check' assumes 'recommended' | packages are installed, so that fails without them available. [...] | BTW, isn't this a bug? Shouldn't this example run conditionally on | 'MASS' being installed, because 'MASS' is a suggested package here; The 'R-admin' manual in Section 1.2 "Getting patched and development versions" ends on If downloading manually from CRAN, do ensure that you have the correct versions of the recommended packages: if the number in the file VERSION is ‘x.y.z’ you need to download the contents of ‘https://CRAN.R-project.org/src/contrib/dir’, where dir is ‘x.y.z/Recommended’ for r-devel or x.y-patched/Recommended for r-patched, respectively, to directory src/library/Recommended in the sources you have unpacked. After downloading manually you need to execute tools/link-recommended from the top level of the sources to make the requisite links in src/library/Recommended. A suitable incantation from the top level of the R sources using wget might be (for the correct value of dir) wget -r -l1 --no-parent -A\*.gz -nd -P src/library/Recommended \ https://CRAN.R-project.org/src/contrib/dir ./tools/link-recommended Dirk -- https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
Thanks, but I don't understand. That's what I usually do when I build R with 'recommended' packages. But here, I explicitly do *not* want to build and install 'recommended' packages with the R installation. So, I'm going down the --without-recommended-packages path on purpose and I'm looking for a way to validate such an installation. If your comment is on the 'stats' examples' hard dependency on 'MASS' despite it's being a suggested packages, I still don't follow. /Henrik On Tue, May 4, 2021 at 10:16 AM Dirk Eddelbuettel wrote: > > > On 4 May 2021 at 09:31, Henrik Bengtsson wrote: > | I'm on Linux (Ubuntu 18.04). How do I check an R build when using > | --without-recommended-packages? 'make check' assumes 'recommended' > | packages are installed, so that fails without them available. > > [...] > > | BTW, isn't this a bug? Shouldn't this example run conditionally on > | 'MASS' being installed, because 'MASS' is a suggested package here; > > The 'R-admin' manual in Section 1.2 "Getting patched and development > versions" ends on > > If downloading manually from CRAN, do ensure that you have the correct > versions of the recommended packages: if the number in the file VERSION > is ‘x.y.z’ you need to download the contents of > ‘https://CRAN.R-project.org/src/contrib/dir’, where dir is > ‘x.y.z/Recommended’ for r-devel or x.y-patched/Recommended for r-patched, > respectively, to directory src/library/Recommended in the sources you > have unpacked. After downloading manually you need to execute > tools/link-recommended from the top level of the sources to make the > requisite links in src/library/Recommended. A suitable incantation from > the top level of the R sources using wget might be (for the correct > value of dir) > > wget -r -l1 --no-parent -A\*.gz -nd -P src/library/Recommended \ > https://CRAN.R-project.org/src/contrib/dir > ./tools/link-recommended > > Dirk > > -- > https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
On 4 May 2021 at 11:07, Henrik Bengtsson wrote: | Thanks, but I don't understand. That's what I usually do when I build | R with 'recommended' packages. But here, I explicitly do *not* want | to build and install 'recommended' packages with the R installation. | So, I'm going down the --without-recommended-packages path on purpose | and I'm looking for a way to validate such an installation. I understand the desire, and am sympathetic, but for all+ years I have been building R (or R-devel) from source this has never been optional. Nor has any optionality (for the build of R has a whole) been documented, at least as far as I know. Dirk -- https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
FWIW, $ ./configure --help ... --with-recommended-packages use/install recommended R packages [yes] /Henrik On Tue, May 4, 2021 at 11:17 AM Dirk Eddelbuettel wrote: > > > On 4 May 2021 at 11:07, Henrik Bengtsson wrote: > | Thanks, but I don't understand. That's what I usually do when I build > | R with 'recommended' packages. But here, I explicitly do *not* want > | to build and install 'recommended' packages with the R installation. > | So, I'm going down the --without-recommended-packages path on purpose > | and I'm looking for a way to validate such an installation. > > I understand the desire, and am sympathetic, but for all+ years I have been > building R (or R-devel) from source this has never been optional. Nor has any > optionality (for the build of R has a whole) been documented, at least as far > as I know. > > Dirk > > -- > https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
On 4 May 2021 at 11:25, Henrik Bengtsson wrote: | FWIW, | | $ ./configure --help | ... | --with-recommended-packages | use/install recommended R packages [yes] Of course. But look at the verb in your Subject: no optionality _in testing_ there. You obviously need to be able to build R itself to then build the recommended packages you need for testing. Dirk -- https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
Two questions to R Core: 1. Is R designed so that 'recommended' packages are optional, or should that be considered uncharted territories? 2. Can such an R build/installation be validated using existing check methods? -- Dirk, it's not clear to me whether you know for sure, or you draw conclusions based your long experience and reading. I think it's very important that others don't find this thread later on and read your comments as if they're the "truth" (unless they are). I haven't re-read it from start to finish, but there are passages in 'R Installation and Administration' suggesting you can build and install R without 'recommended' packages. For example, post-installation, Section 'Testing an Installation' suggests you can run (after making sure `make install-tests`): cd tests ../bin/R CMD make check but they fail the same way. The passage continuous "... and other useful targets are test-BasePackages and test-Recommended to run tests of the standard and recommended packages (if installed) respectively." (*). So, to me that hints at 'recommended' packages are optional just as they're "Priority: recommended". Further down, there's also a mentioning of: $ R_LIBS_USER="" R --vanilla > Sys.setenv(LC_COLLATE = "C", LC_TIME = "C", LANGUAGE = "en") > tools::testInstalledPackages(scope = "base") which also produces errors when 'recommended' packages are missing, e.g. "Failed with error: 'there is no package called 'nlme'". (*) BTW, '../bin/R CMD make test-BasePackages' gives "make: *** No rule to make target 'test-BasePackages'. Stop." Thanks, /Henrik On Tue, May 4, 2021 at 12:22 PM Dirk Eddelbuettel wrote: > > > On 4 May 2021 at 11:25, Henrik Bengtsson wrote: > | FWIW, > | > | $ ./configure --help > | ... > | --with-recommended-packages > | use/install recommended R packages [yes] > > Of course. But look at the verb in your Subject: no optionality _in testing_ > there. > > You obviously need to be able to build R itself to then build the recommended > packages you need for testing. > > Dirk > > -- > https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
Hi Henrik, A couple of things. Firstly, so far asI have ever heard, it's valid that a package have hard dependencies in its tests for packages listed only in Suggests. In fact, that is one of the stated purposes of Suggests. An argument could be made, I suppose, that the base packages should be under stricter guidelines, but stats isn't violating the letter or intention of Suggests by doing this. Secondly, I don't have time to dig through the make files/administration docs, but I do know that R CMD check has --no-stop-on-error, so you can either separately or as part of make check, use that option for stats (and elsewhere as needed?) and just know that the stats tests that depend on MASS are "false positive" (or, more accurately, missing value) test results, rather than real positives, and go from there. You could also "patch" the tests as part of your build process. Somewhere I worked had to do that for parts of the internet tests that were unable to get through the firewall. Best, ~G On Tue, May 4, 2021 at 1:04 PM Henrik Bengtsson wrote: > Two questions to R Core: > > 1. Is R designed so that 'recommended' packages are optional, or > should that be considered uncharted territories? > > 2. Can such an R build/installation be validated using existing check > methods? > > > -- > > Dirk, it's not clear to me whether you know for sure, or you draw > conclusions based your long experience and reading. I think it's very > important that others don't find this thread later on and read your > comments as if they're the "truth" (unless they are). I haven't > re-read it from start to finish, but there are passages in 'R > Installation and Administration' suggesting you can build and install > R without 'recommended' packages. For example, post-installation, > Section 'Testing an Installation' suggests you can run (after making > sure `make install-tests`): > > cd tests > ../bin/R CMD make check > > but they fail the same way. The passage continuous "... and other > useful targets are test-BasePackages and test-Recommended to run tests > of the standard and recommended packages (if installed) respectively." > (*). So, to me that hints at 'recommended' packages are optional just > as they're "Priority: recommended". Further down, there's also a > mentioning of: > > $ R_LIBS_USER="" R --vanilla > > Sys.setenv(LC_COLLATE = "C", LC_TIME = "C", LANGUAGE = "en") > > tools::testInstalledPackages(scope = "base") > > which also produces errors when 'recommended' packages are missing, > e.g. "Failed with error: 'there is no package called 'nlme'". > > (*) BTW, '../bin/R CMD make test-BasePackages' gives "make: *** No > rule to make target 'test-BasePackages'. Stop." > > Thanks, > > /Henrik > > On Tue, May 4, 2021 at 12:22 PM Dirk Eddelbuettel wrote: > > > > > > On 4 May 2021 at 11:25, Henrik Bengtsson wrote: > > | FWIW, > > | > > | $ ./configure --help > > | ... > > | --with-recommended-packages > > | use/install recommended R packages [yes] > > > > Of course. But look at the verb in your Subject: no optionality _in > testing_ there. > > > > You obviously need to be able to build R itself to then build the > recommended > > packages you need for testing. > > > > Dirk > > > > -- > > https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
Sorry if this has been pointed out already, but some relevant text from https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Suggested-packages > Note that someone wanting to run the examples/tests/vignettes may not have a suggested package available (and it may not even be possible to install it for that platform). The recommendation used to be to make their use conditional via if(require("pkgname")): this is OK if that conditioning is done in examples/tests/vignettes, although using if(requireNamespace("pkgname")) is preferred, if possible. ... > Some people have assumed that a ‘recommended’ package in ‘Suggests’ can safely be used unconditionally, but this is not so. (R can be installed without recommended packages, and which packages are ‘recommended’ may change.) On 5/4/21 5:10 PM, Gabriel Becker wrote: Hi Henrik, A couple of things. Firstly, so far asI have ever heard, it's valid that a package have hard dependencies in its tests for packages listed only in Suggests. In fact, that is one of the stated purposes of Suggests. An argument could be made, I suppose, that the base packages should be under stricter guidelines, but stats isn't violating the letter or intention of Suggests by doing this. Secondly, I don't have time to dig through the make files/administration docs, but I do know that R CMD check has --no-stop-on-error, so you can either separately or as part of make check, use that option for stats (and elsewhere as needed?) and just know that the stats tests that depend on MASS are "false positive" (or, more accurately, missing value) test results, rather than real positives, and go from there. You could also "patch" the tests as part of your build process. Somewhere I worked had to do that for parts of the internet tests that were unable to get through the firewall. Best, ~G On Tue, May 4, 2021 at 1:04 PM Henrik Bengtsson wrote: Two questions to R Core: 1. Is R designed so that 'recommended' packages are optional, or should that be considered uncharted territories? 2. Can such an R build/installation be validated using existing check methods? -- Dirk, it's not clear to me whether you know for sure, or you draw conclusions based your long experience and reading. I think it's very important that others don't find this thread later on and read your comments as if they're the "truth" (unless they are). I haven't re-read it from start to finish, but there are passages in 'R Installation and Administration' suggesting you can build and install R without 'recommended' packages. For example, post-installation, Section 'Testing an Installation' suggests you can run (after making sure `make install-tests`): cd tests ../bin/R CMD make check but they fail the same way. The passage continuous "... and other useful targets are test-BasePackages and test-Recommended to run tests of the standard and recommended packages (if installed) respectively." (*). So, to me that hints at 'recommended' packages are optional just as they're "Priority: recommended". Further down, there's also a mentioning of: $ R_LIBS_USER="" R --vanilla Sys.setenv(LC_COLLATE = "C", LC_TIME = "C", LANGUAGE = "en") tools::testInstalledPackages(scope = "base") which also produces errors when 'recommended' packages are missing, e.g. "Failed with error: 'there is no package called 'nlme'". (*) BTW, '../bin/R CMD make test-BasePackages' gives "make: *** No rule to make target 'test-BasePackages'. Stop." Thanks, /Henrik On Tue, May 4, 2021 at 12:22 PM Dirk Eddelbuettel wrote: On 4 May 2021 at 11:25, Henrik Bengtsson wrote: | FWIW, | | $ ./configure --help | ... | --with-recommended-packages | use/install recommended R packages [yes] Of course. But look at the verb in your Subject: no optionality _in testing_ there. You obviously need to be able to build R itself to then build the recommended packages you need for testing. Dirk -- https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
On 4 May 2021 at 14:10, Gabriel Becker wrote: | A couple of things. Firstly, so far asI have ever heard, it's valid that a | package have hard dependencies in its tests for packages listed only in | Suggests. In fact, that is one of the stated purposes of Suggests. An | argument could be made, I suppose, that the base packages should be under | stricter guidelines, but stats isn't violating the letter or intention of | Suggests by doing this. Like Ben, I also take the other side here and point you e.g. to the extended discussion between Duncan and myself on r-package-devel (look for the thread with Subject: "winUCRT failures"). Optional packages need testing for presence before they are used, and we are slowly moving in that direction. Repeating "but that's not what we used to do" is of limited interest as things do sometimes change for the better. Dirk -- https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Testing R build when using --without-recommended-packages?
Hmm, that's fair enough Ben, I stand corrected. I will say that this seems to be a pretty "soft" recommendation, as these things go, given that it isn't tested for by R CMD check, including with the -as-cran extensions. In principle, it seems like it could be, similar checks are made in package code for inappropriate external-package-symbol usage/ Either way, though, I suppose I have a number of packages which have been invisibly non-best-practices compliant for their entire lifetimes (or at least, the portion of that where they had tests/vignettes...). Best, ~G On Tue, May 4, 2021 at 2:22 PM Ben Bolker wrote: > >Sorry if this has been pointed out already, but some relevant text > from > > https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Suggested-packages > > > Note that someone wanting to run the examples/tests/vignettes may not > have a suggested package available (and it may not even be possible to > install it for that platform). The recommendation used to be to make > their use conditional via if(require("pkgname")): this is OK if that > conditioning is done in examples/tests/vignettes, although using > if(requireNamespace("pkgname")) is preferred, if possible. > > ... > > > Some people have assumed that a ‘recommended’ package in ‘Suggests’ > can safely be used unconditionally, but this is not so. (R can be > installed without recommended packages, and which packages are > ‘recommended’ may change.) > > > > On 5/4/21 5:10 PM, Gabriel Becker wrote: > > Hi Henrik, > > > > A couple of things. Firstly, so far asI have ever heard, it's valid that > a > > package have hard dependencies in its tests for packages listed only in > > Suggests. In fact, that is one of the stated purposes of Suggests. An > > argument could be made, I suppose, that the base packages should be under > > stricter guidelines, but stats isn't violating the letter or intention of > > Suggests by doing this. > > > > > > Secondly, I don't have time to dig through the make files/administration > > docs, but I do know that R CMD check has --no-stop-on-error, so you can > > either separately or as part of make check, use that option for stats > (and > > elsewhere as needed?) and just know that the stats tests that depend on > > MASS are "false positive" (or, more accurately, missing value) test > > results, rather than real positives, and go from there. > > > > You could also "patch" the tests as part of your build process. > Somewhere I > > worked had to do that for parts of the internet tests that were unable to > > get through the firewall. > > > > Best, > > ~G > > > > > > > > On Tue, May 4, 2021 at 1:04 PM Henrik Bengtsson < > henrik.bengts...@gmail.com> > > wrote: > > > >> Two questions to R Core: > >> > >> 1. Is R designed so that 'recommended' packages are optional, or > >> should that be considered uncharted territories? > >> > >> 2. Can such an R build/installation be validated using existing check > >> methods? > >> > >> > >> -- > >> > >> Dirk, it's not clear to me whether you know for sure, or you draw > >> conclusions based your long experience and reading. I think it's very > >> important that others don't find this thread later on and read your > >> comments as if they're the "truth" (unless they are). I haven't > >> re-read it from start to finish, but there are passages in 'R > >> Installation and Administration' suggesting you can build and install > >> R without 'recommended' packages. For example, post-installation, > >> Section 'Testing an Installation' suggests you can run (after making > >> sure `make install-tests`): > >> > >> cd tests > >> ../bin/R CMD make check > >> > >> but they fail the same way. The passage continuous "... and other > >> useful targets are test-BasePackages and test-Recommended to run tests > >> of the standard and recommended packages (if installed) respectively." > >> (*). So, to me that hints at 'recommended' packages are optional just > >> as they're "Priority: recommended". Further down, there's also a > >> mentioning of: > >> > >> $ R_LIBS_USER="" R --vanilla > >>> Sys.setenv(LC_COLLATE = "C", LC_TIME = "C", LANGUAGE = "en") > >>> tools::testInstalledPackages(scope = "base") > >> > >> which also produces errors when 'recommended' packages are missing, > >> e.g. "Failed with error: 'there is no package called 'nlme'". > >> > >> (*) BTW, '../bin/R CMD make test-BasePackages' gives "make: *** No > >> rule to make target 'test-BasePackages'. Stop." > >> > >> Thanks, > >> > >> /Henrik > >> > >> On Tue, May 4, 2021 at 12:22 PM Dirk Eddelbuettel > wrote: > >>> > >>> > >>> On 4 May 2021 at 11:25, Henrik Bengtsson wrote: > >>> | FWIW, > >>> | > >>> | $ ./configure --help > >>> | ... > >>> | --with-recommended-packages > >>> | use/install recommended R packages [yes] > >>> > >>> Of course. But look at the verb in your Subject: no optionality _in > >> testing_ there. > >>> > >>> You obviously need to be able to build R itself to then buil