Hi Wolfgang, On 02/21/2013 01:23 PM, Wolfgang Huber wrote:
Hi HerveI have absolutely no objections against 'normalize' in BiocGenerics, I think it is a good idea.
OK, thanks!
However, the concept of a 'universal namespace' and that no package can mask symbols defined in other packages I find objectionable.
Just to clarify, I was not suggesting that no package can mask symbols defined in other packages, only that no BioC package should mask symbols defined in the BiocGenerics package. This is because of the special status of BiocGenerics, it's at the bottom of the BioC stack, and sooner or later, every BioC user will have it in its search path.
There is no 'redefining' of functions as you suggest below in (b). Rather, there are multiple co-existing definitions, associated with the same name, and there are clear and well-designed rules how to resolve such cases. For the Bioconductor project we seem to have gone down the path of finding that a certain amount of name policing and allocation of 'reserved function names' has more benefits than problems. I would vote for making this policy explicit and stating it in the package guidelines. I would also suggest using this approach sparingly... looking at BiocGenerics, there are currently 67 generics defined, and then there are 90 more in Biobase and 158 in IRanges. And some seem rather un-generic, like 'snpCallProbability' or 'pmax.int', or cryptic, such as 'NROW' and 'NCOL'. (Perhaps we're already closer to my suggestion of yesterday's post than I thought :)
But we can fix that. Let's replace those 315 generics defined in BiocGenerics, Biobase, and IRanges, by a single one called FUNCTION and force all packages to define methods for it ;-) Cheers, H.
Best wishes Wolfgang Il giorno Feb 21, 2013, alle ore 9:16 PM, Hervé Pagès <hpa...@fhcrc.org> ha scritto:Hi Laurent, Robert, and others, On 02/20/2013 01:33 PM, Laurent Gautier wrote:On 2013-02-20 22:02, Hervé Pagès wrote:Hi Robert, Nice to hear from you! I'm just a little bit worried that if we put normalize() in BiocGenerics without at the same time have a "take it or leave it" policy, then most of the authors of the 10 packages are likely going to ignore that there is now a new normalize() in BiocGenerics.There are roughly two schools for getting compliance: a- enforcement b- communication and education Start with b-, and most will comply I think (since it will not change their API). For new packages there are already few requirements to be accepted to bioconductor (all of which are not necessary fulfilled by all packages); this would only make one more requirement.Requirement == enforcement. Otherwise we reject the package. My understanding of Robert's email is that people might have good reasons for re-defining a generic that is already defined in BiocGenerics. So here is my attempt at wrapping this up: (a) There seems to be a consensus (Wolfgang?) in favor of adding the normalize() generic to BiocGenerics. (b) We should strongly encourage (not enforce) existing packages and new packages to not redefine their own normalize() function or any generic function already defined in BiocGenerics. If we agree on this, I'll add it to our post-BioC-2.12-release TODO list. Unless I hear a big "NO" from Wolfgang who I believe is the author/maintainer of one of the packages that define a normalize() function. FWIW, as a package reviewer myself, when I load the package I'm reviewing for the first time and see warnings that it's masking stuff defined in other packages, it's an orange light to me. Or a red light if the masked stuff is defined in packages that are at the bottom of the stack. So they'll need to have really really good reasons for doing this ;-) H.They didn't have any problem so far clashing with 9 other Bioc packages so why would they care if now it's 10 instead of 9?I'd suspect that they mostly do/did not care because either they were not using those packages, and if they did probably not within the same R session (so clashes were not so obvious).Furthermore, the good citizens that modify their package to use the normalize() in BiocGenerics will be in a situation worth than before because, from an end user point of view, their normalize() function (which is now a method attached to BiocGenerics::normalize) won't seem to work anymore, even if their package was loaded last, just because one bad citizen was loaded before (unless the end user calls BiocGenerics::normalize()). That would be unfair. So if I was the maintainer of one of those packages, I wouldn't see any benefit of making that move, all the contrary, unless everybody else also makes it.Eh... "the Bioconductor package maintainer" as a variant of the "Prisoner's dilemma". ;-)I kind of agree that namespaces can accommodate the current situation, even though, for the end user working interactively, the experience is not really pleasant, especially when they try to access the man page for normalize(). But if we are happy with that, then I don't really see the need to put normalize() in BiocGenerics.An other possible benefit of having a set on function in a "BiocGenerics" is to suggest that some method names are more commonly found (and make it easier for the end user by having less names to remember). For example, if there is a generic "plot" the right should have more chances to be called "plot" rather than "draw", "sketch", or "paint".Cheers, H. On 02/20/2013 11:52 AM, Robert Gentleman wrote:my 2c worth On Wed, Feb 20, 2013 at 10:45 AM, Hervé Pagès <hpa...@fhcrc.org> wrote:Hi, I agree with Laurent that we can't really play the semantic and concept police. It's the responsibility of package authors to decide whether it's appropriate or not to call "normalization" that particular transformation they're implementing. However I hope that we all agree on the following rule regarding the generics that make it into BiocGenerics: If foo() is a generic function defined in BiocGenerics, no BioC package should redefine the function (either as a generic or an ordinary function). It can only define methods for it, or move away and use a different name for this functionality.but really the point of namespaces is that you don't need to do that. And we really don't want to be the naming police. The sole advantage of BiocGenerics, I think, is that there is a common and standard location for a set of generic functions that get used in different packages. This allows package authors to add methods that specialize the behavior of a generic function. They have some confidence that the generic will always exist and hence can plan accordingly. It hopefully reduces dependencies between packages. I don't think it should define a set of reserved words, that seems counter productive. There are often good reasons why the same name is used for different concepts (normalize being one of them). And in some cases a single generic suffices, but in others it will not. Places where a single generic fall apart are when there are really different argument lists, and where inheritance (and hence things like NextMethod) are going to get messed up if the disparate methods are all linked to a single generic. Generics are really concepts - and the methods are realizations of those concepts. Of course, packages that define functions whose names clash with BiocGenerics will cause problems, and they would generally be best to avoid that, but really I don't think I would advocate any sort of prohibition.Does that sound reasonable? Otherwise that would kind of defeat the purpose of having the BiocGenerics package in the 1st place. To me, having 10 BioC packages defining a normalize() function is far from being ideal. I think having it defined in BiocGenerics would improve things a little bit. Also one potential positive side effect I see is that it would give an opportunity to the authors of those 10 packages to reconsider if they still want to ride the normalize() poney or not. Maybe some of them won't and they'll pick up another name. Not something we can really decide for them... H. On 02/20/2013 09:47 AM, Laurent Gautier wrote:On 2013-02-20 17:32, Schalkwyk, Leonard wrote:Is this not just an indication that normalize is now a poor choice of a function name?If the package authors called the functions "normalize", this means either: 1- at least some of the package authors have named a function performing an action that is inappropriately described as "normalize" 2- all functions "normalize" do perform an action that can be described with that verb Without more details, I'd vote for 2. (more below)LEo On 20 Feb 2013, at 16:14, Wolfgang Huber wrote:Hi is it clear that all these different functions (methods) share similar semantics and enough (conceptually) of their interface?Playing the semantic and concept police would come after defining things like ontologies of data processing; I am not sure this should be a priority. I'd see working out a minimal common signature that keeps everyone going with a minimal fuss come first.Wouldn't the implication be that preemptively every possible string of characters should already be defined as a generic function in BiocGenerics?No. Otherwise this would probably also mean that R's S4 system should in fact define all possible strings as generics, which by extension would also mean that generic functions do not need to be explicitly declared: since all possible generics would be declared, it is more practical to implicitly assume any given function has already generic declared. S4 has notions about implicit generic functions; a starting point is the man page for setGeneric().Best wishes Wolfgang Il giorno Feb 20, 2013, alle ore 11:04 AM, Laurent Gatto <lg...@cam.ac.uk> ha scritto:On 19 February 2013 22:44, Hervé Pagès <hpa...@fhcrc.org> wrote:Hi Laurent, and maintainers of packages with a normalize() function, On 02/15/2013 04:28 AM, Laurent Gatto wrote:A quick (and incomplete) manual search using http://search.bioconductor.jp/ suggest the following usage of normalize: As a function: xps::normalize codelink::normalize EBImage::normalize diffGeneAnalysis::normalize Defining a generic and methods: oligo::normalize flowCore::normalize MSnbase::normalize isobar::normalize and several normalize\.[*+] functions Would it be reasonable to add a normalize generic definition to BiocGenerics? The generic definitions in the above packages differ, however.Sounds good to me. However, since the various normalize() functions have different signatures, we need to agree on what the signature of the generic in BiocGenerics should be. Here is a summary of the situation: ** xps package: normalize() is an ordinary function with the following arg list: normalize(xps.data, filename=character(0), filedir=getwd(), tmpdir="", update=FALSE, select="all", method="mean", option="transcript:all", logbase="0", exonlevel="", refindex=0, refmethod="mean", params=list(0.02, 0), add.data=TRUE, verbose=TRUE) The package also defines normalize.constant(), normalize.lowess(), normalize.quantiles(), normalize.supsmu(), which are also ordinary functions (not S3 methods), and have similar but slightly different arg lists. ** codelink package: Ordinary function with the following args: normalize(object, method="quantiles", log.it=TRUE, preserve=FALSE, weights=NULL, verbose=FALSE) ** EBImage package: Ordinary function with the following args: normalize(x, separate=TRUE, ft=c(0, 1)) ** diffGeneAnalysis package: Ordinary function with the following args: normalize(rawdata, numSlides, ctrl, expm, ctrlbg=0.30, expmbg=0.30) ** deepSNV package: S4 generic with the following args: normalize(test, control, ...) ** isobar package: S4 generic with the following args: normalize(x, f=median, target="intensity", exclude.protein=NULL, use.protein=NULL, f.doapply=TRUE, log=TRUE, channels=NULL, na.rm=FALSE, per.file=TRUE, ...) ** affy package: S4 generic with the following args: normalize(object, ...) ** flowCore package: S4 generic with the following args: normalize(data, x, ...) ** MSnbase package: S4 generic with the following args: normalize(object, method, ...) ** oligo package: S4 generic with the following args: normalize(object, method=normalizationMethods(), copy=TRUE, subset=NULL, target='core', verbose=TRUE, ...) So it looks like the greatest common factor is normalize(x, ...). Not too surprising for a generic that covers such a wide range of related but slightly different concepts / algorithms. One technical difficulty though is that, even though almost all these functions seem to take an S4 object as their 1st arg, some of them don't: (a) For EBImage::normalize(), 'x' can be an ordinary array in addition to an Image object. (b) For diffGeneAnalysis::normalize(), 'rawdata' is an ordinary matrix. (c) For deepSNV::normalize(), 'test' can be an ordinary matrix in addition to a deepSNV object. (d) For oligo::normalize(), 'object' can be an ordinary matrix in addition to a FeatureSet object. So how can we disambiguate when the first arg is an ordinary matrix? IMO normalize() should fail in that case i.e. no package should define methods for ordinary arrays or matrices. Not ideal but better than the current situation where what is returned depends on which package was loaded last. I could put normalize(x, ...) in BiocGenerics if nobody objects, but that's all. I don't have time to fix the 10 packages that this change will affect. However, I'd rather wait the beginning of the Bioc 2.13 devel cycle (April) for such a change. For some packages like diffGeneAnalysis (which doesn't use S4 at all), that will probably require a significant amount of changes since it will need to pass the data to normalize in an S4 container instead of an ordinary matrix. Comments and suggestions are welcome.Fine by me. LaurentThanks, H.Best wishes, Laurent _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel-- Hervé Pagès Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA 98109-1024 E-mail: hpa...@fhcrc.org Phone: (206) 667-5791 Fax: (206) 667-1319_______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel_______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel_______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel_______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel-- Hervé Pagès Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA 98109-1024 E-mail: hpa...@fhcrc.org Phone: (206) 667-5791 Fax: (206) 667-1319 _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel-- Hervé Pagès Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA 98109-1024 E-mail: hpa...@fhcrc.org Phone: (206) 667-5791 Fax: (206) 667-1319 _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
-- Hervé Pagès Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA 98109-1024 E-mail: hpa...@fhcrc.org Phone: (206) 667-5791 Fax: (206) 667-1319 _______________________________________________ Bioc-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/bioc-devel
