hi Javier, Thank you for writing back.
On Mon, Mar 25, 2019 at 12:41 PM Javier Luraschi <jav...@rstudio.com> wrote: > > Hi Wes, sorry for the delay I haven't been monitoring this DL proactively. Yes, I highly recommend setting up some e-mail filters so anything with "[R]" in the subject title lands in your inbox. You can also separate "[jira]" messages with a separate filter; there isn't very much list traffic if you split off the new issue notifications. > > Please notice that I'm not the expert in this topic, so I'll share as much > information > as I can but others with more expertise should feel free comment as well. > Please > also note that some of the restrictions we have are common practices in > R packages that are out of our control, at least without significant > investment. > > I'll document what I know in this email, but please let me know if there is > a wiki > or a better place to move this documentation into. > Yes, let's definitely stash all of the build and packaging information on our wiki at https://cwiki.apache.org/confluence/display/ARROW If you let me know your ASF Confluence username I will give you edit permissions > ## Background > > CRAN, The Comprehensive R Archive Network, is the most popular (primary) > package repo for the R community. You can think of CRAN as Homebrew or > pip.org. CRAN encourages cross-platform packages to be submitted and to > ease compilation and testing, provide support to precompile binaries for OS > X > and Windows. We will focus now on Windows specifics from now on. > > CRAN and R rely on a set of tools based on Mingw to easily compile packages > in Windows, this tools set is known as RTools. Originally, Prof. Brian > Ripley and > Duncan Murdoch put this toolset together; however, Jeroen Ooms is it current > maintainer. RTools is based on Mingw but from past experience, not > completely > interchangeable with the standard Mingw distribution. I'm afraid I don't > have the > details but this is mostly related to specific packages, versions and > compilers > included in Rtools. It's possible to match a Mingw environment with RTools > but > this is, in general, not a straightforward task. It would be good to have some links (on a wiki page) to any additional information about this. > > A few months ago, I naively tried to accomplish this work myself. As in, get > RTools to compile Apache Arrow, how hard can it be? It's hard to explain > all the caveats in a single mail, but if you are interested, you can read > my own exploration of possible solutions to this problem in this gist > writeup [1]. > > The outcome of this investigation, at least for me and my limited knowledge > was > to not try to do this on my own by reinventing the wheel; otherwise, this > would > have taken months of my own time. The solution was then to find out how > other > R packages have solve this problem in the past. > > Given the specifics of the RTools toolchain, for complex projects with > significant > number of components and dependencies, the best (and maybe only!) way > to get R packages into CRAN in Windows is to precompile the binaries outside > of the CRAN build process. The repo of precompiled packages is called > rwinlibs [2] and has 75 packages and growing. When compiling in CRAN, rather > than building the library, it simply gets downloaded from the rwinlibs repo. > > How then are the rwinlibs libraries build then? All the packages are built > through > an automated build system available under theb rtools-packages [3] repo > where > an appveyor script detects changes and builds the appropriate libraries. > This repo > runs with the latest RTools toolchain. To support previous versions of > R/RTools a > the rtools-backports [4] repo provides backward compatibility in an > automated way. > > So now we can get back at discussing how we want to make this work in the > arrow project. One way, which this PR encourages is to say "Lets not worry > about > what the R/CRAN publishing process is, they have their own processes and > tools > to build binaries for Windows. This is similar to brew formulae, the > formula that > builds arrow for OS X using homebrew is in a different repo [5]". When you say "this PR" you mean https://github.com/apache/arrow/pull/4011 or https://github.com/apache/arrow/pull/3932 The challenge I see is that the development procedure is being commingled with packaging issues. I would like to see a write-up to provide instructions for an Arrow developer to create a build of Arrow on the master branch using mingw/Rtools for the purposes of development. If we don't have this written down, this is putting us in a potentially very bad situation where developers cannot debug issues. I think it's fine if all of the other C++ dependencies are snapshotted in rwinlibs > > While splitting the release processes into multiple repos has some > advantages, > it certainly has some caveats. For instance, when publishing a new release > of > arrow in Homebrew, one needs to manually go an update the Hombrew formulae. > > That said, I would hope that the Homebrew release process is documented in > the > Arrow project in the same way that we should document the R release process > in > the Arrow project. Hopefully this mail helps build a first iteration on > this. > > ## Releasing > > These instructions are a bit more pragmatic as to what needs to be done to > release > the R package in CRAN: > > (1) Send PR to the rtools-packages [3], increment the version, the repo > already > downloads the binaries from the Arrow GitHub project. Ensure that the > appveyor > build succeeds. If the build or tests fails, send the appropriate PR > to the official > Arrow repo. How would you suggest testing release candidates or otherwise doing some form of continuous integration / integration testing to ensure we haven't broken this step? > (2) Send PR to the rtools-backports [4], similar to (1) but different repo. Seems like this should be turned into a Crossbow task in this project (see https://github.com/apache/arrow/tree/master/dev/tasks) so that it can be maintained by the Arrow community. This is how we are handling package automation for Linux packages, Python wheels, Gandiva JARs, etc. This also may enable the integration testing I described above to take place (though having an Appveyor build would be superior) > (3) Copy the output produced by (1) and (2) as a PR to the rwinlib/arrow > [6] repo. > (4) Before merging (3) validate that CRAN can build and test using the new > library > using the winbuilder service [7]. This service is maintained to CRAN > and allows > you to pre-check a package builds properly under a CRAN-like build > machine > for Windows. If there a way to simulate this environment locally? > (5) Submit package to CRAN, make sure their practices and processes are > followed [8]. > > While I did my best to document the steps, there is certainly more details > that can be > added over time. Regardless, feel free to reach out to me with questions, > support > requests and why not and I'll try my best to address them. > OK, let's definitely copy this information into a page on the wiki so that these steps can be maintained as time goes on. The goal would be to have sufficient detail to increase the bus factor involved with post-release tasks. Given how manual the process is right now it seems like there's a solid chance that something will be broken after the 0.13 release is out. Speaking of which, can you reply on the "Timeline for 0.13 release" thread about any PRs that need to get merged? Please set the "Fix Version" so they show up in the list of 0.13 issues Thanks, Wes > Best, Javier > > [1]: https://gist.github.com/javierluraschi/2ade2204364a7c20e9c3d95504d12ce5 > [2]: https://github.com/rwinlib/ > [3]: https://github.com/r-windows/rtools-packages > [4]: https://github.com/r-windows/rtools-backports > [5]: > https://github.com/Homebrew/homebrew-core/blob/master/Formula/apache-arrow.rb > [6]: https://github.com/rwinlib/arrow > [7]: https://win-builder.r-project.org/ > [8]: https://cran.r-project.org/submit.html > > > > > On Sat, Mar 16, 2019 at 1:10 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > hi folks, > > > > I have noticed there is work under way to prepare Apache Arrow for > > submission to the CRAN package manager for R users. I'm slightly > > concerned about the lack of information and documentation in the > > project regarding what is involved with this effort. This patch in > > particular raised some eyebrows > > > > https://github.com/apache/arrow/pull/3932 > > > > This introduces a dependency into the project on pre-built static > > libraries based on processes that aren't documented in the project. I > > see this repository containing these static libraries for the R > > Windows toolchain, but if I needed to produce them myself I would not > > know what to do > > > > https://github.com/rwinlib/arrow > > > > Additionally, in general, if I wanted to build and test Arrow and R > > from source on Windows, I also would not know what to do. > > > > In the Python world, this would be akin to depending on e.g. > > conda-forge packages for Windows development, but not having any > > information in the repository about to build Arrow C++ and Python from > > source on Windows. > > > > So I would like to see some transparency / documentation around the > > scripts and processes involved with this so that we don't end up with > > a "bus factor" problem where Arrow PMC members are unable to undertake > > basic maintenance and release management activities. Currently the > > work that is going on seems opaque to me and as such feels contrary to > > the Apache Way. > > > > I understand that there is some urgency to make the Arrow libraries > > available to R users, but I want to make sure we are working in a > > sustainable manner to grow a community of developers who are able to > > do work on each part of the project. > > > > Thanks, > > Wes > >
