+1 On Tue, Aug 13, 2024 at 10:50 PM L. C. Hsieh <vii...@gmail.com> wrote:
> +1 > > On Tue, Aug 13, 2024 at 2:54 AM Dongjoon Hyun <dongjoon.h...@gmail.com> > wrote: > > > > +1 > > > > Dongjoon > > > > On Mon, Aug 12, 2024 at 17:52 Holden Karau <holden.ka...@gmail.com> > wrote: > >> > >> +1 > >> > >> Are the sparklyr folks on this list? > >> > >> Twitter: https://twitter.com/holdenkarau > >> Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau > >> Pronouns: she/her > >> > >> > >> On Mon, Aug 12, 2024 at 5:22 PM Xiao Li <gatorsm...@gmail.com> wrote: > >>> > >>> +1 > >>> > >>> Hyukjin Kwon <gurwls...@apache.org> 于2024年8月12日周一 16:18写道： > >>>> > >>>> +1 > >>>> > >>>> On Tue, Aug 13, 2024 at 7:04 AM Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >>>>> > >>>>> And just for the record, the stats that I screenshotted in that > thread I linked to showed the following page views for each sub-section > under `docs/latest/api/`: > >>>>> > >>>>> - python: 758K > >>>>> - java: 66K > >>>>> - sql: 39K > >>>>> - scala: 35K > >>>>> - r: <1K > >>>>> > >>>>> I don’t recall over what time period those stats were collected for, > and there are certainly some factors of how the stats are gathered and how > the various language API docs are accessed that impact those numbers. So > it’s by no means a solid, objective measure. But I thought it was an > interesting signal nonetheless. > >>>>> > >>>>> > >>>>> On Aug 12, 2024, at 5:50 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >>>>> > >>>>> Not an R user myself, but +1. > >>>>> > >>>>> I first wondered about the future of SparkR after noticing how low > the visit stats were for the R API docs as compared to Python and Scala. (I > can’t seem to find those visit stats for the API docs anymore.) > >>>>> > >>>>> > >>>>> On Aug 12, 2024, at 11:47 AM, Shivaram Venkataraman < > shivaram.venkatara...@gmail.com> wrote: > >>>>> > >>>>> Hi > >>>>> > >>>>> About ten years ago, I created the original SparkR package as part > of my research at UC Berkeley [SPARK-5654]. After my PhD I started as a > professor at UW-Madison and my contributions to SparkR have been in the > background given my availability. I continue to be involved in the > community and teach a popular course at UW-Madison which uses Apache Spark > for programming assignments. > >>>>> > >>>>> As the original contributor and author of a research paper on > SparkR, I also continue to get private emails from users. A common question > I get is whether one should use SparkR in Apache Spark or the sparklyr > package (built on top of Apache Spark). You can also see this in > StackOverflow questions and other blog posts online: > https://www.google.com/search?q=sparkr+vs+sparklyr . While, I have > encouraged users to choose the SparkR package as it is maintained by the > Apache project, the more I looked into sparklyr, the more I was convinced > that it is a better choice for R users that want to leverage the power of > Spark: > >>>>> > >>>>> (1) sparklyr is developed by a community of developers who > understand the R programming language deeply, and as a result is more > idiomatic. In hindsight, sparklyr’s more idiomatic approach would have been > a better choice than the Scala-like API we have in SparkR. > >>>>> > >>>>> (2) Contributions to SparkR have decreased slowly. Over the last two > years, there have been 65 commits on the Spark R codebase (compared to > ~2200 on the Spark Python code base). In contrast Sparklyr has over 300 > commits in the same period.. > >>>>> > >>>>> (3) Previously, using and deploying sparklyr had been cumbersome as > it needed careful alignment of versions between Apache Spark and sparklyr. > However, the sparklyr community has implemented a new Spark Connect based > architecture which eliminates this issue. > >>>>> > >>>>> (4) The sparklyr community has maintained their package on CRAN – it > takes some effort to do this as the CRAN release process requires passing a > number of tests. While SparkR was on CRAN initially, we could not maintain > that given our release process and cadence. This makes sparklyr much more > accessible to the R community. > >>>>> > >>>>> So it is with a bittersweet feeling that I’m writing this email to > propose that we deprecate SparkR, and recommend sparklyr as the R language > binding for Spark. This will reduce complexity of our own codebase, and > more importantly reduce confusion for users. As the sparklyr package is > distributed using the same permissive license as Apache Spark, there should > be no downside for existing SparkR users in adopting it. > >>>>> > >>>>> My proposal is to mark SparkR as deprecated in the upcoming Spark 4 > release, and remove it from Apache Spark with the following major release, > Spark 5. > >>>>> > >>>>> I’m looking forward to hearing your thoughts and feedback on this > proposal and I’m happy to create the SPIP ticket for a vote on this > proposal using this email thread as the justification. > >>>>> > >>>>> Thanks > >>>>> Shivaram > >>>>> > >>>>> > >>>>> > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
