It’s a great point about min R version. From what I see, mostly because of fixes and packages support, most users of R are fairly up to date? So perhaps 3.4 as min version is reasonable esp. for Spark 3.
Are we getting traction with CRAN sysadmin? It seems like this has been broken a few times. ________________________________ From: Liang-Chi Hsieh <vii...@gmail.com> Sent: Saturday, November 10, 2018 2:32 AM To: dev@spark.apache.org Subject: Re: [discuss] SparkR CRAN feasibility check server problem Yeah, thanks Hyukjin Kwon for bringing this up for discussion. I don't know how higher versions of R are widely used across R community. If R version 3.1.x was not very commonly used, I think we can discuss to upgrade minimum R version in next Spark version. If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix it by the service side automatically that prevents malformed R packages info. So we don't need to fix it manually every time. Hyukjin Kwon wrote >> Can upgrading R able to fix the issue. Is this perhaps not necessarily > malform but some new format for new versions perhaps? > That's my guess. I am not totally sure about it tho. > >> Anyway we should consider upgrading R version if that fixes the problem. > Yea, we should. If we should, it should be more them R 3.4. Maybe it's > good > time to start to talk about minimum R version. 3.1.x is too old. It's > released 4.5 years ago. > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0, > deprecating lower versions, bumping up R to 3.4 might be reasonable > option. > > Adding Shane as well. > > If we ended up with not upgrading it, I will forward this email to CRAN > sysadmin to discuss further anyway. > > > > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < > felixcheung@ > >님이 작성: > >> Thanks for being this up and much appreciate with keeping on top of this >> at all times. >> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily >> malform but some new format for new versions perhaps? Anyway we should >> consider upgrading R version if that fixes the problem. >> >> As an option we could also disable the repo check in Jenkins but I can >> see >> that could also be problematic. >> >> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon < > gurwls223@ > > wrote: >> >>> Hi all, >>> >>> I want to raise the CRAN failure issue because it started to block Spark >>> PRs time to time. Since the number >>> of PRs grows hugely in Spark community, this is critical to not block >>> other PRs. >>> >>> There has been a problem at CRAN (See >>> https://github.com/apache/spark/pull/20005 for analysis). >>> To cut it short, the root cause is malformed package info from >>> https://cran.r-project.org/src/contrib/PACKAGES >>> from server side, and this had to be fixed by requesting it to CRAN >>> sysaadmin's help. >>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am >>> pretty sure it's the same issue >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 >>> times >>> https://issues.apache.org/jira/browse/SPARK-22812 >>> >>> This happened 5 times for roughly about 10 months, causing blocking >>> almost all PRs in Apache Spark. >>> Historically, it blocked whole PRs for few days once, and whole Spark >>> community had to stop working. >>> >>> I assume this has been not a super big big issue so far for other >>> projects or other people because apparently >>> higher version of R has some logics to handle this malformed documents >>> (at least I verified R 3.4.0 works fine). >>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated >>> from what I have seen before), >>> which is unable to parse the malformed server's response. >>> >>> So, I want to talk about how we are going to handle this. Possible >>> solutions are: >>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent this >>> issue >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test >>> low R versions) >>> 3. ... >>> >>> If if we fine, I would like to suggest to forward this email to CRAN >>> sysadmin to discuss further about this. >>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this few >>> times before. >>> >>> Thanks all. >>> >>> >>> >>> -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
