Looks it's happening again. Liang-Chi, do you mind if I ask it again? FYI, R 3.4 is officially deprecated as of https://github.com/apache/spark/pull/23012 We could upgrade R version to 3.4.x in Jenkins, which deals with the malformed(?) responses after 3.0 release. Then, we could get rid of this problem..!
2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: > I made a PR to officially drop R prior to version 3.4 ( > https://github.com/apache/spark/pull/23012). > The tests will probably fail for now since it produces warnings for using > R 3.1.x. > > 2018년 11월 11일 (일) 오전 3:00, Felix Cheung <felixcheun...@hotmail.com>님이 작성: > >> It’s a great point about min R version. From what I see, mostly because >> of fixes and packages support, most users of R are fairly up to date? So >> perhaps 3.4 as min version is reasonable esp. for Spark 3. >> >> Are we getting traction with CRAN sysadmin? It seems like this has been >> broken a few times. >> >> >> ------------------------------ >> *From:* Liang-Chi Hsieh <vii...@gmail.com> >> *Sent:* Saturday, November 10, 2018 2:32 AM >> *To:* dev@spark.apache.org >> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem >> >> >> Yeah, thanks Hyukjin Kwon for bringing this up for discussion. >> >> I don't know how higher versions of R are widely used across R community. >> If >> R version 3.1.x was not very commonly used, I think we can discuss to >> upgrade minimum R version in next Spark version. >> >> If we ended up with not upgrading, we can discuss with CRAN sysadmin to >> fix >> it by the service side automatically that prevents malformed R packages >> info. So we don't need to fix it manually every time. >> >> >> >> Hyukjin Kwon wrote >> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily >> > malform but some new format for new versions perhaps? >> > That's my guess. I am not totally sure about it tho. >> > >> >> Anyway we should consider upgrading R version if that fixes the >> problem. >> > Yea, we should. If we should, it should be more them R 3.4. Maybe it's >> > good >> > time to start to talk about minimum R version. 3.1.x is too old. It's >> > released 4.5 years ago. >> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0, >> > deprecating lower versions, bumping up R to 3.4 might be reasonable >> > option. >> > >> > Adding Shane as well. >> > >> > If we ended up with not upgrading it, I will forward this email to CRAN >> > sysadmin to discuss further anyway. >> > >> > >> > >> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < >> >> > felixcheung@ >> >> > >님이 작성: >> > >> >> Thanks for being this up and much appreciate with keeping on top of >> this >> >> at all times. >> >> >> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily >> >> malform but some new format for new versions perhaps? Anyway we should >> >> consider upgrading R version if that fixes the problem. >> >> >> >> As an option we could also disable the repo check in Jenkins but I can >> >> see >> >> that could also be problematic. >> >> >> >> >> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon < >> >> > gurwls223@ >> >> > > wrote: >> >> >> >>> Hi all, >> >>> >> >>> I want to raise the CRAN failure issue because it started to block >> Spark >> >>> PRs time to time. Since the number >> >>> of PRs grows hugely in Spark community, this is critical to not block >> >>> other PRs. >> >>> >> >>> There has been a problem at CRAN (See >> >>> https://github.com/apache/spark/pull/20005 for analysis). >> >>> To cut it short, the root cause is malformed package info from >> >>> https://cran.r-project.org/src/contrib/PACKAGES >> >>> from server side, and this had to be fixed by requesting it to CRAN >> >>> sysaadmin's help. >> >>> >> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am >> >>> pretty sure it's the same issue >> >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved >> 2 >> >>> times >> >>> https://issues.apache.org/jira/browse/SPARK-22812 >> >>> >> >>> This happened 5 times for roughly about 10 months, causing blocking >> >>> almost all PRs in Apache Spark. >> >>> Historically, it blocked whole PRs for few days once, and whole Spark >> >>> community had to stop working. >> >>> >> >>> I assume this has been not a super big big issue so far for other >> >>> projects or other people because apparently >> >>> higher version of R has some logics to handle this malformed documents >> >>> (at least I verified R 3.4.0 works fine). >> >>> >> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated >> >>> from what I have seen before), >> >>> which is unable to parse the malformed server's response. >> >>> >> >>> So, I want to talk about how we are going to handle this. Possible >> >>> solutions are: >> >>> >> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent >> this >> >>> issue >> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to >> test >> >>> low R versions) >> >>> 3. ... >> >>> >> >>> If if we fine, I would like to suggest to forward this email to CRAN >> >>> sysadmin to discuss further about this. >> >>> >> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this >> few >> >>> times before. >> >>> >> >>> Thanks all. >> >>> >> >>> >> >>> >> >>> >> >> >> >> >> >> -- >> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
