Re: [discuss] SparkR CRAN feasibility check server problem

2018-11-10 Thread Felix Cheung
Itโ€™s a great point about min R version. From what I see, mostly because of fixes and packages support, most users of R are fairly up to date? So perhaps 3.4 as min version is reasonable esp. for Spark 3. Are we getting traction with CRAN sysadmin? It seems like this has been broken a few times.

Re: [discuss] SparkR CRAN feasibility check server problem

2018-11-10 Thread Liang-Chi Hsieh
Yeah, thanks Hyukjin Kwon for bringing this up for discussion. I don't know how higher versions of R are widely used across R community. If R version 3.1.x was not very commonly used, I think we can discuss to upgrade minimum R version in next Spark version. If we ended up with not upgrading, w

Re: [discuss] SparkR CRAN feasibility check server problem

2018-11-10 Thread Hyukjin Kwon
> Can upgrading R able to fix the issue. Is this perhaps not necessarily malform but some new format for new versions perhaps? That's my guess. I am not totally sure about it tho. > Anyway we should consider upgrading R version if that fixes the problem. Yea, we should. If we should, it should be

Re: Arrow optimization in conversion from R DataFrame to Spark DataFrame

2018-11-10 Thread Hyukjin Kwon
Thanks guys ! ๐Ÿ‘ 2018๋…„ 11์›” 10์ผ (ํ† ) ์˜ค์ „ 7:35, Bryan Cutler ๋‹˜์ด ์ž‘์„ฑ: > Great work Hyukjin! I'm not too familiar with R, but I'll take a look at > the PR. > > Bryan > > On Fri, Nov 9, 2018 at 9:19 AM Shivaram Venkataraman < > shiva...@eecs.berkeley.edu> wrote: > >> Thanks Hyukjin! Very cool results >>

Re: Spark Utf 8 encoding

2018-11-10 Thread Jรถrn Franke
Is the original file indeed utf-8? Especially Windows environments tend to mess up the files (E.g. Java on Windows does not use by default UTF-8). However, also the software that processed the data before could have modified it. > Am 10.11.2018 um 02:17 schrieb lsn24 : > > Hello, > > Per the d