Dataset.localCheckpoint?

2018-01-22 Thread Tomasz Gawęda
Hi, Today I saw that there is no localCheckpoint() function in Dataset. Is there any reason for that? Checkpointing can truncate logical plans, but in some cases it's quite expensive to save whole Dataset on disk. Is there any workaround for this? Pozdrawiam / Best regards, Tomek Gawęda

[VOTE] Spark 2.3.0 (RC2)

2018-01-22 Thread Sameer Agarwal
Please vote on releasing the following candidate as Apache Spark version 2.3.0. The vote is open until Friday January 26, 2018 at 8:00:00 am UTC and passes if a majority of at least 3 PMC +1 votes are cast. [ ] +1 Release this package as Apache Spark 2.3.0 [ ] -1 Do not release this package beca

Re: [VOTE] Spark 2.3.0 (RC2)

2018-01-22 Thread Marcelo Vanzin
+0 Signatures check out. Code compiles, although I see the errors in [1] when untarring the source archive; perhaps we should add "use GNU tar" to the RM checklist? Also ran our internal tests and they seem happy. My concern is the list of open bugs targeted at 2.3.0 (ignoring the documentation

Re: [VOTE] Spark 2.3.0 (RC2)

2018-01-22 Thread Wenchen Fan
+1 All the blocking issues are resolved(AFAIK), and important data source v2 features have been merged. On Tue, Jan 23, 2018 at 9:09 AM, Marcelo Vanzin wrote: > +0 > > Signatures check out. Code compiles, although I see the errors in [1] > when untarring the source archive; perhaps we should ad