Re: Handling user-facing metadata issues on file stream source & sink

2020-06-14 Thread Jungtaek Lim
Bump again - hope to get some traction because these issues are either long-standing problems or noticeable improvements (each PR has numbers/UI graph to show the improvement). Fixed long-standing problems: * [SPARK-17604][SS] FileStreamSource: provide a new option to have retention on input file

Re: [vote] Apache Spark 3.0 RC3

2020-06-14 Thread Dongjoon Hyun
Hi, Reynold. Is there any progress on 3.0.0 release since the vote was finalized 5 days ago? Apparently, tag `v3.0.0` is not created yet, the binary and docs are still sitting on the voting location, Maven Central doesn't have it, and PySpark/SparkR uploading is not started yet. https://dist