Re: Resolving all JIRAs affecting EOL releases

I am going to resolve those JIRAs now. 2019년 9월 9일 (월) 오전 9:46, Hyukjin Kwon 님이 작성: > Yup, no worries. I roughly set the one week delay considering the official > release date :D > > On Mon, 9 Sep 2019, 09:45 Dongjoon Hyun, wrote: > >> Thank you, Hyukjin. >> >> +1 for closing according to 2.3.x

Re: [VOTE][SPARK-28885] Follow ANSI store assignment rules in table insertion by default

+1 I think this is the most reasonable default behavior among the three. On Mon, Oct 7, 2019 at 6:06 PM Alessandro Solimando < alessandro.solima...@gmail.com> wrote: > +1 (non-binding) > > I have been following this standardization effort and I think it is sound > and it provides the needed flex

Re: Spark 3.0 preview release feature list and major changes

Cogroup Pandas UDF missing: SPARK-27463 Support Dataframe Cogroup via Pandas UDFs Vectorized R execution: SPARK-26759 Arrow optimization in SparkR's interoperability 2019년 10월 8일 (화) 오전 7:50,

Re: Spark 3.0 preview release feature list and major changes

Thanks for bringing the nice summary of Spark 3.0 improvements! I'd like to add some items from structured streaming side, SPARK-28199 Move Trigger implementations to Triggers.scala and avoid exposing these to the end users (removal of deprecate

Re: [SS] How to create a streaming DataFrame (for a custom Source in Spark 2.4.4 / MicroBatch / DSv1)?

Would you mind if I ask the condition of being public API? Source/Sink traits are not marked as @DeveloperApi but they're defined as public, and located to sql-core so even not semantically private (for catalyst), easy to give a signal they're public APIs. Also, if I'm not missing here, creating s

Spark 3.0 preview release feature list and major changes

Hi all, I went over all the finished JIRA tickets targeted to Spark 3.0.0, here I'm listing all the notable features and major changes that are ready to test/deliver, please don't hesitate to add more to the list: SPARK-11215 Multiple columns su

Re: [build system] maven master branch builds timing out en masse...

Moving the conversation here -- yes, why on earth are they taking this long all of the sudden? we'll have to look again when they come back online. The last successful build took 6 hours, of which 4:45 were the unit tests themselves. It's mostly SQL tests; SQLQuerySuite is approaching an hour. ht

Re: [build system] maven master branch builds timing out en masse...

just chatted w/sean privately and i'm going to up the test timeouts to 480mins (8 hours). i still don't like this but at least it should hopefully get things green again. On Mon, Oct 7, 2019 at 11:31 AM Shane Knapp wrote: > > https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-had

[build system] maven master branch builds timing out en masse...

https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/ https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7-ubuntu-testing/ https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-3.2/ https://amplab.cs.berkeley.edu/jenkins/job/spar

Re: [SS] How to create a streaming DataFrame (for a custom Source in Spark 2.4.4 / MicroBatch / DSv1)?

AFAIK there is no public streaming data source API before DS v2. The Source and Sink API is private and is only for builtin streaming sources. Advanced users can still implement custom stream sources with private Spark APIs (you can put your classes under the org.apache.spark.sql package to access

Re: [VOTE][SPARK-28885] Follow ANSI store assignment rules in table insertion by default

+1 (non-binding) I have been following this standardization effort and I think it is sound and it provides the needed flexibility via the option. Best regards, Alessandro On Mon, 7 Oct 2019 at 10:24, Gengliang Wang wrote: > Hi everyone, > > I'd like to call for a new vote on SPARK-28885 >

[VOTE][SPARK-28885] Follow ANSI store assignment rules in table insertion by default

Hi everyone, I'd like to call for a new vote on SPARK-28885 "Follow ANSI store assignment rules in table insertion by default" after revising the ANSI store assignment policy(SPARK-29326 ). When