driver fail-over in Spark streaming 1.2.0

appreciated :) regards, lin

[Spark SQL] quick enhancement for SPARK-28098

Hello spark-devs, we hit a similar case with SPARK-28098 when we tried to read a parquet format table which is generated by hive union operation, and I made a quick fix for it. I'm not sure we should reuse the same configuration with hive or add a new one. And this is my first time to contribute

Re: Spark Union performance issue

Hi Spark devs, I'm experiencing a Union performance degradation as well. Since this email thread is very related, posting it here to see if anyone has any insights. *Background*: After upgrading a Spark job from Spark 2.4 to Spark 3.1 without any code change, we saw *big performance degradation*

Re: Welcoming some new committers and PMC members

Congratulations and thanks for your great job! On Tue, Sep 10, 2019 at 2:13 PM Xingbo Jiang wrote: > Congratulations！ > > Wenchen Fan 于2019年9月9日 周一下午7:49写道： > >> Congratulations! >> >> On Tue, Sep 10, 2019 at 10:19 AM Yuanjian Li >> wrote: >> >>> Congratulations! >>> >>> sujith chacko 于2019年9月

Re: [VOTE] Spark 2.1.2 (RC4)

+1 (non-binding) Cheers, Liwei On Wed, Oct 4, 2017 at 4:03 PM, Nick Pentreath wrote: > Ah right! Was using a new cloud instance and didn't realize I was logged > in as root! thanks > > On Tue, 3 Oct 2017 at 21:13 Marcelo Vanzin wrote: > >> Maybe you're running as root (or the admin account on

Nightly builds for master branch failed

https://amplab.cs.berkeley.edu/jenkins/job/spark-master-maven-snapshots/ Nightly builds for master branch failed due to: [error] error: error reading /home/jenkins/.m2/repository/com/fasterxml/jackson/core/jackson-databind/ 2.6.7.1/jackson-databind-2.6.7.1.jar; zip file is empty Can we get it fi

Re: Welcoming Tejas Patil as a Spark committer

Congratulations! Cheers, Liwei On Wed, Oct 4, 2017 at 2:27 PM, Yuval Itzchakov wrote: > Congratulations and Good luck! :) > > > > -- > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > > - > To unsubscribe

Re: [VOTE][SPIP] SPARK-22026 data source v2 write path

+1 ! Cheers, Liwei On Thu, Oct 12, 2017 at 7:11 PM, vaquar khan wrote: > +1 > > Regards, > Vaquar khan > > On Oct 11, 2017 10:14 PM, "Weichen Xu" wrote: > > +1 > > On Thu, Oct 12, 2017 at 10:36 AM, Xiao Li wrote: > >> +1 >> >> Xiao >> >> On Mon, 9 Oct 2017 at 7:31 PM Reynold Xin wrote: >> >>

Re: welcoming Xiao Li as a committer

Congratulations, Xiao! Cheers, Liwei On Thu, Oct 6, 2016 at 5:38 AM, DB Tsai wrote: > Congrats, Xiao! > > Sincerely, > > DB Tsai > -- > Web: https://www.dbtsai.com > PGP Key ID: 0x9DCC1DBD7FC7BBB2 > > > On Wed, Oct 5, 2016 at 2:36 PM, Fred

Evolutionary algorithm (EA) in Spark

some guidance on how to go about it. Regards, Chris Lin -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Evolutionary-algorithm-EA-in-Spark-tp19715.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Evolutionary algorithm (EA) in Spark

some guidance on how to go about it. Regards, Chris Lin -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Evolutionary-algorithm-EA-in-Spark-tp19716.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Re: [VOTE] Release Apache Spark 1.6.3 (RC2)

+1 (non-binding) Cheers, Liwei On Fri, Nov 4, 2016 at 10:03 AM, Jeff Zhang wrote: > +1 > > Dongjoon Hyun 于2016年11月4日周五 上午9:44写道： > >> +1 (non-binding) >> >> It's built and tested on CentOS 6.8 / OpenJDK 1.8.0_111, too. >> >> Cheers, >> Dongjoon. >> >> On 2016-11-03 14:30 (-0700), Davies Liu wr

Re: [VOTE] Release Apache Spark 2.0.2 (RC3)

+1 (non-binding) Cheers, Liwei On Tue, Nov 8, 2016 at 9:50 PM, Ricardo Almeida < ricardo.alme...@actnowib.com> wrote: > +1 (non-binding) > > over Ubuntu 16.10, Java 8 (OpenJDK 1.8.0_111) built with Hadoop 2.7.3, > YARN, Hive > > > On 8 November 2016 at 12:38, Herman van Hövell tot Westerflier <

Re: SPARK-18689: A proposal for priority based app scheduling utilizing linux cgroups.

Sorry but I don't get the scope of the problem from your description. Seems it's an improvement for spark standalone scheduler (i.e. not for yarn or mesos)? On Sat, Dec 3, 2016 at 4:27 AM, Hegner, Travis wrote: > Hello, > > > I've just created a JIRA to open up discussion of a new feature that I

Re: [VOTE] Apache Spark 2.1.0 (RC5)

+1 Cheers, Liwei On Sat, Dec 17, 2016 at 10:29 AM, Yuming Wang wrote: > I hope https://github.com/apache/spark/pull/16252 can be fixed until > release 2.1.0. It's a fix for broadcast cannot fit in memory. > > On Sat, Dec 17, 2016 at 10:23 AM, Joseph Bradley > wrote: > >> +1 >> >> On Fri, Dec 1

Re: What is mainly different from a UDT and a spark internal type that ExpressionEncoder recognized?

Disclaimer: I'm not a spark guru, and what's written below are some notes I took when reading spark source code, so I could be wrong, in which case I'd appreciate a lot if someone could correct me. > > Let me rephrase this. How does the SparkSQL engine call the codegen APIs > to > do the job of p

A question about creating persistent table when in-memory catalog is used

no use. But I wonder if there are other good reasons for the current logic. If not, I would propose to raise an error when creating the table in the first place. Thanks! Regards, Shuai Lin (@lins05)

Re: A question about creating persistent table when in-memory catalog is used

difference is whether the metadata is >>> persistently stored or not. >>> >>> Thanks, >>> >>> Xiao Li >>> >>> 2017-01-22 11:14 GMT-08:00 Reynold Xin : >>> >>> I think this is something we are going to ch

Re: A question about creating persistent table when in-memory catalog is used

e fetch the latest master. You are unable > to create such a hive serde table without Hive support. > > Thanks, > > Xiao Li > > > 2017-01-23 0:01 GMT-08:00 Shuai Lin : > >> Cool, thanks for the info. >> >> I think this is something we are going to change t

Re: Apache Spark Contribution

> > The goal of the project is to develop an algorithm that automatically > scales the cluster up and down based on the volume of data processed by the > application. By "scale the cluster up and down" do you mean: 1) adding/removing spark executors based on the load? How is that from the dynami

Nightly builds for master branch have been failing

Nightly builds for master branch have been failing: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-maven-snapshots/buildTimeTrend It'd be great if we can get this fixed. Thanks. Cheers, Liwei

Question on how to get appended data from structured streaming

Hello, I am new to Spark. It would be appreciated if anyone could help me understand how to get appended data from structured streaming. According to the document , data stream could be treated as new r

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry! Cheers, Liwei On Tue, Aug 29, 2017 at 10:15 AM, 蒋星博 wrote: > congs！ > > Takeshi Yamamuro 于2017年8月28日 周一下午7:11写道： > >> Congrats! >> >> On Tue, Aug 29, 2017 at 11:04 AM, zhichao wrote: >> >>> Congratulations, Jerry! >>> >>> On Tue, Aug 29, 2017 at 9:57 AM, Weiqing Yang >

Re:Re: Welcoming Saisai (Jerry) Shao as a committer

congrats, 在2017年08月29 10时03分, "Liang-Chi Hsieh"写道: Congratulations, Jerry! Weiqing Yang wrote > Congratulations, Jerry! > > On Mon, Aug 28, 2017 at 6:44 PM, Yanbo Liang < > ybliang8@ > > wrote: > >> Congratulations, Jerry. >> >> On Tue, Aug 29, 2017 at 9:42 AM, John Deng < > mailthis@

Interested in Contributing to Spark as GSoC 2016

Hi All, I am Tao Lin, a senior Computer Science student highly interested in Data Science (Distributed Computing, Machine Learning, Visualization, etc.). I'd like to join Google Summer of Code 2016 and contribute to Spark this year. When I was viewing the past GSoC projects, I was impress

Re: [discuss] ending support for Java 7 in Spark 2.0

Arguments are really convincing; new Dataset API as well as performance improvements is exiting, so I'm personally +1 on moving onto Java8. However, I'm afraid Tencent is one of "the organizations stuck with Java7" -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, and wou

Re: Welcoming Yanbo Liang as a committer

Congratulations Yanbo! On Mon, Jun 6, 2016 at 7:07 AM, Bryan Cutler wrote: > Congratulations Yanbo! > On Jun 5, 2016 4:03 AM, "Kousuke Saruta" > wrote: > >> Congratulations Yanbo! >> >> >> - Kousuke >> >> On 2016/06/04 11:48, Matei Zaharia wrote: >> >>> Hi all, >>> >>> The PMC recently voted to