Re: scala.collection.immutable.Nil$ cannot be cast to org.apache.spark.util.BoundedPriorityQueue

Hi Fabrizio, Can someone explain me why do I get SparkConf not serializable error ? > First, SparkConf is not serializable and that's what the exception tells you. Why you stuck in this situation ? Well, that's must because some of your classes must require a SparkConf class. In your case, that's

Re: spark config params conventions

+1. I agree to keep the old ones only for backward compatibility purpose. On Wed, Mar 12, 2014 at 12:38 PM, Evan Chan wrote: > +1. > > Not just for Typesafe Config, but if we want to consider hierarchical > configs like JSON rather than flat key mappings, it is necessary. It > is also clearer.

Re: Announcing the official Spark Job Server repo

+1 On Tue, Mar 18, 2014 at 3:39 PM, Henry Saputra wrote: > W00t! > > Thanks for releasing this, Evan. > > - Henry > > On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan wrote: > > Dear Spark developers, > > > > Ooyala is happy to announce that we have pushed our official, Spark > > 0.9.0 / Scala 2.10-c

Re: Error reading HDFS file using spark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1

@qingyang, spark 0.9.0 works for me perfectly when accessing (read/write) data on hdfs. BTW, if you look at pom.xml, you have to choose yarn profile to compile spark, so that it won't include protobuf 2.4.1 in your final jars. Here is the command line we use to compile spark with hadoop 2.2: mvn -

Re: Error executing sql using shark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1

.0]# find . -name "proto*.jar" > > > ./lib_managed/jars/org.spark-project.protobuf/protobuf-java/protobuf-java-2.4.1-shaded.jar > > ./lib_managed/bundles/com.google.protobuf/protobuf-java/protobuf-java-2.5.0.jar > > > > > 2014-03-27 2:26 GMT+08:00 yao : > > > @qingya

Re: Changes to sbt build have been merged

Cool~ On Thu, Jul 10, 2014 at 1:29 PM, Sandy Ryza wrote: > Woot! > > > On Thu, Jul 10, 2014 at 11:15 AM, Patrick Wendell > wrote: > > > Just a heads up, we merged Prashant's work on having the sbt build read > all > > dependencies from Maven. Please report any issues you find on the dev > list

Re: Hadoop's Configuration object isn't threadsafe

Good catch Andrew. In addition to your proposed solution, is that possible to fix Configuration class and make it thread-safe ? I think the fix should be trivial, just use a ConcurrentHashMap, but I am not sure if we can push this change upstream (will hadoop guys accept this change ? for them, it

spark 0.9.0 with hadoop 2.4 ?

Hi Everyone, We got some yarn related errors when running spark 0.9.0 on hadoop 2.4 (but it was okay on hadoop 2.2). I didn't find any comments said spark 0.9.0 could support hadoop 2.4, so could I assume that we have to upgrade spark to the latest release version at this point to solve this issue

Re: spark 0.9.0 with hadoop 2.4 ?

I think I might find the root cause, YARN-1931 addressed the incompatible issue. The solution for my case might be either take related Spark patches or do an upgrade. On Wed, Jul 30, 2014 at 2:11 PM, yao wrote: > Hi Everyone, > > We got some yarn related errors when running spark

failed to build spark with maven for both 1.0.1 and latest master branch

Hi Folks, Today I am trying to build spark using maven; however, the following command failed consistently for both 1.0.1 and the latest master. (BTW, it seems sbt works fine: *sbt/sbt -Dhadoop.version=2.4.0 -Pyarn clean assembly)* Environment: Mac OS Mavericks Maven: 3.2.2 (installed by homebre

Re: failed to build spark with maven for both 1.0.1 and latest master branch

using maven under Mavericks, please let me know your result. Thanks Shengzhe On Thu, Jul 31, 2014 at 1:25 AM, Tathagata Das wrote: > Does a "mvn clean" or "sbt/sbt clean" help? > > TD > > On Wed, Jul 30, 2014 at 9:25 PM, yao wrote: > > Hi Folks, > &

Re: failed to build spark with maven for both 1.0.1 and latest master branch

-Phadoop-2.4 -DskipTests install > > FYI > > > On Thu, Jul 31, 2014 at 1:36 PM, yao wrote: > > > Hi TD, > > > > I've asked my colleagues to do the same thing but compile still fails. > > However, maven build succeeded once I built it on my personal

too many CancelledKeyException throwed from ConnectionManager

Hi Folks, We are testing our home-made KMeans algorithm using Spark on Yarn. Recently, we've found that the application failed frequently when doing clustering over 300,000,000 users (each user is represented by a feature vector and the whole data set is around 600,000,000). After digging into the

Re: too many CancelledKeyException throwed from ConnectionManager

you mentioned may be caused by the issues. > Now I'm trying to resolve the issue in https://github.com/apache/ > spark/pull/2019. > Please check it out. > > - Kousuke > > > (2014/08/26 8:53), yao wrote: > >> Hi Folks, >> >> We are testing our home

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

I would prefer keep both of them, it would be better even if that means pom.xml will be generated using sbt. Some company, like my current one, have their own build infrastructures built on top of maven. It is not easy to support sbt for these potential spark clients. But I do agree to only keep on

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

o use maven as alternative. Thanks -Shengzhe On Tue, Feb 25, 2014 at 3:40 PM, Patrick Wendell wrote: > Hey Yao, > > Would you mind explaining exactly how your company extends the Spark > maven build? For instance: > > (a) You are depending on Spark in your build and your build is usin

Re: [ANNOUNCE] Announcing Apache Spark 3.1.1

Congrats, all! Bests, Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark

Re: [VOTE] SPIP: Add FunctionCatalog

+1, looks great! Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi

Re: [VOTE] Release Spark 2.4.8 (RC2)

+1 (non-binding) Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Cool~Thanks, Hyukjin Yuanjian Li 于2021年4月14日周三 下午3:39写道： > Awesome! Thanks for making this happen, Hyukjin! > > Yi Wu 于2021年4月14日周三 下午2:51写道： > >> Thanks for the great work, Hyukjin! >> >> On Wed, Apr 14, 2021 at 1:00 PM Gengliang Wang wrote: >> >>> Thanks for the amazing work, Hyukjin! >>> I

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

to keep up with the current master branch？ Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi-tenant JDBC interface for large-scale d

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Cool, thanks! Hyukjin Kwon 于2021年4月14日周三 下午8:19写道： > Good point! I had to clarify. > Once is enough. The sync is needed for your branch to include the changes > of https://github.com/apache/spark/pull/32092. > > > 2021년 4월 14일 (수) 오후 9:11, Kent Yao 님이 작성: > >> Hi H

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

11:34写道： > > Thank you again, Hyukjin. > > Bests, > Dongjoon. > > On Wed, Apr 14, 2021 at 5:25 AM Kent Yao wrote: >> >> Cool, thanks! >> >> Hyukjin Kwon 于2021年4月14日周三 下午8:19写道： >>> >>> Good point! I had to clarify. >>> O

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

nks Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi-tenant JDBC interface for large-scale d

Re: [VOTE] Release Spark 2.4.8 (RC3)

gt; Version/s" = 2.4.8 >>> >>> Committers should look at those and triage. Extremely important bug >>> fixes, documentation, and API tweaks that impact compatibility should >>> be worked on immediately. Everything else please retarget to an >>>

Re: [VOTE] Release Spark 2.4.8 (RC4)

+1 (non-binding) Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified

Re: Apache Spark 3.1.2 Release?

+1. thanks Dongjoon Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified

Re: Resolves too old JIRAs as incomplete

+1，thanks Takeshi Kent Yao  @ Data Science Center, Hangzhou Research Institute, NetEase Corp. a spark enthusiast kyuubiis a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark. spark-authorizerA Spark SQL extension which provides SQL

Re: [VOTE] Release Spark 3.1.2 (RC1)

+1, non-binding Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi

Re: Apache Spark 3.0.3 Release?

+1. Thanks, Yi ~ Bests, Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a

Re: [VOTE] Release Spark 3.0.3 (RC1)

+1 (non-binding) Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark

Re:Time to start publishing Spark Docker Images?

+1Bests, Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi-tenant

Re: [VOTE] Release Spark 3.2.0 (RC7)

; } font{ line-height: 1.6; } Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiasthttps://github.com/yaooqinn/kyuubi"; style="box-sizing: border-box;">kyuubiis a unified multi-tenant J

Re: [VOTE] Release Spark 3.2.1 (RC1)

+1 (non-binding). Chao Sun 于2022年1月12日周三 16:10写道： > +1 (non-binding). Thanks Huaxin for driving the release! > > On Tue, Jan 11, 2022 at 11:56 PM Ruifeng Zheng > wrote: > >> +1 (non-binding) >> >> Thanks, ruifeng zheng >> >> -- Original -- >> *From:* "Cheng Su" ;

Re: [VOTE][SPIP] Support Customized Kubernetes Schedulers Proposal

+1 (non-binding) Thomas Graves 于2022年1月12日周三 11:52写道： > +1 (binding). > > One minor note since I haven't had time to look at the implementation > details is please make sure resource aware scheduling and the stage > level scheduling still work or any caveats are documented. Feel free > to ping m

Re: [VOTE] Release Spark 3.2.1 (RC2)

+1， non-binding John Zhuge 于2022年1月25日周二 06:56写道： > +1 (non-binding) > > On Mon, Jan 24, 2022 at 2:28 PM Cheng Su wrote: > >> +1 (non-binding) >> >> >> >> Cheng Su >> >> >> >> *From: *Chao Sun >> *Date: *Monday, January 24, 2022 at 2:10 PM >> *To: *Michael Heuer >> *Cc: *dev >> *Subject: *Re

Re: [VOTE] Release Spark 3.3.0 (RC2)

eRowRelation Both 3.2 releases and the master branch work fine with correct errors - 'due to data type mismatch'. Shall we backport the fix from the master to 3.3 too? Bests Kent Yao Yuming Wang 于2022年5月18日周三 19:04写道： > > -1. There is a regression: https://github.com/apache/sp

Re: [VOTE] Release Spark 3.3.0 (RC2)

Thanks, Maxim. Leave my -1 for this release candidate. Unfortunately, I don't know which PR fixed this. Does anyone happen to know? BR, Kent Yao Maxim Gekk 于2022年5月19日周四 13:42写道： > > Hi Kent, > > > Shall we backport the fix from the master to 3.3 too? > > Yes

Re: [VOTE] Release Spark 3.3.0 (RC2)

Wenchen Fan wrote: >> >> I think it should have been fixed by >> https://github.com/apache/spark/commit/0fdb6757946e2a0991256a3b73c0c09d6e764eed >> . Maybe the fix is not completed... >> >> On Thu, May 19, 2022 at 2:16 PM Kent Yao wrote: >&g

Re: 回复： [VOTE] Release Spark 3.3.0 (RC6)

+1, non-binding Xiao Li 于2022年6月14日周二 13:11写道： > > +1 > > Xiao > > beliefer 于2022年6月13日周一 20:04写道： >> >> +1 AFAIK, no blocking issues now. >> Glad to hear to release 3.3.0 ! >> >> >> 在 2022-06-14 09:38:35，"Ruifeng Zheng" 写道： >> >> +1 (non-binding) >> >> Maxim, thank you for driving this release

Re: [DISCUSS] SPIP: Support Docker Official Image for Spark

+1 Gengliang Wang 于2022年9月19日周一 09:23写道： > > +1, thanks for the work! > > On Sun, Sep 18, 2022 at 6:20 PM Hyukjin Kwon wrote: >> >> +1 >> >> On Mon, 19 Sept 2022 at 09:15, Yikun Jiang wrote: >>> >>> Hi, all >>> >>> >>> I would like to start the discussion for supporting Docker Official Image >

Re: [VOTE] SPIP: Support Docker Official Image for Spark

+1 Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi-tenant JDBC

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1, non-binding Gengliang Wang 于2022年11月16日周三 16:36写道： > > Hi all, > > I’d like to start a vote for SPIP: "Better Spark UI scalability and Driver > stability for large applications" > > The goal of the SPIP is to improve the Driver's stability by supporting > storing Spark's UI data on RocksDB.

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1(non-binding) Gengliang Wang 于2023年4月10日周一 15:27写道： > > +1 > > On Sun, Apr 9, 2023 at 3:17 PM Dongjoon Hyun wrote: >> >> +1 >> >> I verified the same steps like previous RCs. >> >> Dongjoon. >> >> >> On Sat, Apr 8, 2023 at 7:47 PM Mridul Muralidharan wrote: >>> >>> >>> +1 >>> >>> Signatures,

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1 (non-binding) L. C. Hsieh 于2023年4月10日周一 15:02写道： > > +1 > > Thanks Dongjoon > > On Sun, Apr 9, 2023 at 5:20 PM Dongjoon Hyun wrote: > > > > I'll start with my +1. > > > > I verified the checksum, signatures of the artifacts, and documentations. > > Also, ran the tests with YARN and K8s module

Re: Time for Spark v3.5.0 release

+1， thank you Kent On 2023/07/04 05:32:52 Dongjoon Hyun wrote: > +1 > > Thank you, Yuanjian > > Dongjoon > > On Tue, Jul 4, 2023 at 1:03 AM Hyukjin Kwon wrote: > > > Yeah one day postponed shouldn't be a big deal. > > > > On Tue, Jul 4, 2023 at 7:10 AM Yuanjian Li wrote: > > > >> Hi All, >

Re: Spark Docker Official Image is now available

Thank you, Yikun! Kent Dongjoon Hyun 于2023年7月20日周四 19:25写道： > Thank you! > > Dongjoon > > On Thu, Jul 20, 2023 at 8:40 AM Xiao Li > wrote: > >> Thank you, Yikun! This is great! >> >> On Wed, Jul 19, 2023 at 7:55 PM Ruifeng Zheng wrote: >> >>> Awesome, thank you YiKun for driving this! >>> >>

Re: Welcome two new Apache Spark committers

Congrats! Peter and Xiduo! Cheng Pan 于2023年8月7日周一 11:01写道： > > Congratulations! Peter and Xiduo! > > Thanks, > Cheng Pan > > > > On Aug 7, 2023, at 10:58, Gengliang Wang wrote: > > > > Congratulations! Peter and Xiduo! > > > > -

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1 (Non-binding) Kent Gengliang Wang 于2023年9月7日周四 14:09写道： > > +1 > > On Wed, Sep 6, 2023 at 9:46 PM Yuanjian Li wrote: >> >> +1 (non-binding) >> >> Xiao Li 于2023年9月6日周三 15:27写道： >>> >>> +1 >>> >>> Xiao >>> >>> Herman van Hovell 于2023年9月6日周三 22:08写道： Tested connect, and everything l

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 (non-binding), great work! Kent Yao Yuming Wang 于2023年9月12日周二 11:32写道： > > +1. > > On Tue, Sep 12, 2023 at 10:57 AM yangjie01 > wrote: >> >> +1 >> >> >> >> 发件人: Jia Fan >> 日期: 2023年9月12日 星期二 10:08 >> 收件人: Ruifeng Zheng >&

Re: Welcome to Our New Apache Spark Committer and PMCs

Congrats! Kent 在 2023年10月7日星期六，John Zhuge 写道： > Congratulations! > > On Fri, Oct 6, 2023 at 6:41 PM Yi Wu wrote: > >> Congrats! >> >> On Sat, Oct 7, 2023 at 9:24 AM XiDuo You wrote: >> >>> Congratulations! >>> >>> Prashant Sharma 于2023年10月6日周五 00:26写道： >>> > >>> > Congratulations 🎉 >>> > >>

Re: Apache Spark 3.4.2 (?)

+1 Maxim Gekk 于2023年11月9日周四 18:18写道： > > +1 > > On Wed, Nov 8, 2023 at 5:29 AM kazuyuki tanimura > wrote: >> >> +1 >> >> Kazu >> >> On Nov 7, 2023, at 5:23 PM, L. C. Hsieh wrote: >> >> +1 >> >> On Tue, Nov 7, 2023 at 4:56 PM Dongjoon Hyun wrote: >> >> >> Thank you all! >> >> Dongjoon >> >> On

[DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

. Thanks, Kent Yao [1] https://docs.google.com/document/d/1hWl5Q2CNNOjN5Ubyoa28XmpJtDyD9BtGtiEG2TT94rg/edit?usp=sharing [2] https://github.com/jestjs/jest [3] https://github.com/apache/spark/pull/43903 - To unsubscribe e-mail

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

posing a new UI test framework for Apache Spark 4.0. > > > > It looks very useful. > > > > Thanks, > > Dongjoon. > > > > > > On Tue, Nov 21, 2023 at 1:51 AM Kent Yao wrote: > > > >> Hi Spark Dev, > >> > >> This is a cal

[VOTE] SPIP: Testing Framework for Spark UI Javascript files

ccept the proposal as an official SPIP [ ] +0 [ ] -1: I don’t think this is a good idea because … Thank you! Kent Yao [1] https://lists.apache.org/thread/5rqrho4ldgmqlc173y2229pfll5sgkff [2] https://docs.google.com/document/d/1hWl5Q2CNNOjN5Ubyoa28XmpJtDyD9BtGtiEG2TT94rg/edit?usp=sh

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

023 at 6:58 AM Gengliang Wang wrote: > > > > > +1 > > > > > > On Sat, Nov 25, 2023 at 2:50 AM yangjie01 > > > wrote: > > > > > >> +1 > > >> > > >> > > >> > > >> *发件人**: *Reynold Xin &g

[RESULT][VOTE] SPIP: Testing Framework for Spark UI Javascript files

Hi Spark dev, The vote[1] has now closed. The results are: +1 Votes(*=binding): - Mridul Muralidharan* - Ye Zhou - Dongjoon Hyun* - Reynold Xin* - Yang Jie - Gengliang Wang* - Ruifeng Zheng* - Binjie Yang - Kent Yao 0 Votes: None -1 Votes: None The vote is successful with 5 binding +1 votes

Re: [VOTE] Release Spark 3.4.2 (RC1)

+1(non-binding) Kent Yao On 2023/11/27 01:12:53 Dongjoon Hyun wrote: > Hi, Marc. > > Given that it exists in 3.4.0 and 3.4.1, I don't think it's a release > blocker for Apache Spark 3.4.2. > > When the patch is ready, we can consider it for 3.4.3. > > In

Re: Apache Spark 3.3.4 EOL Release?

+1 Thank you for driving this EOL release, Dongjoon! Kent Yao On 2023/12/04 19:40:10 Mridul Muralidharan wrote: > +1 > > Regards, > Mridul > > On Mon, Dec 4, 2023 at 11:40 AM L. C. Hsieh wrote: > > > +1 > > > > Thanks Dongjoon! > > >

Re: [VOTE] Release Spark 3.3.4 (RC1)

+1(non-binding Kent Yao Yuming Wang 于2023年12月11日周一 09:33写道： > > +1 > > On Mon, Dec 11, 2023 at 5:55 AM Dongjoon Hyun wrote: >> >> +1 >> >> Dongjoon >> >> On 2023/12/08 21:41:00 Dongjoon Hyun wrote: >> > Please vote on releasing t

Re: [DISCUSS] Release Spark 3.5.1?

+1 Jungtaek Lim 于2024年2月3日周六 21:14写道： > > Hi dev, > > looks like there are a huge number of commits being pushed to branch-3.5 > after 3.5.0 was released, 200+ commits. > > $ git log --oneline v3.5.0..HEAD | wc -l > 202 > > Also, there are 180 JIRA tickets containing 3.5.1 as fixed version, and

Re: [DISCUSS] Support year-month and day-time Intervals

our proposal need to revert something from the master branch?I'm just asking because it's not clear in the proposal document.Bests,Dongjoon.On Fri, Jan 10, 2020 at 5:31 AM Dr. Kent Yao <yaooq...@qq.com> wrote:Hi, Devs I’d like to propose to add two new interval types which are year

Re: [Spark SQL]: SQL, Python, Scala and R API Consistency

really test most of the functions beyond the simple> "callability". One the complex ones, that require some non-trivial> transformations of arguments, are fully tested.> --> Best regards,> Maciej Szymkiewicz> Web: https://zero323.net> Keybase: https://keybas

Re: [VOTE] Release Spark 3.1.1 (RC1)

27;s behavior for tuning the IO performance w/ HDFS if there is an existing io.file.buffer.size in hive-site.xml Kent Yao @ Data Science Center, Hangzhou Research Institut

Re:[VOTE] Release Spark 3.1.1 (RC2)

+1 Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified multi-tenant JDBC

Re: Spark development with IntelliJ

*I followed the procedures instructed by https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-IntelliJ. But problems still occurs which has made me a little bit annoyed. My environment settings are:JAVA 1.7.0 Scala: 2.10.4 Spark:1.2.0, Intellij Idea 14.0.2

Re: Spark development with IntelliJ

Followed is the discussion between Imran and me. 2015-01-18 4:12 GMT+08:00 Chunnan Yao : > Thank you for your patience! Im now not so familiar with the mailing list. > I just clicked "reply" in Gmail, thinking it would be automatically > attached to the list. I will la

Re: Spark development with IntelliJ

Nice! - Feel the sparking Spark! -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-development-with-IntelliJ-tp10032p10167.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. ---

Is this a bug in MLlib.stat.test ? About the mapPartitions API used in Chi-Squared test

Hi everyone! I am digging into MLlib of Spark 1.2.1 currently. When reading codes of MLlib.stat.test, in the file ChiSqTest.scala under /spark/mllib/src/main/scala/org/apache/spark/mllib/stat/test, I am confused by the usage of mapPartitions API in the function def chiSquaredFeatures(data: RDD[La

Support parallelized online matrix factorization for Collaborative Filtering

On-line Collaborative Filtering(CF) has been widely used and studied. To re-train a CF model from scratch every time when new data comes in is very inefficient (http://stackoverflow.com/questions/27734329/apache-spark-incremental-training-of-als-model). However, in Spark community we see few discus

Indices of SparseVector must be ordered while computing SVD

Hi all, I am using Spark 1.3.1 to write a Spectral Clustering algorithm. This really confused me today. At first I thought my implementation is wrong. It turns out it's an issue in MLlib. Fortunately, I've figured it out. I suggest to add a hint on user document of MLlib ( as far as I know, ther

Possible long lineage issue when using DStream to update a normal RDD

Hi all, Recently in our project, we need to update a RDD using data regularly received from DStream, I plan to use "foreachRDD" API to achieve this: var MyRDD = ... dstream.foreachRDD { rdd => MyRDD = MyRDD.join(rdd)... ... } Is this usage correct? My concern is, as I am repeatedly

How can I do pair-wise computation between RDD feature columns?

Hi all, Recently I've ran into a scenario to conduct two sample tests between all paired combination of columns of an RDD. But the networking load and generation of pair-wise computation is too time consuming. That has puzzled me for a long time. I want to conduct Wilcoxon rank-sum test (http://en

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

+1 (non-binding) Kent Yao Hyukjin Kwon 于2024年3月11日周一 17:26写道： > > +1 > > On Mon, 11 Mar 2024 at 18:11, yangjie01 wrote: >> >> +1 >> >> >> >> Jie Yang >> >> >> >> 发件人: Haejoon Lee >> 日期: 2024年3月11日 星期一 17:09

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

+1(non-binding). Thank you, Hyukjin. Kent Yao Takuya UESHIN 于2024年4月1日周一 18:04写道： > > +1 > > On Sun, Mar 31, 2024 at 6:16 PM Hyukjin Kwon wrote: >> >> Hi all, >> >> I'd like to start the vote for SPIP: Pure Python Package in PyPI (Spark >&g

Re: Apache Spark 3.4.3 (?)

+1, thank you, Dongjoon Kent Holden Karau 于2024年4月7日周日 14:54写道： > > Sounds good to me :) > > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > > On Sat, Ap

Re: [VOTE] Release Spark 3.4.3 (RC2)

+1(non-binding) Thanks, Kent Yao bo yang 于2024年4月17日周三 09:49写道： > > +1 > > On Tue, Apr 16, 2024 at 1:38 PM Hyukjin Kwon wrote: >> >> +1 >> >> On Wed, Apr 17, 2024 at 3:57 AM L. C. Hsieh wrote: >>> >>> +1 >>> >>> On Tue, A

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 Nit: the umbrella ticket is SPARK-44111, not SPARK-4. Thanks, Kent Yao Dongjoon Hyun 于2024年4月25日周四 14:39写道： > > Hi, All. > > It's great to see community activities to polish 4.0.0 more and more. > Thank you all. > > I'd like to bring SPARK-46122 (ano

Re: [FYI] SPARK-47993: Drop Python 3.8

+1 yangjie01 于2024年4月26日周五 17:16写道： > > +1 > > > > 发件人: Ruifeng Zheng > 日期: 2024年4月26日 星期五 15:05 > 收件人: Xinrong Meng > 抄送: Dongjoon Hyun , "dev@spark.apache.org" > > 主题: Re: [FYI] SPARK-47993: Drop Python 3.8 > > > > +1 > > > > On Fri, Apr 26, 2024 at 10:26 AM Xinrong Meng wrote: > > +1 > >

Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 Kent Yao On 2024/04/30 09:07:21 Yuming Wang wrote: > +1 > > On Tue, Apr 30, 2024 at 3:31 PM Ye Xianjin wrote: > > > +1 > > Sent from my iPhone > > > > On Apr 30, 2024, at 3:23 PM, DB Tsai wrote: > > > >  > > +1 > > > > On

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

+1 Dongjoon Hyun 于2024年5月13日周一 08:39写道： > > +1 > > On Sun, May 12, 2024 at 3:50 PM huaxin gao wrote: >> >> +1 >> >> On Sat, May 11, 2024 at 4:35 PM L. C. Hsieh wrote: >>> >>> +1 >>> >>> On Sat, May 11, 2024 at 3:11 PM Chao Sun wrote: >>> > >>> > +1 >>> > >>> > On Sat, May 11, 2024 at 2:10 PM L

Re: [VOTE] SPARK 4.0.0-preview1 (RC2)

-1 You've updated your key in [2] with a new one [1]. I believe you should add your new key without removing the old one. Otherwise, users cannot verify those archived releases you published. Thanks, Kent Yao [1] https://dist.apache.org/repos/dist/dev/spark/KEYS [2]

Re: [VOTE] SPARK 4.0.0-preview1 (RC3)

necessary. Thanks, Kent Yao [1] jackson-core-asl-1.9.13.jar [2] jackson-mapper-asl-1.9.13.jar On 2024/05/28 18:52:32 Wenchen Fan wrote: > one correction: "The tag to be voted on is v4.0.0-preview1-rc2 (commit > 7cfe5a6e44e8d7079ae29ad3e2cee7231cd3dc66)" should be "The tag to be

Re: [外部邮件] Re: [VOTE] Move Spark Connect server to builtin package (Client API layer stays external)

+1 (non-binding), Kent Martin Grund 于2024年7月3日周三 14:11写道： > > +1 (non-binding) > > On Wed, Jul 3, 2024 at 07:25 Holden Karau wrote: >> >> +1 >> >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 >> YouTube Live Streams: h

[DISCUSS] Release Apache Spark 3.5.2

3.5.2, including some blockers and critical issues. What do you think about releasing 3.5.2? I am volunteering to take on the role of release manager for 3.5.2. Bests, Kent Yao [1] https://spark.apache.org/news/spark-3-5-1-released.html [2] https://issues.apache.org/jira/projects/SPARK/versions/123

Re: [DISCUSS] Release Apache Spark 3.5.2

>>> Thank you for the head-up and volunteering, Kent. > >>> > >>> +1 for 3.5.2 release. > >>> > >>> I can help you with the release steps which require Spark PMC permissions. > >>> > >>> Please let me know if you have any

[VOTE] Release Spark 3.5.2 (RC1)

hat has not been correctly targeted please ping me or a committer to help target the issue. Thanks, Kent Yao - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Spark 3.5.2 (RC1)

Thank you Wenchen. The vote is open until Jul 21, 11 AM UTC. Considering that the deadline falls on a weekend, the results might be counted on the following Monday. Bests, Kent Yao Wenchen Fan 于2024年7月18日周四 18:54写道： > > > The vote is open until Jul 18 > > Is it a typo? It&

Re: [VOTE] Release Spark 3.5.2 (RC1)

Thank you, Huaxin and L. C. Hsieh, for your input. We shall also include PRs like https://github.com/apache/spark/pull/47412 for correctness So, 3.5.2-RC1 failed, I will start RC2 in two or three days. Kent Yao L. C. Hsieh 于2024年7月19日周五 13:02写道： > > I also support -1 to include t

[VOTE] Release Spark 3.5.2 (RC2)

ression that has not been correctly targeted please ping me or a committer to help target the issue. Thanks, Kent Yao - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Differentiate Spark without Spark Connect from Spark Connect

+1 On 2024/07/23 02:04:17 Herman van Hovell wrote: > +1 > > On Mon, Jul 22, 2024 at 8:56 PM Wenchen Fan wrote: > > > +1 > > > > On Tue, Jul 23, 2024 at 8:40 AM Xinrong Meng wrote: > > > >> +1 > >> > >> Thank you @Hyukjin Kwon ! > >> > >> On Mon, Jul 22, 2024 at 5:20 PM Gengliang Wang wrote:

Re: [外部邮件] [VOTE] Release Spark 3.5.2 (RC2)

+1(non-binding), I have checked: - Download links are OK - Signatures, Checksums, and the KEYS file are OK - LICENSE and NOTICE are present - No unexpected binary files in source releases - Successfully built from source Thanks, Kent Yao On 2024/07/23 06:55:28 yangjie01 wrote: > +1, Thanks K

Re: [外部邮件] [VOTE] Release Spark 3.5.2 (RC2)

ly from the Delta metadata and > given to `PartitionedFile` and the Parquet reader output column vector for > partition column has the correct values. but somewhere the partition value > becomes null. Still debugging the RCA. > > Thanks > Venki > > On Tue, Jul 23, 2024 a

Re: [外部邮件] [VOTE] Release Spark 3.5.2 (RC2)

pache/spark/pull/47483 > > Can we cut a new RC? > > On Thu, Jul 25, 2024 at 3:13 PM Mridul Muralidharan wrote: >> >> >> +1 >> >> Signatures, digests, etc check out fine. >> Checked out tag and build/tested with -Phive -Pyarn -Pkubernetes >> &g

[VOTE] Release Spark 3.5.2 (RC3)

ression that has not been correctly targeted please ping me or a committer to help target the issue. Thanks, Kent Yao - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Spark 3.5.2 (RC3)

gt; > FYI, Python 3.12 Support was added to Apache Spark 4.0.0 only and `master` > branch is not affected. > > Dongjoon. > > > On Thu, Jul 25, 2024 at 6:06 AM Kent Yao wrote: > > > Hi dev, > > > > Please vote on releasing the following candidate as Apache Sp

[VOTE] Release Spark 3.5.2 (RC4)

ression that has not been correctly targeted please ping me or a committer to help target the issue. Thanks, Kent Yao - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Spark 3.5.2 (RC4)

: >> >> +1 (non-binding) >> >> - All links are valid and look good >> - Successful built from source code on Ubuntu 22.04 x86 with Java 17 >> - Have integrated and played with Zeppelin, Kyuubi, Iceberg and Hadoop, no >> unexpected issues found. >> >

[NOTICE] Progress of 3.5.2-RC5

/jira/browse/SPARK-49000 https://issues.apache.org/jira/browse/SPARK-49054 === ONGOING === https://issues.apache.org/jira/browse/SPARK-48950 https://issues.apache.org/jira/browse/SPARK-49030 Thanks, Kent Yao [1] https://lists.apache.org/thread/9lj57fh3zbo2h4koh5hr7nhdky21p6zg

Re: [NOTICE] Progress of 3.5.2-RC5

anymore; I will proceed to cut it soon. Thank you, everyone, for your understanding and patience. Kent Yao On 2024/08/01 20:32:07 Dongjoon Hyun wrote: > Thank you for summarizing them and leading the release, Kent. :) > > Dongjoon. > > On Wed, Jul 31, 2024 at 10:39 PM

  1   2   >