Re: [VOTE] Release Spark 4.0.0 (RC2)

Here is one more problem I found during RC2 verification: https://github.com/apache/spark/pull/50173 This one is just a test issue. Chris Nauroth On Tue, Mar 4, 2025 at 2:55 PM Jules Damji wrote: > - 1 (non-binding) > > A ran into number of installation and launching problems. Ma

Re: [VOTE] Release Spark 4.0.0 (RC2)

-1 (non-binding) I think I found some missing license information in the binary distribution. We may want to include this in the next RC: https://github.com/apache/spark/pull/50158 Thank you for putting together this RC, Wenchen. Chris Nauroth On Mon, Mar 3, 2025 at 6:10 AM Wenchen Fan

Re: [VOTE] Release Spark 3.5.5 (RC1)

/network_wordcount.py localhost Chris Nauroth On Wed, Feb 26, 2025 at 2:13 PM Sakthi wrote: > +1 (non-binding) > > On Wed, Feb 26, 2025 at 9:20 AM Dongjoon Hyun wrote: > >> Thank you for the explicit casting, Vlad. >> >> BTW, the Apache Spark 3.5.5 RC1 vote will continue l

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

/src/main/python/streaming/network_wordcount.py localhost Thank you, Dongjoon! Chris Nauroth On Wed, Apr 12, 2023 at 3:49 AM Shaoyun Chen wrote: > +1 (non-binding) > > On 2023/04/12 04:36:59 Jungtaek Lim wrote: > > +1 (non-binding) > > > > Thanks for driving the r

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

/network_wordcount.py localhost Chris Nauroth On Tue, Apr 11, 2023 at 10:36 PM beliefer wrote: > +1 > > > At 2023-04-08 07:29:46, "Xinrong Meng" wrote: > > Please vote on releasing the following candidate(RC7) as Apache Spark > version 3.4.0. > > The vote is open u

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1 (non-binding) Gengliang, thank you for the SPIP. Chris Nauroth On Wed, Nov 16, 2022 at 4:27 AM Maciej wrote: > +1 > > On 11/16/22 13:19, Yuming Wang wrote: > > +1, non-binding > > > > On Wed, Nov 16, 2022 at 8:12 PM Yang,Jie(INF) > <mailto:yangji...@baidu

Re: [VOTE] Release Spark 3.2.3 (RC1)

* bin/spark-submit examples/src/main/python/streaming/network_wordcount.py localhost Chao, thank you for preparing the release. Chris Nauroth On Wed, Nov 16, 2022 at 5:22 AM Yuming Wang wrote: > +1 > > On Wed, Nov 16, 2022 at 2:28 PM Yang,Jie(INF) wrote: > >> I swit

Re: Missing data in spark output

re using speculative execution, then you could also potentially try turning that off. Chris Nauroth On Wed, Oct 19, 2022 at 8:18 AM Martin Andersson wrote: > Is your spark job batch or streaming? > -- > *From:* Sandeep Vinayak > *Sent:* Tuesday, Oc

Re: [VOTE] Release Spark 3.3.0 (RC6)

+1 (non-binding) I repeated all checks I described for RC5: https://lists.apache.org/thread/ksoxmozgz7q728mnxl6c2z7ncmo87vls Maxim, thank you for your dedication on these release candidates. Chris Nauroth On Mon, Jun 13, 2022 at 3:21 PM Mridul Muralidharan wrote: > > +1 > >

Re: [VOTE] Release Spark 3.3.0 (RC5)

#x27;2018-11-17' > 1" * SPARK-39293 ArrayAggregate fix Chris Nauroth On Tue, Jun 7, 2022 at 1:30 PM Cheng Su wrote: > +1 (non-binding). Built and ran some internal test for Spark SQL. > > > > Thanks, > > Cheng Su > > > > *From: *L. C. Hsieh >

Re: [VOTE] Release Spark 3.3.0 (RC3)

that blocked RC2: * bin/spark-sql -e 'SELECT (SELECT IF(x, 1, 0)) AS a FROM (SELECT true) t(x) UNION SELECT 1 AS a;' * bin/spark-sql -e "select date '2018-11-17' > 1" Chris Nauroth On Wed, May 25, 2022 at 8:00 AM Sean Owen wrote: > +1 works for me as

Re: CVE-2021-38296: Apache Spark Key Negotiation Vulnerability - 2.4 Backport?

Thanks for the quick reply, Sean! Chris Nauroth On Thu, Apr 14, 2022 at 10:15 AM Sean Owen wrote: > It does affect 2.4.x, yes. 2.4.x was EOL a while ago, so there wouldn't be > a new release of 2.4.x in any event. It's recommended to update instead, at > least to 3.1.3.

CVE-2021-38296: Apache Spark Key Negotiation Vulnerability - 2.4 Backport?

A fix for CVE-2021-38296 was committed and released in Apache Spark 3.1.3. I'm curious, is the issue relevant to the 2.4 version line, and if so, are there any plans for a backport? https://lists.apache.org/thread/70x8fw2gx3g9ty7yk0f2f1dlpqml2smd Chris Nauroth

[no subject]

Unsubscribe

Re: PyCharm 2020 :: pyspark installation issue

Hi, Try installing pypandoc first (“pip install pypandoc” or “pip3 install pypandoc” depending on how your python is set up). Then install pyspark. Chris > On 16 May 2020, at 07:25, kanchan pradhan wrote: > >  > Hi, > > Please help me to resolve the below issue is comin

Fwd: Sample date_trunc error for webpage （https://spark.apache.org/docs/2.3.0/api/sql/#date_trunc ）

Spark, We received this message. I have not ACKd it. -Chris INFRA > Begin forwarded message: > > From: "binggan1989" > Subject: Sample date_trunc error for webpage > （https://spark.apache.org/docs/2.3.0/api/sql/#date_trunc ） > Date: July 5, 2019 at 2:54:54 AM PD

Re: Thoughts on dataframe cogroup?

Yes, totally agreed with Li here. For clarity, I'm happy to do the work to implement this, but it would be good to get feedback from the community in general and some of the Spark committers in particular. thanks, Chris On Wed, Apr 17, 2019 at 9:17 PM Li Jin wrote: > I have l

Re: Thoughts on dataframe cogroup?

Ah sorry- I've updated the link which should give you access. Can you try again now? thanks, Chris On Mon, Apr 15, 2019 at 9:49 PM Li Jin wrote: > Hi Chris, > > Thanks! The permission to the google doc is maybe not set up properly. I > cannot view the doc by default. >

Re: Thoughts on dataframe cogroup?

ssion as the grouping key (this would mean that you wouldn't have to marshal the key into a JVM object and could possible lend itself to some catalyst optimisations) but I don't think that this should be done as part of this SPIP. thanks, Chris On Mon, Apr 15, 2019 at 6:27 PM Ryan Blue w

Re: Thoughts on dataframe cogroup?

Hi, As promised I’ve raised SPARK-27463 for this. All feedback welcome! Chris > On 9 Apr 2019, at 13:22, Chris Martin wrote: > > Thanks Bryan and Li, that is much appreciated. Hopefully should have the > SPIP ready in the next couple of days. > > thanks, > > Chr

Re: Thoughts on dataframe cogroup?

Thanks Bryan and Li, that is much appreciated. Hopefully should have the SPIP ready in the next couple of days. thanks, Chris On Mon, Apr 8, 2019 at 7:18 PM Bryan Cutler wrote: > Chirs, an SPIP sounds good to me. I agree with Li that it wouldn't be too > difficult to extend th

Re: Thoughts on dataframe cogroup?

multiple dataframes may not be straightforward. If anyone has any ideas as to how this might be achieved in an elegant manner I’d be happy to hear them! Thanks, Chris > On 26 Feb 2019, at 14:55, Li Jin wrote: > > Thank you both for the reply. Chris and I have very similar use cases for

Re: [DISCUSS] Enable blacklisting feature by default in 3.0

eedback now: two question survey "Have you enabled blacklisting?" and "What settings did you use?" -Chris On Mon, Apr 1, 2019 at 9:05 AM Ankur Gupta wrote: > Hi Chris, > > Thanks for sending over the example. As far as I can understand, it seems > that this would

Re: Thoughts on dataframe cogroup?

on dataframes this would have been a bit nicer. The ideal would have been some pandas udf version of cogroup that gave me a pandas dataframe for each spark dataframe in the cogroup! Chris > On 26 Feb 2019, at 00:38, Jonathan Winandy wrote: > > For info, in our team have defined our ow

Hive Bucketing Support

pache/spark/pull/19001) that's been open for almost a year now, with the last comment being over a month ago. Does anyone know if I should remain hopeful that this support will be added in the near future or is it one of those things that's realistically going to be some distance off. thanks, Chris

Re: Revisiting Online serving of Spark models?

g Runtime is only available to Databricks customers. This seems in conflict with the community efforts described here. Can you comment on behalf of Databricks? Look forward to your response, joseph. See you all soon! — Chris Fregly Founder @ PipelineAI <https://pipeline.ai/> (100,000 User

Using Spark 2.2.0 SparkSession extensions to optimize file filtering

I have an external catalog that has additional information on my Parquet files that I want to match up with the parsed filters from the plan to prune the list of files included in the scan. I’m looking at doing this using the Spark 2.2.0 SparkSession extensions similar to the built in partition

Re: SPIP: Spark on Kubernetes

@reynold: Databricks runs their proprietary product on Kubernetes. how about contributing some of that work back to the Open Source Community? — Chris Fregly Founder and Research Engineer @ PipelineAI <http://pipeline.io/> Founder @ Advanced Spark and TensorFlow Meetup <http://www.m

Re: Structured Streaming Spark Summit Demo - Databricks people

they’re presented on stage - especially the proprietary demos involving dbutils() and display(). Chris Fregly Research Scientist @ PipelineIO Founder @ Advanced Spark and TensorFlow Meetup San Francisco - Chicago - Washington DC - London On Feb 15, 2017, 12:14 PM -0800, Nicholas Chammas , wrote

Re: unhelpful exception thrown on predict() when ALS trained model doesn't contain user or product?

Ah cool, thanks for the link! On 6 December 2016 at 12:25, Nick Pentreath wrote: > Indeed, it's being tracked here: https://issues.apache. > org/jira/browse/SPARK-18230 though no Pr has been opened yet. > > > On Tue, 6 Dec 2016 at 13:36 chris snow wrote

unhelpful exception thrown on predict() when ALS trained model doesn't contain user or product?

I'm using the MatrixFactorizationModel.predict() method and encountered the following exception: Name: java.util.NoSuchElementException Message: next on empty iterator StackTrace: scala.collection.Iterator$$anon$2.next(Iterator.scala:39) scala.collection.Iterator$$anon$2.next(Iterator.scala:37) sc

Evolutionary algorithm (EA) in Spark

some guidance on how to go about it. Regards, Chris Lin -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Evolutionary-algorithm-EA-in-Spark-tp19716.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Evolutionary algorithm (EA) in Spark

some guidance on how to go about it. Regards, Chris Lin -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Evolutionary-algorithm-EA-in-Spark-tp19715.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Re: Straw poll: dropping support for things like Scala 2.10

i seem to remember a large spark user (tencent, i believe) chiming in late during these discussions 6-12 months ago and squashing any sort of deprecation given the massive effort that would be required to upgrade their environment. i just want to make sure these convos take into consideration la

Re: Serving Spark ML models via a regular Python web app

ection and reinventing the wheel. not quite sure where that project will go. doesn't seem like it will have a long shelf-life in my opinion. check out pipeline.io. some cool stuff in there. > On Aug 11, 2016, at 9:35 AM, Chris Fregly wrote: > > this is exactly what my http://pipeline.

Re: Serving Spark ML models via a regular Python web app

this is exactly what my http://pipeline.io project is addressing. check it out and send me feedback or create issues at that github location. > On Aug 11, 2016, at 7:42 AM, Nicholas Chammas > wrote: > > Thanks Michael for the reference, and thanks Nick for the comprehensive > overview of exi

Re: Spark 2.0.1 / 2.1.0 on Maven

nditions placed on >> the package. If you find that the general public are downloading such test >> packages, then remove them. >> > > On Tue, Aug 9, 2016 at 11:32 AM, Chris Fregly wrote: > >> this is a valid question. there are many people building products and >&

Re: [VOTE] Release Apache Spark 2.0.0 (RC1)

, also, there is no python support, no samples on the pr > > demonstrating how to use security capabilities and no documentation > updates. > > > > Thanks > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.c

Re: Return binary mode in ThriftServer

+1 on bringing it back. causing all sorts of problems on my end that was not obvious without digging in I was having problems building spark, as well, with the --hive-thriftserver flag. also thought I was doing something wrong on my end. > On Jun 13, 2016, at 9:11 PM, Reynold Xin wrote: > >

Re: Creating Spark Extras project, was Re: SPARK-13843 and future of streaming backends

Spark* which is *at the ASF*. We welcome comments and as Luciano said, this is meant to invite and be open to those in the Apache Spark PMC to join and help. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and

Re: Creating Spark Extras project, was Re: SPARK-13843 and future of streaming backends

moved out of Apache source control from Apache Spark - it’s meant to be an enabler to suggest that code could be managed here just as easily (see the difference?) Let me know what you think thanks Reynold. Cheers, Chris ++ Chris

Re: Creating Spark Extras project, was Re: SPARK-13843 and future of streaming backends

Yeah, so it’s the *Apache Spark* project. Just to clarify. Not once did you say Apache Spark below. On 4/15/16, 9:50 AM, "Sean Owen" wrote: >On Fri, Apr 15, 2016 at 5:34 PM, Luciano Resende wrote: >> I know the name might be confusing, but I also think that the projects have >> a very big

Re: Creating Spark Extras project, was Re: SPARK-13843 and future of streaming backends

> > >> Project Name > >> > >> It would be ideal if we could have a project name that shows close ties > to > >> Spark (e.g. Spark Extras or Spark Connectors) but we will need > permission > >> and support from whoever is going to evaluate the project proposal (e.g. > >> Apache Board) > >> > >> > >> Thoughts ? > >> > >> Does anyone have any big disagreement or objection to moving into this > >> direction ? > >> > >> Otherwise, who would be interested in joining the project, so I can > start > >> working on some concrete proposal ? > >> > >> > > > > > > > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > > -- *Chris Fregly* Principal Data Solutions Engineer IBM Spark Technology Center, San Francisco, CA http://spark.tc | http://advancedspark.com

Re: Switch RDD-based MLlib APIs to maintenance mode in Spark 2.0

perhaps renaming to Spark ML would actually clear up code and documentation confusion? +1 for rename > On Apr 5, 2016, at 7:00 PM, Reynold Xin wrote: > > +1 > > This is a no brainer IMO. > > >> On Tue, Apr 5, 2016 at 7:32 PM, Joseph Bradley wrote: >> +1 By the way, the JIRA for tracking

Re: SPARK-13843 and future of streaming backends

misreading your intent. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email

RE: Specifying Scala types when calling methods from SparkR

--- From: Shivaram Venkataraman [mailto:shiva...@eecs.berkeley.edu] Sent: Thursday, December 10, 2015 8:21 AM To: Chris Freeman Cc: dev@spark.apache.org Subject: Re: Specifying Scala types when calling methods from SparkR The SparkR callJMethod can only invoke methods as they show up in the Java byte co

Specifying Scala types when calling methods from SparkR

pelineModel]", modelPath, SparkR:::getMinPartitions(sc, NULL)) return(modelRDD) } Any help is appreciated! -- Chris Freeman

Re: Removing the Mesos fine-grained mode

think the direction is clear that the coarse mode is the proper way forward, and having the two code paths is just noise. -Chris From: Iulian Dragoș mailto:iulian.dra...@typesafe.com>> Date: Thursday, November 19, 2015 at 6:42 AM To: "dev@spark.apache.org<mailto:dev@spark.apache.org

Re: Please reply if you use Mesos fine grained mode

something like Tachyon to achieve in fine grain mode. From: Timothy Chen mailto:tnac...@gmail.com>> Date: Wednesday, November 4, 2015 at 11:05 AM To: "Heller, Chris" mailto:chel...@akamai.com>> Cc: Reynold Xin mailto:r...@databricks.com>>, "dev@spark.apach

Re: Please reply if you use Mesos fine grained mode

We’ve been making use of both. Fine-grain mode makes sense for more ad-hoc work loads, and coarse-grained for more job like loads on a common data set. My preference is the fine-grain mode in all cases, but the overhead associated with its startup and the possibility that an overloaded cluster w

Maven issues with 1.5-RC

? Running on CentOS with Java 7, for what it’s worth. -- Chris Freeman Senior Content Engineer - Alteryx (657) 900 5462

Fwd: posts are not accepted

Sent from my iPhone Begin forwarded message: From: Rob Sargent mailto:rob.sarg...@utah.edu>> Date: July 23, 2015 at 1:14:04 PM PDT To: mailto:user-ow...@spark.apache.org>> Subject: posts are not accepted Hello, my user name is iceback and my email is rob.sarg...@utah.edu

A proposal for Test matrix decompositions for speed/stability (SPARK-7210)

me know how this sounds, and I can provide some personal details about my experience and motivations. Thanks, Chris

Re: Sidebar: issues targeted for 1.4.0

were complete? If a JIRA targeting a release is blocking entry to the RC phase, and its determined that the JIRA should not hold up the release, than it should get re-targeted to the next release. -Chris On 6/17/15, 3:55 PM, "Patrick Wendell" wrote: >Hey Sean, > >Thanks for b

Spark Packages: using sbt-spark-package tool with R

JAR created via SBT assembly. I know there’s an existing option for including Python files but I haven’t been able to crack the code on how I might be able to include R files. Any advice is appreciated! -Chris Freeman

Re: Spark + Kinesis

n this. thanks! -chris On Tue, Apr 7, 2015 at 6:17 PM, Vadim Bichutskiy wrote: > Hey y'all, > > While I haven't been able to get Spark + Kinesis integration working, I > pivoted to plan B: I now push data to S3 where I set up a DStream to > monitor an S3 bucket with textFi

FW: Trouble posting to the list

FYI ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW

SciSpark: NASA AIST14 proposal

. html I will keep everyone posted and plan on interacting with the list over here to get it done. I expect that we’ll start work in March. In the meanwhile you guys can scope the abstract at the link provided. Happy to chat about it if you have any questions too. Cheers! Chris

Re: Data Locality In Spark

and even the same process where the data might be cached. these are the different locality levels: PROCESS_LOCAL NODE_LOCAL RACK_LOCAL ANY relevant code: https://github.com/apache/spark/blob/7712e724ad69dd0b83754e938e9799d13a4d43b9/core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSu

2nd Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2)

anta Barbara, USA =80 Jong-Suk Ruth Lee, National Institute of Supercomputing and Networking, KISTI (Korea Institute of Science and Technology Information), Korea =80 James Lin, Shanghai Jiao Tong University, China =80 Frank L=F6ffler, Louisiana State University, USA =80 Chris A. Mattmann, NASA JPL

Re: Could you undo the JIRA dev list e-mails?

No worries, thanks Patrick, agreed. -Original Message- From: Patrick Wendell Date: Saturday, March 29, 2014 1:47 PM To: Chris Mattmann Cc: Chris Mattmann , "dev@spark.apache.org" Subject: Re: Could you undo the JIRA dev list e-mails? >Okay cool - sorry about that. In

Re: Could you undo the JIRA dev list e-mails?

.apache.org > Note, that I an other moderators will now receive moderation emails until the infra ticket is fixed but others will not. I'll set up a mail filter. Chris -Original Message- From: , Chris Mattmann Date: Saturday, March 29, 2014 1:11 PM To: Patrick Wendell

Re: Could you undo the JIRA dev list e-mails?

Patrick, No problem -- at the same time realize that I and the other moderators were getting spammed by moderation emails from JIRA, so you should take that into consideration as well. Cheers, Chris -Original Message- From: Patrick Wendell Date: Saturday, March 29, 2014 11:59 AM To

[jira] [Commented] (SPARK-732) Recomputation of RDDs may result in duplicated accumulator updates

[ https://issues.apache.org/jira/browse/SPARK-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954349#comment-13954349 ] Mattmann, Chris A (388J) commented on SPARK-732: Guys I fixed thi

Re: [jira] [Commented] (SPARK-732) Recomputation of RDDs may result in duplicated accumulator updates

Guys I fixed this by adding j...@apache.org to the mailing list, no more moderation required. Cheers, Chris -Original Message- From: "ASF GitHub Bot (JIRA)" Reply-To: "dev@spark.apache.org" Date: Saturday, March 29, 2014 10:14 AM To: "dev@spark.ap