Re: Reporting serialized task size after task broadcast change?

Hmm, well I can't find it now, must have been hallucinating. Do you know off the top of your head where I'd be able to find the size to log it? On Thu, Sep 11, 2014 at 6:33 PM, Reynold Xin wrote: > I didn't know about that > > On Thu, Sep 11, 2014 at 6:29 PM, Sand

Re: Spark authenticate enablement

Hi Jun, I believe that's correct that Spark authentication only works against YARN. -Sandy On Thu, Sep 11, 2014 at 2:14 AM, Jun Feng Liu wrote: > Hi, there > > I am trying to enable the authentication on spark on standealone model. > Seems like only SparkSubmit load the properties from spark-d

A couple questions about shared variables

Hey All, A couple questions came up about shared variables recently, and I wanted to confirm my understanding and update the doc to be a little more clear. *Broadcast variables* Now that tasks data is automatically broadcast, the only occasions where it makes sense to explicitly broadcast are: *

Re: hash vs sort shuffle

Thanks for the heads up Cody. Any indication of what was going wrong? On Mon, Sep 22, 2014 at 7:16 AM, Cody Koeninger wrote: > Just as a heads up, we deployed 471e6a3a of master (in order to get some > sql fixes), and were seeing jobs fail until we set > > spark.shuffle.manager=HASH > > I'd be

Re: A couple questions about shared variables

) > > Best, > > -- > Nan Zhu > > On Sunday, September 21, 2014 at 1:10 AM, Matei Zaharia wrote: > > Hey Sandy, > > On September 20, 2014 at 8:50:54 AM, Sandy Ryza (sandy.r...@cloudera.com) > wrote: > > Hey All, > > A couple questions came up about sh

Re: A couple questions about shared variables

Filed https://issues.apache.org/jira/browse/SPARK-3642 for documenting these nuances. -Sandy On Mon, Sep 22, 2014 at 10:36 AM, Nan Zhu wrote: > I see, thanks for pointing this out > > > -- > Nan Zhu > > On Monday, September 22, 2014 at 12:08 PM, Sandy Ryza wrote: > >

Re: spark_classpath in core/pom.xml and yarn/porm.xml

Hi Ye, I think git blame shows me because I fixed the formatting in core/pom.xml, but I don't actually know the original reason for setting SPARK_CLASSPATH there. Do the tests run OK if you take it out? -Sandy On Thu, Sep 25, 2014 at 1:59 AM, Ye Xianjin wrote: > hi, Sandy Ryza:

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

To perhaps restate what some have said, Maven is by far the most common build tool for the Hadoop / JVM data ecosystem. While Maven is less pretty than SBT, expertise in it is abundant. SBT requires contributors to projects in the ecosystem to learn yet another tool. If we think of Spark as a pr

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

@patrick - It seems like my point about being able to inherit the root pom was addressed and there's a way to handle this. The larger point I meant to make is that Maven is by far the most common build tool in projects that are likely to share contributors with Spark. I personally know 10 people

[DISCUSS] SPIP: Declarative Pipelines

Hi all – starting a discussion thread for a SPIP that I've been working on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA ] [Doc ]. The SPIP

[VOTE] SPIP: Declarative Pipelines

We started to get some votes on the discussion thread, so I'd like to move to a formal vote on adding support for declarative pipelines. *Discussion thread: * https://lists.apache.org/thread/lsv8f829ps0bog41fjoqc45xk7m574ly *SPIP:* https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHF

Re: [DISCUSS] SPIP: Declarative Pipelines

uted via separate shell command ? > As a background Databricks imposes similar limitation where as you cannot > run normal Spark code and DLT on the same cluster for some reason and > forces to use two clusters increasing the cost and latency. > > On Sat, 5 Apr 2025 at 23:03, Sandy Ryz

Re: [VOTE][RESULT] SPIP: Declarative Pipelines

you are a PMC member of > Apache Spark, please find yourself from the phonebook link. > > > > Thanks! > > > > On Sat, Apr 12, 2025 at 11:30 PM Sandy Ryza wrote: > >> > >> The vote passes with 30 +1s (15 binding +1s) and no -1s. > >> Thanks t

[VOTE][RESULT] SPIP: Declarative Pipelines

The vote passes with 30 +1s (15 binding +1s) and no -1s. Thanks to all who helped with the vote! (* = binding) +1: Sem Rishab Joshi Huaxin Gao (*) Jules Damji Reynold Xin (*) DB Tsai (*) Michael Armbrust (*) Peter Toth L.C. Hsieh (*) Chao Sun (*) Denny Lee Martin Grund Gengliang Wang (*) Mich Tal

Re: [VOTE] Release Apache Spark Connect Swift Client 0.3.0 (RC1)

+1 (non-binding) On Mon, Jun 2, 2025 at 7:20 AM Dongjoon Hyun wrote: > +1 > > Dongjoon > > On 2025/06/02 13:13:45 "Rozov, Vlad" wrote: > > +1 (non-binding) > > > > Thank you, > > > > Vlad > > > > On Jun 1, 2025, at 7:21 PM, Wenchen Fan wrote: > > > > +1 > > > > On Mon, Jun 2, 2025 at 9:55 AM Yu

Re: [VOTE] SPIP: Real-Time Mode in Apache Spark Structured Streaming

+1 (non-binding) On Mon, Jun 2, 2025 at 7:34 AM Chao Sun wrote: > +1 > > On Mon, Jun 2, 2025 at 7:31 AM Jungtaek Lim > wrote: > >> +1 (non-binding) >> >> On Mon, Jun 2, 2025 at 11:09 PM Wenchen Fan wrote: >> >>> +1 >>> >>> On Mon, Jun 2, 2025 at 8:55 PM Peter Toth wrote: >>> +1

Re: [VOTE] Release Spark 4.1.0-preview1 (RC1)

+1 (non-binding) On Wed, Jul 9, 2025 at 6:57 AM Wenchen Fan wrote: > +1 > > On Wed, Jul 9, 2025 at 1:16 AM Kousuke Saruta wrote: > >> +1 >> >> 2025年7月9日(水) 2:12 Rozov, Vlad : >> >>> +1 (non-binding) >>> >>> >>> >>> Thank you, >>> >>> >>> >>> Vlad >>> >>> >>> >>> *From: *Dongjoon Hyun >>> *Date

Re: [VOTE] SPIP: Monthly preview release

+1 (non-binding) On Thu, Jul 3, 2025 at 6:47 AM Jules Damji wrote: > +1 (non-binding) > — > Sent from my iPhone > Pardon the dumb thumb typos :) > > > On Jul 2, 2025, at 11:44 PM, L. C. Hsieh wrote: > > > > +1 > > > >> On Wed, Jul 2, 2025 at 9:38 PM Hyukjin Kwon > wrote: > >> > >> Hi all, > >

<    1   2