Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API

; data. >> >> >> >> You may find the SPIP document here [docs.google.com] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1d6egnL6WHOwWZe8MWv3m8n4PToNacdx7n-5F0iMSWwhCQ_edit&d=DwMFJg&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=UG2t14gfU8QHfoj4tUD__9bIVg1xxTM3R8GHmvMUXTU&s=rCSgQGD6L4of4oa0QxiTJ8IPaVdGlZVarhA4-QvO80Q&e=> >> . >> >> >> >> The discussion thread for the SPIP was conducted here [lists.apache.org] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_2fe82b6b86daadb1d2edaef66a2d1c4dd2f45449656098ee38c50079-40-253Cdev.spark.apache.org-253E&d=DwMFJg&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=UG2t14gfU8QHfoj4tUD__9bIVg1xxTM3R8GHmvMUXTU&s=kSJizQH7v4OHG6D7aVsLA-m0ApZxOa24CzHZv1EzLxg&e=> >> . >> >> >> >> Please vote on whether or not this proposal is agreeable to you. >> >> >> >> Thanks! >> >> >> >> -Matt Cheah >> >> >> >> >> -- >> >> Ryan Blue >> >> Software Engineer >> >> Netflix >> >> > > -- > John > -- -Dhruve Ashar

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

7;d prefer we cut another rc with the fix, > but understand the tradeoffs here. > > On Wed, Mar 20, 2019 at 10:17 AM Sean Owen wrote: > >> Is it a regression from 2.4.0? that's not the only criteria but part of >> it. >> The version link is >> https://iss

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

gt;> > >> > In order to make timely releases, we will typically not hold the >> > release unless the bug in question is a regression from the previous >> > release. That being said, if there is something which is a regression >> > that has not been correctly targeted please ping me or a committer to >> > help target the issue. >> > >> > >> > DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, >> Inc >> > >> > >> > - >> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > >> > >> > >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > Stavros Kontopoulos > > *Senior Software Engineer* > *Lightbend, Inc.* > > *p: +30 6977967274 <%2B1%20650%20678%200020>* > *e: stavros.kontopou...@lightbend.com* > > > -- -Dhruve Ashar

Checkpointing in Spark - Cleaning files and Support across app attempts

While checkpointing RDDs as a part of an application that doesn't use spark-streaming, I observed that the checkpointed files are not being cleaned up even after the application completes successfully. Is it because we assume that checkpointing would be primarily used for spark-streaming applicati

Re: Setting YARN executors' JAVA_HOME

raryPath don't appear to set JAVA_HOME (and maybe > aren't even supposed to?). > > The 1.0.1 docs > <https://spark.apache.org/docs/1.0.1/running-on-yarn.html#environment-variables> > are the last ones to reference the SPARK_YARN_USER_ENV var, afaict. > > What's the preferred way of passing YARN a custom JAVA_HOME that will be > applied to executors' containers? > > Thanks! > -- -Dhruve Ashar

Re: Welcoming Felix Cheung as a committer

>> > - >> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> -- -Dhruve Ashar

Re: Build speed

ch a person), you may not > copy or deliver this message to anyone. In such case, > you should destroy this message and kindly notify the sender by reply > email. > > > -- -Dhruve Ashar

Re: Question about equality of o.a.s.sql.Row

x.html#org.apache.spark.sql.Row > > > Pull requests for documentation welcome! > > >> (3) If (3) is expected, is there any recommendation to write code of >> equality of two Rows that have an Array or complex types (e.g. Map)? > > > Internally for tests, we usually compare the string representation of the > Row. > -- -Dhruve Ashar

Re: SBT doesn't pick resource file after clean

ting to either "compile" or "test" (probably compile in your > case). So, the simplest way to add the extra settings would be > something like: > > resourceGenerators in Compile += { > val file = //generate properties file > IO.copy(file, (resourceMan

Re: SBT doesn't pick resource file after clean

the appropriate module > > depend on the task that generates the resource file? > > > > Sorry but my knowledge of sbt doesn't really go too far. > > > > On Tue, May 17, 2016 at 11:58 AM, dhruve ashar > wrote: > >> We are trying to pick the spark versi

SBT doesn't pick resource file after clean

We are trying to pick the spark version automatically from pom instead of manually modifying the files. This also includes richer pieces of information like last commit, version, user who built the code etc to better identify the framework running. The setup is as follows : - A shell script genera