Re: How does Spark set task indexes?

gt; On Tue, May 24, 2016 at 1:00 PM, Adrien Mogenet < > adrien.moge...@contentsquare.com> wrote: > >> Hi, >> >> I'm wondering how Spark is setting the "index" of task? >> I'm asking this question because we have a job that constantly fails a

How does Spark set task indexes?

try to go further in our understanding on how does Spark behaves. We're using Spark 1.5.2, scala 2.11, on top of hadoop 2.6.0 -- *Adrien Mogenet* Head of Backend/Infrastructure adrien.moge...@contentsquare.com http://www.contentsquare.com 50, avenue Montaigne - 75008 Paris

Re: How to add an accumulator for a Set in Spark

or-for-a-Set-in-Spark-tp26510p26514.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional

Re: df.partitionBy().parquet() java.lang.OutOfMemoryError: GC overhead limit exceeded

(CoarseGrainedSchedulerBackend.scala:283) >> at >> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:180) >> at >> org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:439) >> at >> org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1439) >> at >> org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1724) >> at >> org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185) >> at org.apache.spark.SparkContext.stop(SparkContext.scala:1723) >> at >> org.apache.spark.SparkContext$$anonfun$3.apply$mcV$sp(SparkContext.scala:587) >> at >> org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:264) >> at >> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:234) >> at >> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234) >> at >> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234) >> at >> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699) >> at >> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:234) >> at >> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234) >> at >> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234) >> at scala.util.Try$.apply(Try.scala:161) >> at >> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:234) >> at >> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:216) >> at >> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) >> Caused by: akka.pattern.AskTimeoutException: >> Recipient[Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1432624242]] >> had already been terminated. >> at >> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:132) >> at >> org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:307) >> ... 23 more >> >> >> -- >> Donald Drake >> Drake Consulting >> http://www.drakeconsulting.com/ >> https://twitter.com/dondrake <http://www.MailLaunder.com/> >> 800-733-2143 >> > > > > -- > Donald Drake > Drake Consulting > http://www.drakeconsulting.com/ > https://twitter.com/dondrake <http://www.MailLaunder.com/> > 800-733-2143 > > > -- *Adrien Mogenet* Head of Backend/Infrastructure adrien.moge...@contentsquare.com (+33)6.59.16.64.22 http://www.contentsquare.com 50, avenue Montaigne - 75008 Paris

Re: [POWERED BY] Please add our organization

wiki. > > On Wed, Dec 2, 2015 at 10:53 AM, Adrien Mogenet > wrote: > > Hi folks, > > > > You're probably busy, but any update on this? :) > > > > > > On 16 November 2015 at 16:04, Adrien Mogenet > > wrote: > >> > >> Name

Re: [POWERED BY] Please add our organization

Hi folks, You're probably busy, but any update on this? :) On 16 November 2015 at 16:04, Adrien Mogenet < adrien.moge...@contentsquare.com> wrote: > Name: Content Square > URL: http://www.contentsquare.com > > Description: > We use Spark to regularly read raw data,

[POWERED BY] Please add our organization

Name: Content Square URL: http://www.contentsquare.com Description: We use Spark to regularly read raw data, convert them into Parquet, and process them to create advanced analytics dashboards: aggregation, sampling, statistics computations, anomaly detection, machine learning. -- *Adrien

Re: Split content into multiple Parquet files

eAsHadoopFile? > > Cheng > > > On 9/8/15 2:34 PM, Adrien Mogenet wrote: > > Hi there, > > We've spent several hours to split our input data into several parquet > files (or several folders, i.e. > /datasink/output-parquets//foobar.parquet), based on a > low

Split content into multiple Parquet files

th Parquet files. The only working solution so far is to persist the RDD and then loop over it N times to write N files. That does not look acceptable... Do you guys have any suggestion to do such an operation? -- *Adrien Mogenet* Head of Backend/Infrastructure adrien.moge...@contentsquar

Re: How to determine the value for spark.sql.shuffle.partitions?

erage/maximum record size >>4. cache configuration >>5. shuffle configuration >>6. serialization >>7. etc? >> >> Any general best practices? >> >> Thanks! >> >> Romi K. >> > > -- *Adrien Mogenet* Head of Backend/Infrastructure adrien.moge...@contentsquare.com (+33)6.59.16.64.22 http://www.contentsquare.com 50, avenue Montaigne - 75008 Paris

Re: Parquet partitioning for unique identifier

ash value and add it as a separate column, >> but it doesn't sound right to me. Is there any other ways I can try ? >> >> Regards, >> -- >> Kohki Nishio >> > -- *Adrien Mogenet* Head of Backend/Infrastructure adrien.moge...@contentsquare.com (+33)6.59.16.64.22 http://www.contentsquare.com 50, avenue Montaigne - 75008 Paris

Unable to understand error “SparkListenerBus has already stopped! Dropping event …”

is totally non-deterministic and I can't reproduce this, probably due to the asynchronous nature and my lack of understand on how/when stop() is supposed to be called. Any idea? Best, -- *Adrien Mogenet* Head of Backend/Infrastructure adrien.moge...@contentsquare.com (+33)6.59.16.64.22 http://w