More details about executor pod which died abruptly from spark driver pod
logs
2018-07-30 19:58:41 ERROR TaskSchedulerImpl:70 - Lost executor 3 on
10.*.*.*.*: Executor lost for unknown reasons.
2018-07-30 19:58:41 WARN TaskSetManager:66 - Lost task 32.0 in stage 9.0
(TID 133, 10.10.147.6, execu
> Hello,
>
>
>
> I’m getting below error in spark driver pod logs and executor pods are
> getting killed midway through while the job is running and even driver pod
> Terminated with below intermittent error ,this happens if I run multiple
> jobs in parallel.
>
>
>
> Not able to see executor logs
Thanks Bryan for the pointer +1
Hichame
From: cutl...@gmail.com
Sent: July 30, 2018 6:40 PM
To: hich...@elkhalfi.com
Cc: hol...@pigscanfly.ca; user@spark.apache.org
Subject: Re: Use Arrow instead of Pickle without pandas_udf
Here is a link to the JIRA for adding StructType support for scalar pa
FYI
The relevant StackOverflow query on the same -
https://stackoverflow.com/questions/51610482/how-to-do-pca-with-spark-streaming-dataframe
On Tue, Jul 31, 2018 at 3:18 PM, Aakash Basu
wrote:
> Hi,
>
> Just curious to know, how can we run a Principal Component Analysis on
> streaming data in d
Hi,
Just curious to know, how can we run a Principal Component Analysis on
streaming data in distributed mode? If we can, is it mathematically valid
enough?
Have anyone done that before? Can you guys share your experience over it?
Is there any API Spark provides to do the same on Spark Streaming
Okay, sure!
On Tue, Jul 31, 2018 at 1:06 PM, Patil, Prashasth <
prashasth.pa...@spglobal.com> wrote:
> Hi Aakash,
>
> On a related note, you may want to try SparkLens for profiling which is
> quite helpful in my opinion.
>
>
>
>
>
> -Prash
>
>
>
> *From:* Aakash Basu [mailto:aakash.spark@gmai
Hi Aakash,
On a related note, you may want to try SparkLens for profiling which is quite
helpful in my opinion.
-Prash
From: Aakash Basu [mailto:aakash.spark@gmail.com]
Sent: Tuesday, July 17, 2018 12:41 PM
To: user
Subject: Query on Profiling Spark Code
Hi guys,
I'm trying to profile my
Hi,
Have you tried using spark dataframe's Pivot feature ?
-Original Message-
From: nookala [mailto:srinook...@gmail.com]
Sent: Thursday, July 26, 2018 7:33 AM
To: user@spark.apache.org
Subject: Split a row into multiple rows Java
I'm trying to generate multiple rows from a single row
I
