gt;
> Thanks
> Amit
>
> On Fri, May 8, 2020 at 12:59 PM Hrishikesh Mishra
> wrote:
>
>> We submit spark job through spark-submit command, Like below one.
>>
>>
>> sudo /var/lib/pf-spark/bin/spark-submit \
>> --total-executor-cores 30 \
>> --d
t; "The Internals Of" Online Books <https://books.japila.pl/>
> Follow me on https://twitter.com/jaceklaskowski
>
> <https://twitter.com/jaceklaskowski>
>
>
> On Fri, May 8, 2020 at 2:32 PM Hrishikesh Mishra
> wrote:
>
>> Thanks Jacek for quic
p-related errors that may not
> necessarily be Spark's.
>
> Pozdrawiam,
> Jacek Laskowski
>
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books <https://books.japila.pl/>
> Follow me on https://twitter.com/jaceklaskowski
>
>
d/1/spark/local/spark-e045e069-e126-4cff-9512-d36ad30ee922
On Thu, May 7, 2020 at 10:16 PM Hrishikesh Mishra
wrote:
> It's only happening for Hadoop config. The exceptions trace are different
> for each time it gets died. And Jobs run for couple hours then worker dies.
>
&g
ing to load
> configuration (XML files). Make sure they're well formed.
>
> On Thu, May 7, 2020 at 6:12 AM Hrishikesh Mishra
> wrote:
>
>> Hi
>>
>> I am getting out of memory error in worker log in streaming jobs in every
>> couple of hours. After this
Hi
I am getting out of memory error in worker log in streaming jobs in every
couple of hours. After this worker dies. There is no shuffle, no
aggression, no. caching in job, its just a transformation.
I'm not able to identify where is the problem, driver or executor. And why
worker getting dead a
the same behavior? Was there a
> reason you chose to start reading again from the beginning by using a new
> consumer group rather then sticking to the same consumer group?
>
> In your application, are you manually committing offsets to Kafka?
>
> Regards,
>
> Waleed
>
&g
Hi
Our Spark streaming job was working fine as expected (the number of events
to process in a batch). But due to some reasons, we added compaction on
Kafka topic and restarted the job. But after restart it was failing for
below reason:
org.apache.spark.SparkException: Job aborted due to stage fa
Hi
My spark stream job consumes from multiple Kafka topics. How can I process
parallely? Should I try for *spark.streaming.concurrentJobs,* but it has
some adverse effects as mentioned by the creator. Is it still valid with
Spark 2.4 and Direct Kafka Stream? What about FAIR scheduling mode, will i
