Re: Understanding Batch Processing Time

2015-09-02 Thread Snehal Nagmote
Hi , Thank you for your reply , Yes , I confirm , the system is stuck after job 352 finishes and before job 353 starts (shows up in the UI) I will start job again and will take jstack if I can reproduce the problem Thanks, Snehal On 2 September 2015 at 13:34, Tathagata Das wrote: > Can you

Re: Understanding Batch Processing Time

2015-09-02 Thread Tathagata Das
Can you jstack into the driver and see what is process doing after job 352? Also to confirm, the system is stuck after job 352 finishes, and before job 353 starts (shows up in the UI), right? TD On Wed, Sep 2, 2015 at 12:55 PM, Snehal Nagmote wrote: > Hi All, > > I have spark job where I read

Understanding Batch Processing Time

2015-09-02 Thread Snehal Nagmote
Hi All, I have spark job where I read data from Kafka every 5 seconds interval and query Cassandra based on Kafka data using spark Cassandra Connector , I am using spark 1.4 , Often the batch gets stuck in processing after job Id 352 . Spark takes long time to spawn job 353 where it reads from Ca