Yes,
In my case, my StateSpec had a small partition size. I increased the
numPartitions and the problem went away. (Details of why the problem was
happening in the first place is elided.)
TL;DR
StateSpec takes a "numPartitions" which can be set to high enough number.
--
View this message in
klog or other issue to fix maybe you’ll get lucky
> too J.
>
>
>
> Cheers
>
> Iain
>
>
>
> *From:* manas kar [mailto:poorinsp...@gmail.com]
> *Sent:* 15 March 2016 14:49
> *To:* Ted Yu
> *Cc:* user
> *Subject:* [MARKETING] Re: mapwithstate Hangs with Error
lucky too ☺.
Cheers
Iain
From: manas kar [mailto:poorinsp...@gmail.com]
Sent: 15 March 2016 14:49
To: Ted Yu
Cc: user
Subject: [MARKETING] Re: mapwithstate Hangs with Error cleaning broadcast
I am using spark 1.6.
I am not using any broadcast variable.
This broadcast variable is probably used by
I am using spark 1.6.
I am not using any broadcast variable.
This broadcast variable is probably used by the state management of
mapwithState
...Manas
On Tue, Mar 15, 2016 at 10:40 AM, Ted Yu wrote:
> Which version of Spark are you using ?
>
> Can you show the code snippet w.r.t. broadcast vari
Which version of Spark are you using ?
Can you show the code snippet w.r.t. broadcast variable ?
Thanks
On Tue, Mar 15, 2016 at 6:04 AM, manasdebashiskar
wrote:
> Hi,
> I have a streaming application that takes data from a kafka topic and uses
> mapwithstate.
> After couple of hours of smoot
