This advice solved the problem: "Stop having your object extend App, and
instead give it a
main method." https://issues.apache.org/jira/browse/SPARK-4170
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Broadcast-var-is-null-tp24927p24959.html
Sent from th
Yes, see https://issues.apache.org/jira/browse/SPARK-4170 The reason
was kind of complicated, and the 'fix' was just to warn you against
subclassing App! yes, use a main() method.
On Tue, Oct 6, 2015 at 3:15 PM, Nick Peterson wrote:
> This might seem silly, but...
>
> Stop having your object ex
This might seem silly, but...
Stop having your object extend App, and instead give it a main method.
That's worked for me recently when I've had this issue. (There was a very
old issue in Spark related to this; it would seem like a possible
regression, if this fixes it for you.)
-- Nick
On Tue,
I've reduced the code to the code below - no streaming, no Kafka, no
checkpoint. Unfortunately the end result is the same - "broadcastVar is
null" printed in the worker log. Any suggestion on what I'm missing would be
very much appreciated !
object BroadcastTest extends App {
val logger = Logge
Hi guys,
thanks a lot for responding so quickly!
I've reduced the code to the code below - no streaming, no Kafka, no
checkpoint. Unfortunately the end result is the same. Any suggestion to
where I'm messing up would be very much appreciated !
object BroadcastTest extends App {
val logger = Log
FYI the same happens with accumulators when recovering from checkpoint. I'd
love to see this fixed somehow as the workaround (using a singleton factory in
foreachRdd to make sure the accumulators are initialized instead of null) is
really intrusive...
Sent from my iPhone
On 05 Oct 2015, at 22:
Make sure the broadcast variable works independent of the streaming
application. Then make sure it work without have
StreamingContext.getOrCreate(). That will disambiguate whether that error
is thrown when starting a new context, or when recovering a context from
checkpoint (as getOrCreate is suppo
