Hi,
I'm in the process of migrating our DStream jobs to structured streaming and
I'm looking for an advise on how to provide initial state for the
mapGroupsWithState, similarly to what DStream's mapWithState does:
stream.mapWithState(StateSpec.function(updateState
_).*initialState*(historicalStat
This advice solved the problem: "Stop having your object extend App, and
instead give it a
main method." https://issues.apache.org/jira/browse/SPARK-4170
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Broadcast-var-is-null-tp24927p24959.html
Sent from th
I've reduced the code to the code below - no streaming, no Kafka, no
checkpoint. Unfortunately the end result is the same - "broadcastVar is
null" printed in the worker log. Any suggestion on what I'm missing would be
very much appreciated !
object BroadcastTest extends App {
val logger = Logge
Hi,
Can anyone point me out to what I'm doing wrong? I've implemented a very
basic spark streaming app that uses a single broadcast variable. When it
runs locally it produces a proper output (the array I broadcast). But when
deployed on the cluster I get "broadcastVar is null". We use v 1.4.1. Her
