Replaying an RDD in spark streaming to update an accumulator

2015-12-10 Thread AliGouta
I am actually running out of options. In my spark streaming application. I want to keep a state on some keys. I am getting events from Kafka. Then I extract keys from the event, say userID. When there is no events coming from Kafka I want to keep updating a counter relative to each user ID each 3 s

SparkStreaming: Updating internal variables without catching events from Kafka

2015-12-02 Thread AliGouta
I am actually looking for a way in spark streaming to keep updating an accumulator during the application life cycle. Actually, my use case is like the following: My spark streaming application is consuming events from Kafka. The event consists on a userID and some other information. On the other