Re: How to know whether I'm in the first batch of spark streaming

2016-04-21 Thread Yu Xie
- > "Courage doesn't always roar. Sometimes courage is the quiet voice at the > end of the day saying I will try again" > > > > From:Yu Xie > To:user@spark.apache.org > D

How to know whether I'm in the first batch of spark streaming

2016-04-19 Thread Yu Xie
hi spark users I'm running a spark streaming application, with concurrentJobs > 1, so maybe more than one batches could run together. Now I would like to do some init work in the first batch based on the "time" of the first batch. So even the second batch runs faster than the first batch, I still

can checkpoint and write ahead log save the data in queued batch?

2016-03-11 Thread Yu Xie
Hi spark user I am running an spark streaming app that use receiver from a pubsub system, and the pubsub system does NOT support ack. And I don't want the data to be lost if there is a driver failure, and by accident, the batches queue up at that time. I tested by generating some queued ba