The direct stream will let you do both of those things. Is there a reason you want to use receivers?
http://spark.apache.org/docs/latest/streaming-kafka-integration.html http://spark.apache.org/docs/latest/configuration.html#spark-streaming look for maxRatePerPartition On Mon, Feb 22, 2016 at 1:06 PM, vaibhavrtk1 <learnings.vaib...@gmail.com> wrote: > Hi > > I am using kafka with spark streaming 1.3.0 . When the spark application is > not running kafka is still receiving messages. When i start the application > those messages which have already been received when spark was not running > are not processed. I am using a unreliable receiver based approach. > > What can I do to process earlier messages also, which came while > application > was shut down? > > PS: If application was down for a long time can i also limit the max number > of message consumed in one batch interval? > > Regards > Vaibhav > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
