Hi, this JIRA is a good starting point: https://issues.apache.org/jira/browse/FLINK-3755
If you don't care about processing guarantees and you are using a stateless streaming job, you can implement a simple Kafka consumer that uses Kafka's consumer group mechanism. I recently implemented such a Kafka consumer if you want to use it as a base for your work: https://github.com/rmetzger/scratch/blob/kafka-group-consumer/src/main/java/com/dataartisans/FlinkKafkaGroupConsumer.java Regards, Robert On Mon, Aug 29, 2016 at 11:54 AM, Abhishek Agarwal <abhishc...@gmail.com> wrote: > Thanks Stephan. I understand guaranteeing exactly once semantics with the > dynamic scaling is tough. If I were to let go of the exactly once > requirement, is it not possible in current version? It would be really > great if you can point me to the JIRA tracking this work. > > On Mon, Aug 29, 2016 at 2:30 PM, Stephan Ewen <se...@apache.org> wrote: > >> Hi! >> >> There is a lot of work in progress on that feature, and it looks like you >> can expect the next version to have some upscale/downscale feature that >> maintains exactly-once semantics. >> >> Stephan >> >> >> On Mon, Aug 29, 2016 at 9:00 AM, Abhishek Agarwal <abhishc...@gmail.com> >> wrote: >> >>> Is it possible to upscale or downscale a flink application without >>> re-deploying (similar to rebalancing in storm)? >>> >>> -- >>> Regards, >>> Abhishek Agarwal >>> >>> >> > > > -- > Regards, > Abhishek Agarwal > >
