Timur, there is not API to pause/resume partitions in Streams, because Streams handles/manages its internal consumer by itself.
The "batch processing KIP" is currently delayed -- but I am sure we will pick it up again. Hopefully after 0.11 got released. > So we are considering to just pause specific >> topic partitions as soon as we arrive to stop offsets for them. I am just wondering how you would do this in a fault-tolerant way (if you would have pause API)? >> 2) Assume we process multiple topics in some parallel way and want to pause >> some topics while waiting for other topics to catch up. Streams synchronizes topics on time automatically for your. So I am wondering why this does not work for you? -Matthias On 4/27/17 8:52 AM, Timur Yusupov wrote: > I see it is possible to pause specific topic partition consumption when > using KafkaConsumer directly, but looks like it is not possible when using > KafkaStreams. > > There are following use cases for that: > 1) Doing batch processing using Kafka Streams (I found > https://cwiki.apache.org/confluence/display/KAFKA/KIP-95%3A+Incremental+Batch+Processing+for+Kafka+Streams > proposal for Kafka Streams, but according to > https://issues.apache.org/jira/browse/KAFKA-4437 there is no active > development on that side). So we are considering to just pause specific > topic partitions as soon as we arrive to stop offsets for them. > > 2) Assume we process multiple topics in some parallel way and want to pause > some topics while waiting for other topics to catch up. > > Actually, the first use case is more important for us, so would be good to > know if there is a possibility or some improvements are already planned for > allowing to pause specific topic partition consumption in KafkaStream. >
signature.asc
Description: OpenPGP digital signature
