On Sun, Jan 13, 2019, at 18:13, n...@afshartous.com wrote: > Thanks Colin and Mathias. > > > On Jan 12, 2019, at 8:27 PM, Matthias J. Sax <matth...@confluent.io> wrote: > > > > Thus, I would suggest to limit this KIP to the consumer only, otherwise, > > the scope will be too large and this KIP will drag on even longer. If we > > really want to add this to Kafka Streams, I expect a long and difficult > > discussion about this by itself, and thus, doing this in a follow up KIP > > (if there is any demand) seems to be the better approach. > > > > Agreed, and my intent is to limit the scope to the consumer. > > > About the starvation issue: maybe it's a bold claim, but is a potential > > starvation of a low-priority topic not intended by design if topics have
Yeah, I was thinking this as well. If you want strict priority behavior, high priority tasks should always take priority over low priority ones. > > On reflection, it would be hard to describe the semantics of an API > that tried to address starvation by temporarily disabling > prioritization, and then oscillating back and forth. > Thus I agree that it makes sense not to try and address starvation to > Mathias’ point that this is intended by design. The KIP has been > updated to reflect this by removing the second method. Yeah, I agree with that. The problem is, the actual policy you want is kind of complex. It's probably better in the application rather than in Kafka. > > Regarding incremental fetch, Colin do you have any suggestion on which > option to adopt or how to proceed ? I think it makes sense to go back to use-cases again. So far, all of the use-cases we discussed could be handled by pause and resume. So it makes sense to try to figure out what the issue with those APIs is. Are they not well-documented enough? Is there something higher-level we could build on top to make them easier to use? It would be better to wait until a user comes forward and with a case where priorities are needed, to implement them. Since then we would know more about what the API should be, etc. best, Colin
