Thanks for the summary Colin. One remark from my side: I have my doubts that topic priorities make sense for Kafka Streams (at least not for the DSL).
Thus, I would suggest to limit this KIP to the consumer only, otherwise, the scope will be too large and this KIP will drag on even longer. If we really want to add this to Kafka Streams, I expect a long and difficult discussion about this by itself, and thus, doing this in a follow up KIP (if there is any demand) seems to be the better approach. About the starvation issue: maybe it's a bold claim, but is a potential starvation of a low-priority topic not intended by design if topics have priorities? Thus, do we really need to address/fix this? To me, it seems the topic priorities plus avoiding starvation for low-priority topics is a contradiction. -Matthias On 1/11/19 7:00 AM, Adam Bellemare wrote: > Hi Colin > > Thanks for the sober second thought - I actually didn't see the > inconclusive parts of the DISCUSS (I must have missed them when going > through) so I am grateful you highlighted these. I will have to remove my > +1 in light of the issues Colin has mentioned, but I will follow the > discussion more carefully. > > > > On Thu, Jan 10, 2019 at 5:41 PM Colin McCabe <cmcc...@apache.org> wrote: > >> Hi all, >> >> Just as a quick reminder, this is not really a complete proposal. There >> are a bunch of unresolved issues with this KIP. One example is how this >> interacts with incremental fetch sessions. It is not mentioned anywhere in >> the KIP text. Previously we discussed some approaches, but there was no >> clear consensus. >> >> Another example is the issue of starvation. The KIP discusses "an idea" >> for handling starvation, but the details are very sparse-- just a sentence >> of two. At minimum we would need some kind of configuration for the >> proposed "lag deltas". It's also not clear that the proposed mechanism >> would work, since we don't receive lag metrics for partitions that we don't >> fetch. But if we do fetch from the partitions, we may receive data, which >> would cause our policy to not be strict prioties. Keep in mind, even >> attempting to fetch 1 byte may cause us to read an entire message, as >> described in KIP-74. >> >> It seems that we don't understand the potential use-cases. The only >> use-case referenced by the KIP is this one, by Bala Prassanna: >> >> > We use Kafka to process the asynchronous events of our Document >> Management >> > System such as preview generation, indexing for search etc. >> > The traffic gets generated via Web and Desktop Sync application. In >> such >> > cases, we had to prioritize the traffic from web and consume them >> first. >> > But this might lead to the starvation of events from sync if the >> consumer >> > speed is slow and the event rate is high from web. A solution to >> handle >> > the starvation with a timeout after which the events are consumed >> normally >> > for a specified period of time would be great and help us use our >> > resources effectively. >> >> Reading this carefully, it seems that the problem is actually starvation, >> not implementing priorities. Bala already implemented priorities outside >> of Kafka. If you read the discussion on KAFKA-6690, Bala also makes this >> comment: "We would need this in both Consumer API and Streams API." The >> current KIP does not discuss adding priorities to Streams-- only to the >> basic consumer API. So it seems clear that KIP-349 does not address Bala's >> use-case at all. >> >> Stepping back a little bit, it seems like a few people have spoken up >> recently asking for some way to re-order the messages they receive from the >> Kafka consumer. For example, ChienHsing Wu has discussed a use-case where >> he wants to receive messages in a "round robin" order. All of this is >> possible by doing some local buffering and using the pause and resume >> APIs. Perhaps we should consider better documenting these APIs, and adding >> some examples. Or perhaps we should consider some kind of API to do >> pluggable buffering on the client side. >> >> In any case, this needs more discussion. We need to be clear and definite >> about what use cases we want to solve, and the tradeoffs we're making to >> solve them. For now, I have to reiterate my -1 (binding). >> >> Colin >> >> >> On Thu, Jan 10, 2019, at 10:46, Adam Bellemare wrote: >>> Looks good to me then! >>> >>> +1 non-binding >>> >>> >>> >>>> On Jan 10, 2019, at 1:22 PM, Afshartous, Nick <nafshart...@wbgames.com> >> wrote: >>>> >>>> >>>> Hi Adam, >>>> >>>> >>>> This change is only intended for the basic consumer API. >>>> >>>> >>>> Cheers, >>>> >>>> -- >>>> >>>> Nick >>>> >>>> >>>> ________________________________ >>>> From: Adam Bellemare <adam.bellem...@gmail.com> >>>> Sent: Sunday, January 6, 2019 11:45 AM >>>> To: dev@kafka.apache.org >>>> Subject: Re: [VOTE] KIP-349 Priorities for Source Topics >>>> >>>> Hi Nick >>>> >>>> Is this change only for the basic consumer? How would this affect >> anything with Kafka Streams? >>>> >>>> Thanks >>>> >>>> >>>>> On Jan 5, 2019, at 10:52 PM, n...@afshartous.com wrote: >>>>> >>>>> Bumping again for more votes. >>>>> -- >>>>> Nick >>>>> >>>>> >>>>>> On Dec 26, 2018, at 12:36 PM, n...@afshartous.com wrote: >>>>>> >>>>>> Bumping this thread for more votes >>>>>> >>>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= >> < >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= >>> < >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= >> < >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= >>>> >>>>> >>>>> >>>>> >>>>> >>> >> >
signature.asc
Description: OpenPGP digital signature
