Hi Colin Thanks for the sober second thought - I actually didn't see the inconclusive parts of the DISCUSS (I must have missed them when going through) so I am grateful you highlighted these. I will have to remove my +1 in light of the issues Colin has mentioned, but I will follow the discussion more carefully.
On Thu, Jan 10, 2019 at 5:41 PM Colin McCabe <cmcc...@apache.org> wrote: > Hi all, > > Just as a quick reminder, this is not really a complete proposal. There > are a bunch of unresolved issues with this KIP. One example is how this > interacts with incremental fetch sessions. It is not mentioned anywhere in > the KIP text. Previously we discussed some approaches, but there was no > clear consensus. > > Another example is the issue of starvation. The KIP discusses "an idea" > for handling starvation, but the details are very sparse-- just a sentence > of two. At minimum we would need some kind of configuration for the > proposed "lag deltas". It's also not clear that the proposed mechanism > would work, since we don't receive lag metrics for partitions that we don't > fetch. But if we do fetch from the partitions, we may receive data, which > would cause our policy to not be strict prioties. Keep in mind, even > attempting to fetch 1 byte may cause us to read an entire message, as > described in KIP-74. > > It seems that we don't understand the potential use-cases. The only > use-case referenced by the KIP is this one, by Bala Prassanna: > > > We use Kafka to process the asynchronous events of our Document > Management > > System such as preview generation, indexing for search etc. > > The traffic gets generated via Web and Desktop Sync application. In > such > > cases, we had to prioritize the traffic from web and consume them > first. > > But this might lead to the starvation of events from sync if the > consumer > > speed is slow and the event rate is high from web. A solution to > handle > > the starvation with a timeout after which the events are consumed > normally > > for a specified period of time would be great and help us use our > > resources effectively. > > Reading this carefully, it seems that the problem is actually starvation, > not implementing priorities. Bala already implemented priorities outside > of Kafka. If you read the discussion on KAFKA-6690, Bala also makes this > comment: "We would need this in both Consumer API and Streams API." The > current KIP does not discuss adding priorities to Streams-- only to the > basic consumer API. So it seems clear that KIP-349 does not address Bala's > use-case at all. > > Stepping back a little bit, it seems like a few people have spoken up > recently asking for some way to re-order the messages they receive from the > Kafka consumer. For example, ChienHsing Wu has discussed a use-case where > he wants to receive messages in a "round robin" order. All of this is > possible by doing some local buffering and using the pause and resume > APIs. Perhaps we should consider better documenting these APIs, and adding > some examples. Or perhaps we should consider some kind of API to do > pluggable buffering on the client side. > > In any case, this needs more discussion. We need to be clear and definite > about what use cases we want to solve, and the tradeoffs we're making to > solve them. For now, I have to reiterate my -1 (binding). > > Colin > > > On Thu, Jan 10, 2019, at 10:46, Adam Bellemare wrote: > > Looks good to me then! > > > > +1 non-binding > > > > > > > > > On Jan 10, 2019, at 1:22 PM, Afshartous, Nick <nafshart...@wbgames.com> > wrote: > > > > > > > > > Hi Adam, > > > > > > > > > This change is only intended for the basic consumer API. > > > > > > > > > Cheers, > > > > > > -- > > > > > > Nick > > > > > > > > > ________________________________ > > > From: Adam Bellemare <adam.bellem...@gmail.com> > > > Sent: Sunday, January 6, 2019 11:45 AM > > > To: dev@kafka.apache.org > > > Subject: Re: [VOTE] KIP-349 Priorities for Source Topics > > > > > > Hi Nick > > > > > > Is this change only for the basic consumer? How would this affect > anything with Kafka Streams? > > > > > > Thanks > > > > > > > > >> On Jan 5, 2019, at 10:52 PM, n...@afshartous.com wrote: > > >> > > >> Bumping again for more votes. > > >> -- > > >> Nick > > >> > > >> > > >>> On Dec 26, 2018, at 12:36 PM, n...@afshartous.com wrote: > > >>> > > >>> Bumping this thread for more votes > > >>> > > >>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= > < > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= > >< > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= > < > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= > >> > > >> > > >> > > >> > > >> > > >
