Re: [DISCUSS] KIP-349 Priorities for Source Topics

I see. That makes sense. Actually, GlobalKTable work exactly this way -- it's not just a broadcasted table, it's also a non-synchronized table. Not sure if this would work for you -- the broadcasted property might be a deal breaker for your use case. Personally, I believe there is a design space

Re: [DISCUSS] KIP-349 Priorities for Source Topics

To sum up here, I don't disagree that timestamp semantics are nice and often useful. But currently, there is no way to opt-out of these semantics. In our case the timestamp of an updated item record, which say, provides a better or corrected description, is simply preferred over the old record,

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Hi Matthias, I'm familiar with how the timestamp synchronization currently works. I also submit that it does not work for our use-case, which is the following: The table-backing topic contains records with the best available data we have for a given item. IF a record in this topic is updated, we

Re: [DISCUSS] KIP-349 Priorities for Source Topics

I am not sure if this feature would help with stream-table joins. Also note, that we recently merged a PR that improves the timestamp synchronization of Kafka Streams -- this will vastly improve the guarantees. What I don't understand: > So table records that have been updated recently will not b

Re: [DISCUSS] KIP-349 Priorities for Source Topics

For my part, a major use-case for this feature is stream-table joins. Currently, KafkaStreams does the wrong thing in some cases because the only message choosing strategy available is timestamp-based. So table records that have been updated recently will not be read until the stream records rea

Re: [DISCUSS] KIP-349 Priorities for Source Topics

I think, we can keep incremental fetch request with the design idea I described in my previous email. Of course, brokers would need to be updated and understand topic priorities, ie, we would also need to change the protocol to send topic priority information to the brokers. Thus, if somebody only

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Oh, also, I am -1 on disabling incremental fetch requests when the prioritization feature is used. Users often have performance problems that are difficult to understand when they use various combinations of features. Of course as implmentors we "know" what the right combination of features is

Re: [DISCUSS] KIP-349 Priorities for Source Topics

On Thu, Sep 6, 2018, at 05:24, Jan Filipiak wrote: > > On 05.09.2018 17:18, Colin McCabe wrote: > > Hi all, > > > > I agree that DISCUSS is more appropriate than VOTE at this point, since I > > don't remember the last discussion coming to a definite conclusion. > > > > I guess my concern is that

Re: [DISCUSS] KIP-349 Priorities for Source Topics

On Thu, Sep 6, 2018, at 20:21, Matthias J. Sax wrote: > I am still not sure how Samza's MessageChooser actually works and how > this would align with KafkaConsumer fetch requests. > > > Maybe I can give some background (conceptually); @Colin, please correct > me if I say anything wrong: > > > W

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Hi All, Thanks for all the comments on KIP-349. I agree with Mathias’ comment that topic prioritization and Message chooser are complimentary. And since the two API’s are not strongly coupled my inclination is not to add MessageChooser to KIP-349. MessageChoose could be done in a separate f

Re: [DISCUSS] KIP-349 Priorities for Source Topics

On 07.09.2018 05:21, Matthias J. Sax wrote: I am still not sure how Samza's MessageChooser actually works and how this would align with KafkaConsumer fetch requests. Maybe I can give some background (conceptually); @Colin, please correct me if I say anything wrong: When a fetch request is s

Re: [DISCUSS] KIP-349 Priorities for Source Topics

I am still not sure how Samza's MessageChooser actually works and how this would align with KafkaConsumer fetch requests. Maybe I can give some background (conceptually); @Colin, please correct me if I say anything wrong: When a fetch request is send, all assigned topic partitions of the consum

Re: [DISCUSS] KIP-349 Priorities for Source Topics

On 05.09.2018 17:18, Colin McCabe wrote: Hi all, I agree that DISCUSS is more appropriate than VOTE at this point, since I don't remember the last discussion coming to a definite conclusion. I guess my concern is that this will add complexity and memory consumption on the server side. In t

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Hi all, I agree that DISCUSS is more appropriate than VOTE at this point, since I don't remember the last discussion coming to a definite conclusion. I guess my concern is that this will add complexity and memory consumption on the server side. In the case of incremental fetch requests, we wil

Re: [DISCUSS] KIP-349 Priorities for Source Topics

On 05.09.2018 02:38, n...@afshartous.com wrote: On Sep 4, 2018, at 4:20 PM, Jan Filipiak wrote: what I meant is litterally this interface: https://samza.apache.org/learn/documentation/0.7.0/api/javadocs/org/apache/samza/system/chooser/MessageChooser.html

Re: [DISCUSS] KIP-349 Priorities for Source Topics

> On Sep 4, 2018, at 4:20 PM, Jan Filipiak wrote: > > what I meant is litterally this interface: > > https://samza.apache.org/learn/documentation/0.7.0/api/javadocs/org/apache/samza/system/chooser/MessageChooser.html > >

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Hi Nick, sorry for not beeing so helpfull. I don't quite understand what _this_ would be in your email. Is this the part in question? /interface TopicPrioritizer { List prioritize(List topicPriorities); } // //public void registerTopicPrioritizer(TopicPrioritizer topicPrioritizer);// / this

Re: [DISCUSS] KIP-349 Priorities for Source Topics

@Jan - can you comment on whether or not this is what you had in mind ? -- Nick > On Aug 30, 2018, at 10:18 AM, n...@afshartous.com wrote: > > > Just clarifying that the API below would be in addition to the API specified > in KIP-349 > > > https://cwiki.apache.org/confluence/displ

Re: [DISCUSS] KIP-349 Priorities for Source Topics

On 30.08.2018 15:17, Matthias J. Sax wrote: Nick, Would be good to understand the difference between the current approach and what Jan suggested. If we believe that the current proposal is too limited in functionality and also hard to extend later on, it might make sense to work on a more gen

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Just clarifying that the API below would be in addition to the API specified in KIP-349 https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics -- Nick > On Aug 30, 2018, at 9:57 AM, n...@afshartous.com wrote: > > Here’s an attempt at incorporating

[DISCUSS] KIP-349 Priorities for Source Topics

Changing subject back to DISCUSS. > On Aug 20, 2018, at 6:52 AM, Jan Filipiak wrote: > > Just saying that we should peek at the Samza approach, it's a much more > powerful abstraction. We can ship a default MessageChooser > that looks at the topics priority. Thanks Jan for your input. Here’

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Nick, Would be good to understand the difference between the current approach and what Jan suggested. If we believe that the current proposal is too limited in functionality and also hard to extend later on, it might make sense to work on a more generic solution from the beginning on. On the other

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Ah, sorry, yes it does. On Sun, Aug 12, 2018 at 2:58 PM wrote: > Does this clarify ? > -- > Nick > > On Aug 9, 2018, at 7:44 PM, n...@afshartous.com wrote: > > Since there are questions I changed the heading from VOTE to DISCUSS > > On Aug 8, 2018, at 9:09 PM, Matt Farmer wrote: > > s it

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Does this clarify ? -- Nick > On Aug 9, 2018, at 7:44 PM, n...@afshartous.com wrote: > > Since there are questions I changed the heading from VOTE to DISCUSS > >> On Aug 8, 2018, at 9:09 PM, Matt Farmer mailto:m...@frmr.me>> >> wrote: >> >> s it worth spelling out explicitly what the b

[DISCUSS] KIP-349 Priorities for Source Topics

Since there are questions I changed the heading from VOTE to DISCUSS > On Aug 8, 2018, at 9:09 PM, Matt Farmer wrote: > > s it worth spelling out explicitly what the behavior is when two topics > have the same priority? I'm a bit fuzzy on how we choose what topics to > consume from right now, i

Re: [DISCUSS] KIP-349 Priorities for Source Topics

> From: "Matthias J. Sax" One general question: The Jira is marked as "stream", so I am wondering what > the intended scope the KIP is because, it suggests a new consumer > API only. Can you clarify? Based on the thread in KAFKA-6690, I’ve changed the component from streams to consumer. --

Re: [DISCUSS] KIP-349 Priorities for Source Topics

On 2018/08/04 00:35:05, "Matthias J. Sax" wrote: > One general question: The Jira is marked as "stream", so I am wondering > what the intended scope the KIP is because, it suggests a new consumer > API only. Can you clarify? Based on the thread in the Jira ticket, I've changed the component

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Thanks for the KIP. One general question: The Jira is marked as "stream", so I am wondering what the intended scope the KIP is because, it suggests a new consumer API only. Can you clarify? -Matthias On 8/3/18 1:13 PM, Afshartous, Nick wrote: > > Hi All, > > This message is to initiate discuss

[DISCUSS] KIP-349 Priorities for Source Topics

Hi All, This message is to initiate discussion on a feature to add optional priorities to source topics. Please review https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics Cheers, -- Nick