Hey Eno, Thanks for the response. We have considered, but not yet tried that. One of the nice things about the GlobalKTable is that it fully "bootstraps" before the rest of the topology is started. But if part of the topology is itself generating the topic that backs the global table, it seems like that would effectively break. Also, doing this obviously requires re-materializing the data in a new topic. To be fair, the topic we are building the table from has a bit of an unusual format, which is why I was trying to see if anyone else would think this was useful.
-Tommy ________________________________________ From: Eno Thereska [eno.there...@gmail.com] Sent: Thursday, May 25, 2017 12:03 PM To: dev@kafka.apache.org Subject: Re: GlobalKTable limitations Hi Thomas, Have you considered doing the transformations on the topic, then outputting to another topic and then constructing the GlobalKTable from the latter? The GlobalKTable has the limitations you mention since it was primarily designed for joins only. We should consider allowing a less restrictive interface if it makes sense. Eno > On 25 May 2017, at 14:48, Thomas Becker <tobec...@tivo.com> wrote: > > We need to do a series of joins against a KTable that we can't co- > partition with the stream, so we're looking at GlobalKTable. But the > topic backing the table is not ideally keyed for the sort of lookups > this particular processor needs to do. Unfortunately, GlobalKTable is > very limited in that you can only build one with the exact keys/values > from the backing topic. I'd like to be able to perform various > transformations on the topic before materializing the table. I'd > envision it looking something like the following: > > builder.globalTable(keySerde, valueSerde, topicName) > .filter((k, v) -> k.isFoo()) > .map((k, v) -> new KeyValue<>(k.getBar(), v.getBaz())) > .build(tableKeySerde, tableValueSerde, storeName); > > Is this something that has been considered or that others would find > useful? > > -- > > > Tommy Becker > > Senior Software Engineer > > O +1 919.460.4747 > > tivo.com > > > ________________________________ > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, copying, or > distribution of this email (or any attachments) by others is prohibited. If > you are not the intended recipient, please contact the sender immediately and > permanently delete this email and any attachments. No employee or agent of > TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo > Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed > written agreement. ________________________________ This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
