We did originally think of allowing `filter`, `map` etc on a GlobalKTable, though it would have been slightly different to what you are suggesting, i.e., we'd materialize the topic as a Store and then provide views on top of that. They'd also be global, but not materialized as physical state stores.
On Fri, 26 May 2017 at 02:01 Thomas Becker <tobec...@tivo.com> wrote: > Hey Eno, > Thanks for the response. We have considered, but not yet tried that. One > of the nice things about the GlobalKTable is that it fully "bootstraps" > before the rest of the topology is started. But if part of the topology is > itself generating the topic that backs the global table, it seems like that > would effectively break. Also, doing this obviously requires > re-materializing the data in a new topic. To be fair, the topic we are > building the table from has a bit of an unusual format, which is why I was > trying to see if anyone else would think this was useful. > > -Tommy > > ________________________________________ > From: Eno Thereska [eno.there...@gmail.com] > Sent: Thursday, May 25, 2017 12:03 PM > To: dev@kafka.apache.org > Subject: Re: GlobalKTable limitations > > Hi Thomas, > > Have you considered doing the transformations on the topic, then > outputting to another topic and then constructing the GlobalKTable from the > latter? > > The GlobalKTable has the limitations you mention since it was primarily > designed for joins only. We should consider allowing a less restrictive > interface if it makes sense. > > Eno > > > On 25 May 2017, at 14:48, Thomas Becker <tobec...@tivo.com> wrote: > > > > We need to do a series of joins against a KTable that we can't co- > > partition with the stream, so we're looking at GlobalKTable. But the > > topic backing the table is not ideally keyed for the sort of lookups > > this particular processor needs to do. Unfortunately, GlobalKTable is > > very limited in that you can only build one with the exact keys/values > > from the backing topic. I'd like to be able to perform various > > transformations on the topic before materializing the table. I'd > > envision it looking something like the following: > > > > builder.globalTable(keySerde, valueSerde, topicName) > > .filter((k, v) -> k.isFoo()) > > .map((k, v) -> new KeyValue<>(k.getBar(), v.getBaz())) > > .build(tableKeySerde, tableValueSerde, storeName); > > > > Is this something that has been considered or that others would find > > useful? > > > > -- > > > > > > Tommy Becker > > > > Senior Software Engineer > > > > O +1 919.460.4747 <(919)%20460-4747> > > > > tivo.com > > > > > > ________________________________ > > > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, copying, > or distribution of this email (or any attachments) by others is prohibited. > If you are not the intended recipient, please contact the sender > immediately and permanently delete this email and any attachments. No > employee or agent of TiVo Inc. is authorized to conclude any binding > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > Inc. may only be made by a signed written agreement. > > ________________________________ > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, copying, > or distribution of this email (or any attachments) by others is prohibited. > If you are not the intended recipient, please contact the sender > immediately and permanently delete this email and any attachments. No > employee or agent of TiVo Inc. is authorized to conclude any binding > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > Inc. may only be made by a signed written agreement. >
