Hello Nicolas, If this missing matched record issue is mainly due to the order these two streams were processed (e.g., say your corresponding changelog record was a bit late compared with the record stream's record with the same key), you can try to "hint" Kafka Streams library to give the changelog stream a bit more time ahead by specifying its timestamps using the TimestampExtractor with an earlier value against the record stream. And Kafka Streams will do a best-effort "stream synchronization" to make sure these two streams were processed at roughly the same pace based on record timestamps, which will result in records from the changelog stream to be processed in-priori to the record stream.
Guozhang On Tue, Jul 19, 2016 at 6:10 AM, Matthias J. Sax <matth...@confluent.io> wrote: > Hi Nicolas, > > your are right, it is currently not possible to get a result from a > KTable update (and this is by design). The idea is, that the KStream is > enriched with the *current state* of KTable -- thus, for each KStream > record a look-up in KTable is done. (In this sense, a KStream-KTable > join in asymmetric.) > > If you need a symmetric join (ie, lookup for both directions), you can > either use a KTable-KTable or KStream-KStream join. Not sure, if this > might work for your use case. > > -Matthias > > > On 07/19/2016 01:36 PM, Nicolas PHUNG wrote: > > Hi, > > > > I'm using Kafka 0.10.0.0 with the Confluent platform 3.0.0 > > > > I manage to join a record stream (KStream / clicks stream) with a > changelog > > stream (KTable / an entity like a campaign related to a click for > example). > > When the entity in the KTable is inserted first (and the first time of > > course) in Kafka, the record stream is processed as expected with the > join > > in a new enriched stream. This is good. > > > > My issue is when the record stream generate a record that contains a > key/id > > that hasn't been insert yet in the changelog stream/KTable. My process > > generate a record stream without information in the enriched stream. > Would > > it be possible to recall this enriched stream process once the changelog > > record on my KTable received the missing id/key ? From my understanding, > > it's not possible right now to this with a KStream-KTable join. Is there > a > > way to do something like this ? > > > > Thanks. > > > > Regards, > > Nicolas PHUNG > > > > -- -- Guozhang
