Thanks for the information Guozhang. Any recommendations for handling or working around this? It's making tests very flakey.
On 24 August 2017 at 23:48, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Ian, > > I suspect it has something to do with your specified topology, in which it > triggers the join first, then the aggregation updates. > > For example, take a look at this ticket: > https://issues.apache.org/jira/browse/KAFKA-4601 > > As from its printed topology, due to the repartition topic the join > operator may be triggered before the aggregation operator, and hence it > would cause the Counts table to return `null` indicating it has not > received a record yet. > > > Guozhang > > > On Thu, Aug 24, 2017 at 3:17 AM, Ian Duffy <i...@ianduffy.ie> wrote: > > > Hi All, > > > > I'm building a streams applications where I wish to take action on the > > input when a certain frequency of the input has been seen. > > > > At the moment the application roughly goes: > > > > frequency table = Input Stream -> groupByKey -> Count > > > > input stream with counts = leftJoin frequency table and input stream > > > > branch input stream with counts where count > some threshold send to some > > other topic, if <= some threshold send to delay processing topic. > > > > delay stream with counts = leftJoin frequency table and delay stream > > > > branch delay stream with counts where count > some threshold send to some > > other topic, if <= some threshold send to delay processing topic. > > > > I've seen intermittent failures where the frequency table is giving a > null > > on join, no matter how long I wait content coming back in on the delayed > > stream keeps getting nulls for the count when looking up the frequency > > table. > > > > Any ideas why this might be occurring? > > > > Thanks, > > > > Ian. > > > > > > -- > -- Guozhang >
