Re: Filter with large key set

2020-01-20 Thread Jin Yi
Hi Fabian, Thanks for the suggestion and sorry for the late reply, as I was trying out. The Broadcast Stream is the one I am looking for! And I should make myself clear that the elements in the key set is one type only, e.g. they could be all string, all int, etc, not some are string and some are

Re: Filter with large key set

2020-01-17 Thread Fabian Hueske
Hi Eleanore, A dynamic filter like the one you need, is essentially a join operation. There is two ways to do this: * partitioning the key set and the message on the attribute. This would be done with a KeyedCoProcessFunction. * broadcasting the key set and just locally forwarding the messages. T

Filter with large key set

2020-01-15 Thread Jin Yi
Hi there, I have the following usecase: a key set say [A,B,C,] with around 10M entries, the type of the entries can be one of the types in BasicTypeInfo, e.g. String, Long, Integer etc... and each message looks like below: message: { header: A body: {} } I would like to use Flink to fi