Re: Waiting for a signal on one stream to start processing on another

2023-03-09 Thread Ioannis Polyzos
If I understand this correctly *Stream A *should be a changelog stream and *Stream B *an append-only stream. Then you want to use *Stream A* as a *lookup Table *for *Stream B?* I don't think there is an out-of-the box way for saying - start reading StreamB only after I have read up to this point fr

Re: Waiting for a signal on one stream to start processing on another

2023-03-08 Thread Yuval Itzchakov
Thank you Mason and Ken for your replies The source itself isn't actually hybrid. The requirement is to materialize a stream first since the stream itself may contain updates to existing keys, changing their meanings, i.e. we can have key A go from medium importance to high importance and vice ver

Re: Waiting for a signal on one stream to start processing on another

2023-03-08 Thread Mason Chen
Hi Yuval, It seems you are trying to perform bootstrapping on a Flink job by doing the bounded read first. A good pattern to follow is to use HybridSource [1] and the docs have some examples with File and Kafka sources. The point of switching can be coordinated by the source so that you can dynami

Waiting for a signal on one stream to start processing on another

2023-03-08 Thread Yuval Itzchakov
Hi, I have a use-case where I have two streams, call them A and B. I need to consume stream A up to a certain point, and only then start processing on stream B. What could be a way to go about this?