Re: Better way to clean up state when connect

Thanks again Xingcan! Appreciate your help! On Tue, May 15, 2018, 9:31 PM Xingcan Cui wrote: > Hi Chengzhi, > > more details about partitioning mechanisms can be found at > https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/#physical-partitioning > . > > Best, > Xingcan

Re: Better way to clean up state when connect

Hi Chengzhi, more details about partitioning mechanisms can be found at https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/#physical-partitioning . Best, Xingcan > On M

Re: Better way to clean up state when connect

Hi Xingcan, Thanks a lot for providing your inputs on the possible solutions here. Can you please clarify on how to broadcasted in Flink? Appreciate your help!! Best, Chengzhi On Tue, May 15, 2018 at 10:22 AM, Xingcan Cui wrote: > Hi Chengzhi, > > currently, it's impossible to process both a s

Re: Better way to clean up state when connect

Hi Chengzhi, currently, it's impossible to process both a stream and a (dynamically updated) dataset in a single job. I'll provide you with some workarounds, all of which are based on that the file for active test names is not so large. (1) You may define your own stream source[1] which should

Re: Better way to clean up state when connect

Hi Xingcan, Thanks for your response, to give your more background about my use case, I have Stream B with some split test name, and Stream A will be the actual test. I want to have Stream A connect to Stream B to figure out whether this test is still active or not. I am not sure this is the right

Re: Better way to clean up state when connect

Hi Chengzhi, you said the Stream B which comes from a file will be updated unpredictably. I wonder if you could share more about how to judge an item (from Stream A I suppose) is not in the file and what watermark generation strategy did you choose? Best, Xingcan > On May 12, 2018, at 12:48 A

Better way to clean up state when connect

Hi there, I have a use case to check for active ID, there are two streams and I connect them: one has actual data (Stream A) and the other one is for lookup purpose (Stream B), I am getting Stream B as a file which includes all active ID, so inactive ID would not be show up on this list. I tried t