Re: Communication between two queries

No worries and great idea! I will play around with it and see what I manage to do. Cheers! Den tis 17 mars 2020 kl 15:59 skrev Piotr Nowojski : > Ops, sorry there was a misleading typo/auto correction in my previous > e-mail. Second sentence should have been: > > > First of all you would have to

Re: Communication between two queries

Ops, sorry there was a misleading typo/auto correction in my previous e-mail. Second sentence should have been: > First of all you would have to use event time semantic for consistent results Piotrek > On 17 Mar 2020, at 14:43, Piotr Nowojski wrote: > > Hi, > > Yes, you are looking in the ri

Re: Communication between two queries

Hi, Yes, you are looking in the right directions with the watermarks. First of all you would have to use event time semantic for constant results. With processing time everything would be simpler, but it would be more difficult to reason about the results (your choice). Secondly, you would hav

Re: Communication between two queries

Hi Piotr! Continuing with my scenario, since both of the queries will share the same sink, I've realized that some issues will appear when I switch queries. Especially with regards to stateful operators, e.g aggregation. Let me provide an example: So, let say that both of the queries ingest a seq

Re: Communication between two queries

Hi, Let us know if something doesn’t work :) Piotrek > On 16 Mar 2020, at 08:42, Mikael Gordani wrote: > > Hi, > I'll try it out =) > > Cheers! > > Den mån 16 mars 2020 kl 08:32 skrev Piotr Nowojski >: > Hi, > > In that case you could try to implement your `Fil

Re: Communication between two queries

Hi, I'll try it out =) Cheers! Den mån 16 mars 2020 kl 08:32 skrev Piotr Nowojski : > Hi, > > In that case you could try to implement your `FilterFunction` as two input > operator, with broadcast control input, that would be setting the > `global_var`. Broadcast control input can be originating

Re: Communication between two queries

Hi, In that case you could try to implement your `FilterFunction` as two input operator, with broadcast control input, that would be setting the `global_var`. Broadcast control input can be originating from some source, or from some operator. Piotrek > On 13 Mar 2020, at 15:47, Mikael Gordani

Re: Communication between two queries

Hi Piotr! Thanks for your response, I'll try to explain what I'm trying to achieve in more detail: Essentially, If I've two queries, in which has the same operators and runs in the same task, I would want to figure out some way of controlling the ingestion from *a source* to the respective queries

Re: Communication between two queries

Hi, Could you explain a bit more what are you trying to achieve? One problem that pops into my head is that currently in Flink Streaming (it is possible for processing bounded data), there is no way to “not ingest” the data reliably in general case, as this might deadlock the upstream operator

Communication between two queries

Hello everyone! So essentially, I've two identical queries (q1 and q2) running in parallel (Streams). I'm trying to activate the ingestion of data to q2 based on what is processed in q1. E.g say that we want to start ingesting data to q2 when a tuple with timestamp > 5000 appears in q1. The queri