Querying a database in a page able manner

2024-02-02 Thread Lasse Nedergaard
Hi. I have a case where I consume queries from a Kafka topic. I have to execute the queries against a database in page able manner. I would like to send each result page downstream as the query can return millions of rows and I can’t/don’t want to wait until all rows are returned. Normally I w

Re: Re: Case for a Broadcast join + hint in Flink Streaming SQL

2024-02-02 Thread David Anderson
I've seen enough demand for a streaming broadcast join in the community to justify a FLIP -- I think it's a good idea, and look forward to the discussion. David On Fri, Feb 2, 2024 at 6:55 AM Feng Jin wrote: > +1 a FLIP for this topic. > > > Best, > Feng > > On Fri, Feb 2, 2024 at 10:26 PM Mart

Re: Re: Case for a Broadcast join + hint in Flink Streaming SQL

2024-02-02 Thread Feng Jin
+1 a FLIP for this topic. Best, Feng On Fri, Feb 2, 2024 at 10:26 PM Martijn Visser wrote: > Hi, > > I would definitely expect a FLIP on this topic before moving to > implementation. > > Best regards, > > Martijn > > On Fri, Feb 2, 2024 at 12:47 PM Xuyang wrote: > >> Hi, Prabhjot. >> >> IIUC,

Re: Re: Case for a Broadcast join + hint in Flink Streaming SQL

2024-02-02 Thread Martijn Visser
Hi, I would definitely expect a FLIP on this topic before moving to implementation. Best regards, Martijn On Fri, Feb 2, 2024 at 12:47 PM Xuyang wrote: > Hi, Prabhjot. > > IIUC, the main reasons why the community has not previously considered > supporting join hints only in batch mode are as

Re:Re: Case for a Broadcast join + hint in Flink Streaming SQL

2024-02-02 Thread Xuyang
Hi, Prabhjot. IIUC, the main reasons why the community has not previously considered supporting join hints only in batch mode are as follows: 1. In batch mode, multiple join type algorithms were implemented quite early on, and 2. Stream processing represents a long-running scenario, and it i

Re:sink upsert materializer in SQL job

2024-02-02 Thread Xuyang
Hi, Maj. > 1. Does the materializer support jobs containing different types of joins > (more specifically regular and temporal joins)? > 2. Does the materializer support different types of input connectors: kafka > with both debezium-avro-confluent and avro-confluent formats and upsert-kafka

Re: Jobmanager restart after it has been requested to stop

2024-02-02 Thread Yang Wang
If you could find the "Deregistering Flink Kubernetes cluster, clusterId" in the JobManager log, then it is not the expected behavior. Having the full logs of JobManager Pod before restarted will help a lot. Best, Yang On Fri, Feb 2, 2024 at 1:26 PM Liting Liu (litiliu) via user < user@flink.a