Apache Beam Summit 2024 is September 4 2024!

Hey Flink Users, Heads up! The Apache Beam Summit 2024 is happening next month( September 04, 2024) at the Google Campus in Sunnyvale, CA. It's two days packed with learning and networking with the Beam community. Key highlights: - Improving Stability for Running Python SDK with Flink Runner: im

Re: Issue in PrefetchCount

Hi ajay, When you have 3 parallelisms you will have 3 independent clients. If you want to keep prefetch count 3 you need to set setRequestedChannelMax as 1 and setParallelism 3. So All 3 clients can have one connection. Talat On Tue, May 7, 2024 at 5:52 AM ajay pandey wrote: > Hi Flink Team, >

Re: Exception in Flink 1.18 (Time should be non negative)

Hi Lasse, If there's a significant difference in the system time between Flink TaskManagers, it can lead to negative time calculations when comparing timestamps from different sources. On Mon, May 6, 2024 at 5:40 AM Lasse Nedergaard < lassenedergaardfl...@gmail.com> wrote: > Hi. > > In Flink job

Re: Evolving Flink SQL statement set and restoring from savepoints

Hi Keith, When you add a new insert statement to your EXECUTE STATEMENT you change your job graph with independent two graphs.Unfortunately, Flink doesn't currently provide a way to directly force specific UIDs for operators through configuration or SQL hints. This is primarily due to how Flink's

Re: Count(distinct) not working in beam sql

Hi, I saw this a little bit late. I implement a custom count distinct for our streaming use case. If you are looking for something close enough but not exact you can use my UDF. It uses the HyperLogLogPlus algorithm, which is an efficient and scalable way to estimate cardinality with a controlled

Watermark Alignment on Flink Runner's UnboundedSourceWrapper

Hi All, I have a stream aggregation job which reads from Kafka and writes some Sinks. When I submit my job Flink checkpoint size keeps increasing if I use unaligned checkpoint settings and it does not emit any window results. If I use an aligned checkpoint, size is somewhat under control(still bi

Local Combiner for GroupByKey on Flink Streaming jobs

Hi, I have a stream aggregation job which is running on Flink 1.13 I generate DAG by using Beam SQL. My SQL query has a TUMBLE window. Basically My pipeline reads from kafka aggregate, counts/sums some values by streamin aggregation and writes a Sink. BeamSQl uses Groupbykey for the aggregation p

Flink Kubernetes Operator Scale Issue

Hi All, We are using Flink Kubernetes Operator on our production. We have 3k+ jobs in standalone mode. But after 2.5k jobs operator getting slow. Now when we submit a job it takes 10+ minutes to the job runs. Does anyone use similar scale or more job ? Now we run as a single pod. Does operator su

Re: Task Failure Strategy for Adaptive Scheduler

WpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=BkW1L6EF7ergAVYDXCo-3Vwkpy6qjsWAz7_GD7pAR8g&m=io1uqfos6mZy0_MjhX-EMn6Kv1O-J51WEdXoLFL2-JdSmUURlkVSK9Jo06K7PXbt&s=duMv5JJ1nSCwJNlA1eG3uaopaPpgwqrBfBnbegFEb7s&e=> >>> >>> Best, >>> D. >>> >>> On

Re: Task Failure Strategy for Adaptive Scheduler

, > D. > > On Tue, Apr 11, 2023 at 4:35 AM Weihua Hu wrote: > >> Hi, >> >> AFAIK, the reactive mode always restarts the whole pipeline now. >> >> Best, >> Weihua >> >> >> On Tue, Apr 11, 2023 at 8:38 AM Talat Uyarer via user < >&

Task Failure Strategy for Adaptive Scheduler

Hi All, We use Flink 1.13 with reactive mode for our streaming jobs. When we have an issue/exception on our pipeline. Flink rescheduled all tasks. Is there any way to reschedule only task that had exceptions ? Thanks

Re: Beam SQL Alias issue while using With Clause

gt; >>>>>>>> All the rules share a common call to > RexProgramBuilder.mergePrograms, so I suspect the problem lies there. I > spent some time looking but wasn't able to find it by code inspection, it > looks like this code path is doing the right thing with names. I&#

Re: Beam SQL Alias issue while using With Clause

COLLECTION) SELECT id AS fout_int, v AS fout_string FROM tempTable >>>>>>>> WHERE >>>>>>>> id > 1 >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>

Re: Beam SQL Alias issue while using With Clause

reQuidemTest >>>>>>>> class. >>>>>>>> >>>>>>>> !use post >>>>>>>> !set outputformat mysql >>>>>>>> >>>>>>>> #Test aliases with with clause &

Re: How to submit beam python pipeline to GKE flink cluster

Hi, Do you use Flink operator or manually deployed session cluster ? Thanks On Fri, Feb 3, 2023, 4:32 AM P Singh wrote: > Hi Team, > > I have set up a flink cluster on GKE and am trying to submit a beam > pipeline with below options. I was able to run this on a local machine but > I don't unde

Re: Reducing Checkpoint Count for Chain Operator

not operator > chain. So what is the significance of checkpoint based on the granularity > of operator chain？ > > > > Best regards, > > Weijie > > > > > > Talat Uyarer 于2023年2月2日周四 02:20写道： > > Hi Weijie, > > > > Thanks for replying back. > &

Beam Portable Framework Question for Same SDK and Runner Language

Hi, I know we use the portability framework when the sdk language (python) is different from the runner's language(java) . If my runner is Java based and I want to use the portability framework for Java SDK. Is there any optimization on the Beam side rather than running two separate docker images

Re: Reducing Checkpoint Count for Chain Operator

; operator creates their own checkpoint at checkpointing time . Rather than > creating a checkpoint per operator in checkpointing time. Can I have one > checkpoint per OperatorChain? This is my question. > > > > Thanks > > > > On Wed, Feb 1, 2023 at 1:02 AM weijie guo &g

Re: Reducing Checkpoint Count for Chain Operator

o checkpoint > independently for each task, this is not supported. > > > Best regards, > > Weijie > > > Talat Uyarer via user 于2023年2月1日周三 15:34写道： > >> Hi, >> >> We have a job that is reading from kafka and writing some endpoints. The >> job does no

Reducing Checkpoint Count for Chain Operator

Hi, We have a job that is reading from kafka and writing some endpoints. The job does not have any shuffling steps. I implement it with multiple steps. Flink chained those operators in one operator in submission time. However I see all operators are doing checkpointing. Is there any way to crea

Re: Beam SQL Alias issue while using With Clause

beam_sdk_extensions_sql_impl_rule_BeamCalcRule.java-23L69&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=BkW1L6EF7ergAVYDXCo-3Vwkpy6qjsWAz7_GD7pAR8g&m=KXc2qSceL6qFbFnQ_2qUOHr9mKuc6zYY8rJTNZC8p_wTcNs4M6mHQoCuoc4JfeaA&s=KjzplEf29oFB6uivvdjixpQiArWtfV-1SXpALL-ugEM&e=>

Re: Beam SQL Alias issue while using With Clause

t;> Context: "CalcRel" is an optimized relational operation that is >>>> somewhat like ParDo, with a small snippet of a single-assignment DSL >>>> embedded in it. Calcite will choose to merge all the projects and filters >>>> into the node, an

Re: Beam SQL Alias issue while using With Clause

tes Java bytecode to directly execute the DSL. >> >> Problem: it looks like the CalcRel has output columns with aliases "id" >> and "v" where it should have output columns with aliases "id" and "value". >> >> Kenn >> >&

Beam SQL Alias issue while using With Clause

Hi All, I am using Beam 2.43 with Calcite SQL with Java. I have a query with a WITH clause and some aliasing. Looks like Beam Query optimizer after optimizing my query, it drops Select statement's aliases. Can you help me to identify where the problem is ? This is my query INFO: SQL: WITH `tempT