Re: Kubernetes Operator --allowNonRestoredState config

Thank you! On Fri, Apr 29, 2022 at 5:31 PM Gyula Fóra wrote: > Hi! > > We did not expose this as a top level flag in the spec, but you can enable > this through the flink configuration using: > > execution.savepoint.ignore-unclaimed-state: "true" > > Cheers, > Gyula > > On Fri, Apr 29, 2022 at 5

Re: Kubernetes Operator --allowNonRestoredState config

Hi! We did not expose this as a top level flag in the spec, but you can enable this through the flink configuration using: execution.savepoint.ignore-unclaimed-state: "true" Cheers, Gyula On Fri, Apr 29, 2022 at 5:26 PM Shqiprim Bunjaku wrote: > Hi all, > > I am using Flink Kubernetes Operato

Re: [DataStream API] Watermarks not closing TumblingEventTimeWindow with Reduce function

Glad to hear it! thank you for feedback ！ Ryan van Huuksloot 于2022年4月5日周二 06:26写道： > Sorry for the hassle. I ended up working with a colleague and found that > the Kafka Source had a single partition but the pipeline had a > parallelism of 4 and there was no withIdleness associated so the Waterm

Kubernetes Operator --allowNonRestoredState config

Hi all, I am using Flink Kubernetes Operator to manage my Flink applications. I have one issue when making changes in pipeline and I need to pass --allowNonRestoredState configuration, but I cannot find how I can do this using Flink Operator. Tried below method but didn't work: job: args

Re:Re: How to define event time and Watermark for intermediary joined table in FlinkSQL?

I think it's not a good idea to defining a watermark on a view, because currently the view is only a set of SQL query text in Flink , and a query should not contain a watermark definition. You can see the discussion here: https://issues.apache.org/jira/browse/FLINK-22804 Maybe you can open a jir

Re: Multiple INSERT INTO within single PyFlink job?

Ah, found it: I need to use add_insert_sql()​ in order to use multiple insert statements: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/sql/insert/#insert-statement https://aws.amazon.com/blogs/big-data/build-a-real-time-streaming-application-using-apache-flink-python

Multiple INSERT INTO within single PyFlink job?

Hi all, Is it possible to have more than one `INSERT INTO ... SELECT ...` statement within a single PyFlink job (on Flink 1.13.6)? I have a number of output tables that I create and I am trying to write to write to these within a single job, where the example SQL looks like (assume there is an

Re: Broadcast State + Stateful Operator + Async IO

Thanks a lot for your quick response! Your suggestion however would never work for our use case. Ours is a streaming system that must process 100 thousand messages per second and produce immediate results and it's simply impossible to rerun the job. Our job is a streaming job broken down into vari

flink operator sometimes cannot start jobmanager after upgrading

Hi, We found that flink operator [0] sometimes cannot start jobmanager after upgrading FlinkDeployment. We need to recreate FlinkDeployment to fix the problem. Anyone has this issue? The following is redacted log from flink operator. After status becomes MISSING, it keeps in MISSING status for at

Re: Broadcast State + Stateful Operator + Async IO

Hi Vishal If your scenario is to update the data in full every time. One idea is to rerun the job every time. For example, you have an `EnrichWithDatabaseAndWebSerivce` job, which is responsible for reading all data from a data source every time, and then joins the data with DB and Web services. E

Re: Exception Handling in ElasticsearchSink

Hi Rion, Sorry for the late reply. There should be no problems instantiating the metric in the open() function and passing down its reference through createSink and buildSinkFromRoute. I'd be happy to help in case you encounter any issues. Best, Alexander On Thu, Apr 21, 2022 at 10:49 PM Rion Wil