Is it practicle to enrich a Flink DataStream in middle operator with Flink Stateful Functions?

I indeed see the value of Flink Stateful Functions. However, if I already have a Flink Job, is it possible to enrich a datastream with it? For example, like this: I really don't see how it would fit such a purpose. But, I do see that it would be very at the end of a Flink Job, not enrichm

Re: Sorting by source event time

You can change to "order by eventTIme". And it should work. You can sort on event time, but it must be time-ascending-order without 'limit'. If you still want to a descending order, I think you can try to set the internal configuration `__table.exec.sort.non-temporal.enabled__` to true. But remem

Sorting by source event time

Hi, I have a temporary view created from a datastream. tableEnv.createTemporaryView("productDetailsView", productStream, Schema.newBuilder() .columnByMetadata("eventTime", "TIMESTAMP_LTZ(3)", "rowtime", Boolean.TRUE) .watermar

RE: Kafka source stops consuming messages from topic after some minutes

Hello, Thank you for your response, Just updating on this issue, this was not an issue on the flink job but I found out that was related to this fluentd issue (https://github.com/fluent/fluentd/issues/3614) that’s why I was not getting all logs expected, Checking the output kafka topic I see all

Loading broadcast state on BroadcastProcessFunction instantiation or open method

Hello community. Currently we have a BroadcastProcessFunction implementation that is storing the broadcast state using a MapStateDescriptor. I have a use case that needs to load the BroadcastState to perform some operation before receiving any processElement or processBroadcastElement message.

Re: Cancel a job in status INITIALIZING

Can you provide the JobManager logs for this case. It sounds odd that the job was stuck in the INITIALIZING phase. Matthias On Wed, Sep 21, 2022 at 11:50 AM Christian Lorenz via user < user@flink.apache.org> wrote: > Hi, > > > > we’re running a Flink Cluster in standalone/session mode. During a

Re: Jobmanager fails to come up if the job has an issue

Yes, the JobManager will failover in HA mode and all jobs would be recovered. On Mon, Sep 26, 2022 at 2:06 PM ramkrishna vasudevan < ramvasu.fl...@gmail.com> wrote: > Thanks @Matthias Pohl . This is informative. So > generally in a session cluster if I have more than one job and only one of > t

Re: JobManager restarts on job failure

That's a good point. I forgot about these options. You're right. Cleanup wouldn't be done in that case. So, upgrading would be a viable option as you suggested. Matthias On Mon, Sep 26, 2022 at 12:53 PM Gyula Fóra wrote: > Maybe it is a stupid question but in Flink 1.15 with the following confi

Re: Jobmanager fails to come up if the job has an issue

Thanks @Matthias Pohl . This is informative. So generally in a session cluster if I have more than one job and only one of them has this issue, still we will face the same problem? Regards Ram On Mon, Sep 26, 2022 at 4:32 PM Matthias Pohl wrote: > I see. Thanks for sharing the logs. It's rela

Re: Jobmanager fails to come up if the job has an issue

I see. Thanks for sharing the logs. It's related to a FLINK-9097 [1]. In order for the job to not be cleaned up entirely after a failure while submitting the job, the JobManager is failed fatally resulting in a failover. That's what you're experiencing. One solution is to fix the permission issue

Re: JobManager restarts on job failure

Maybe it is a stupid question but in Flink 1.15 with the following configs enabled: SHUTDOWN_ON_APPLICATION_FINISH = false SUBMIT_FAILED_JOB_ON_APPLICATION_ERROR = true I think jobmanager pod would not restart but simply go to a terminal failed state right? Gyula On Mon, Sep 26, 2022 at 12:31 P

Re: JobManager restarts on job failure

Thanks Evgeniy for reaching out to the community and Gyula for picking it up. I haven't looked into the k8s operator in much detail, yet. So, help me out if I miss something here. But I'm afraid that this is not something that would be fixed by upgrading to 1.15. The issue here is that we're recove

Re: Jobmanager fails to come up if the job has an issue

I got some logs and stack traces from our backend storage. This is not the entire log though. Can this be useful? With these set of logs messages the job manager kept restarting. Regards Ram On Mon, Sep 26, 2022 at 3:11 PM ramkrishna vasudevan < ramvasu.fl...@gmail.com> wrote: > Thank you very

Re: [FEEDBACK] Metadata Platforms / Catalogs / Lineage integration

Hi Yun Tang, Sorry for the late reply. I haven't seen any tickets related to this topic. Still think this is an important feature to have supported in Flink, would love some volunteers on this topic. Best regards, Martijn On Tue, Sep 13, 2022 at 7:47 AM Yun Tang wrote: > An interesting topic,

Re: Jobmanager fails to come up if the job has an issue

Thank you very much for the reply. I have lost the k8s cluster in this case before I could capture the logs. I will try to repro this and get back to you. Regards Ram On Mon, Sep 26, 2022 at 12:42 PM Matthias Pohl wrote: > Hi Ramkrishna, > thanks for reaching out to the Flink community. Could y

Re: Jobmanager fails to come up if the job has an issue

Hi Ramkrishna, thanks for reaching out to the Flink community. Could you share the JobManager logs to get a better understanding of what's going on? I'm wondering why the JobManager is failing when the actual problem is that the job is struggling to access a folder. It sounds like there are multipl