Re: KafkaSource, consumer assignment, and Kafka ‘stretch’ clusters for multi-DC streaming apps

Hi Andrew and Martijn, Thanks for looping me in, this is an interesting discussion! I'm trying to solve a higher level problem about Kafka topic routing/assignment with FLIP-246. The main idea is that there can exist an external service that can provide the coordination between Kafka and Flink to

Kubernetes operator assign Job ID

Hi everyone After updating kuberneter operator to version 1.2.0 noticed that it started generating jobid for all deployments. 2022-10-13 06:18:30,724 o.a.f.k.o.c.FlinkDeploymentController [INFO ][infojob/infojob] Starting reconciliation 2022-10-13 06:18:30,725 o.a.f.k.o.l.AuditUtils [IN

Re: Job uptime metric in Flink Operator managed cluster

Sorry, what I said applies to Flink 1.15+ and the savepoint upgrade mode (not stateless). In any case if there is no job manager there are no metrics... So not sure how to answer your question. Gyula On Thu, Oct 13, 2022 at 8:24 AM Meghajit Mazumdar < meghajit.mazum...@gojek.com> wrote: > Hi Gy

Re: Job uptime metric in Flink Operator managed cluster

Hi Gyula, Thanks for the prompt response. > The Flink operator currently does not delete the jobmanager pod when a deployment is suspended. Are you sure this is true ? I have re-tried this many times, but each time the pods get deleted, along with the deployment resources. Additionally, the flin

Re: allowNonRestoredState doesn't seem to be working

Hi! If you have last-state upgrade mode configured it may happen that the allowNonRestoredState config is ignored by Flink (as the last-state upgrade mechanism somewhat bypasses the regular submission). Worst case scenario, you can suspend the deployment, manually record the last checkpoint/savep

RE: Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

Yep, I agree with that, but I guess losing the first-class citizen status within Flink will make many companies currently in doubt finally adopt Java. For non-FP shops or companies without a strong command of Scala, using Java will simplify things in general and avoid some unnecessary pains (hiring

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

Hi Martijn, Maybe a bit of an off-topic, but regarding Java 17 support, will it be possible to replace POJOs with Java records in existing applications? In a project I maintain we use Lombok a lot, but with Java records we would probably stop using it (or significantly reduce its usage). Will th

allowNonRestoredState doesn't seem to be working

Hey everyone, I'm trying to redeploy an application using a savepoint. The new version of the application has a few operators with new uids and a few operators with the old uids. I'd like to keep the state for the old ones. I passed the allowNonRestoredState flag (using Apache Kubernetes Operator

Re: Question about SQL gateway

Thanks Xuyang. Jack > On Oct 12, 2022, at 8:46 AM, Xuyang wrote: > > Hi, currently I think there is no ha about gateway. When the gateway crashes, > the job about being submitted sync will be cancelled, and the async job will > continue running. When the gateway restarts, the async job could

[ANNOUNCE] Apache Flink Table Store 0.2.1 released

The Apache Flink community is very happy to announce the release of Apache Flink Table Store 0.2.1. Apache Flink Table Store is a unified storage to build dynamic tables for both streaming and batch processing in Flink, supporting high-speed data ingestion and timely data query. Please check out

Re: Flink KafkaSource still referencing deleted topic

Hi, Robert, The configuration allowNonRestoredState should be used like this: ./bin/flink run --detached --allowNonRestoredState Best, Hang Robert Cullen 于2022年10月12日周三 23:13写道： > I don't see AllowNonRestoredState in the configuration documentation. How > would it be passed to a job? On the c

Re: Job Manager getting restarted while restarting task manager

I meant your jobmanager also received a SIGTERM signal, and you would need to figure out where it comes from. To be specific, this line of log: > 2022-10-11 22:11:21,683 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - RECEIVED > SIGNAL 15: SIGTERM. Shutting down as reques

Re: Job Manager getting restarted while restarting task manager

Hi, Which deployment mode do you use? What is the Flink version? I think killing TaskManagers won't make the JobMananger restart. You can provide the whole log as an attachment to investigate. On Wed, 12 Oct 2022 at 6:01 PM, Puneet Duggal wrote: > Hi Xintong Song, > > Thanks for your immediate

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

Hi everyone, Thanks again for all your feedback. It's very much appreciated. My overall feeling is that people are not opposed to the FLIP. There is demand for adding Java 17 support before dropping the Scala APIs. Given that the proposal for actually dropping the Scala APIs would only happen wit

Re: Utilizing Kafka headers in Flink Kafka connector

Hi, You can implement a custom KafkaRecordDeserializationSchema (example https://docs.immerok.cloud/docs/cookbook/reading-apache-kafka-headers-with-apache-flink/#the-custom-deserializer) and just avoid emitting the record if the header value matches what you need. On Wed, Oct 12, 2022 at 11:04 AM

Utilizing Kafka headers in Flink Kafka connector

I have some flink applications that read streams from Kafka, now the producer side code has introduced some additional information in Kafka headers while producing records. Now I need to change my consumer-side logic to process the records if the header contains a specific value, if the header valu

Re:Question about SQL gateway

Hi, currently I think there is no ha about gateway. When the gateway crashes, the job about being submitted sync will be cancelled, and the async job will continue running. When the gateway restarts, the async job could be found by gateway. BTW, the work about ha is in continuous progress. At 20

Re: Flink KafkaSource still referencing deleted topic

I don't see AllowNonRestoredState in the configuration documentation. How would it be passed to a job? On the command line like this: ./bin/flink run --detached -Dallownonrestoredstate=true ... On Tue, Oct 4, 2022 at 4:52 PM Martijn Visser wrote: > Hi Mason, > > Definitely! Feel free to open a

Re: Job uptime metric in Flink Operator managed cluster

Hello! The Flink operator currently does not delete the jobmanager pod when a deployment is suspended. This way the rest api stay available but no other resources are consumed (taskmanagers are deleted) When you delete the FlinkDeployment resource completely, then the jobmanager deployment is also

Job uptime metric in Flink Operator managed cluster

Hello, I recently deployed a Flink Operator in Kubernetes and wrote a simple FlinkDeployment CRD to run it in application mode following this . I noticed that, even after I edited the CRD and marked the sp

回复: [DISCUSS] Reverting sink metric name changes made in 1.15

Thanks for raising the discussion, Qingsheng, +1 on reverting the breaking changes. +1 for making a unified and clearer metric definition in Flink 2.0 Best, Hang 发件人: Jing Ge 发送时间: 2022年10月12日 19:20 收件人: Qingsheng Ren 抄送: Chesnay Schepler ; dev ; user ; Martij

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Hi Qingsheng, Just want to make sure we are on the same page. Are you suggesting switching the naming between "numXXXSend" and "numXXXOut" or reverting all the changes we did with FLINK-26126 and FLINK-26492? For the naming switch, please pay attention that the behaviour has been changed since we

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

As a supplement, considering it could be a big reconstruction redefining internal and external traffic and touching metric names in almost all operators, this requires a lot of discussions and we might do it finally in Flink 2.0. I think compatibility is a bigger blocker in front of us, as the outp

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Thanks Chesnay for the reply. +1 for making a unified and clearer metric definition distinguishing internal and external data transfers. As you described, having IO in operators is quite common such as dimension tables in Table/SQL API. This definitely deserves a FLIP and an overall design. Howeve

Re: Job Manager getting restarted while restarting task manager

Hi Xintong Song, Thanks for your immediate reply. Yes, I do restart task manager via kill command and then flink restart because I have seen cases where simple flink restart does not pickup the latest configuration. But what I am confused about is why killing the task manager process and then r

Re: Flink falls back on to kryo serializer for GenericTypes

There's no alternative to Kryo for generic types, apart from implementing your Flink serializer (but technically at that point the type is no longer treated as a generic type). enableForAvro only forces Avro to be used for POJO types. On 11/10/2022 09:29, Sucheth S wrote: Hello, How to avoid

fail to mount hadoop-config-volume when using flink-k8s-operator

Hi, community: I'm using flink-k8s-operator v1.2.0 to deploy flink job. And the "HADOOP_CONF_DIR" environment variable was setted in the image that i buiilded from flink:1.15. I found the taskmanager pod was trying to mount a volume named "hadoop-config-volume" from configMap. But the config

Re: videos Flink Forward San Francisco 2022

Thanks for your open feedback Jun - I appreciate it Very best wishes from Basel Günter On 11.10.22 18:12, Jun Qin wrote: Hi Totally agree, rest assured that it was some venue limitations and some post-pandemic organizational challenges that meant no videos this year. Thanks a lot for the fe