Re: Query on Metrics | Cumulative Metrics via Prometheus Reporter

Hello, Please suggest. Regards, Bhawana On Wed, Aug 25, 2021 at 5:07 PM bhawana gupta wrote: > Hi, > > We are using prometheus reporter to fetch metrics (flink 1.12.1 on k8s > environment). We were facing an issue since the prometheus reporter has > various targets corresponding to each job &

Re: Flink SQL: Custom exception handling External

Hi! You can open a JIRA ticket for this feature. However from my perspective this feature should only be added to some specific connectors (mostly message queues) and formats. You might want to attach a list of proposed connectors and formats in that ticket. Chong Yun Long 于2021年8月25日周三 下午5:46写道

Re: 【Announce】Zeppelin 0.10.0 is released, Flink on Zeppelin Improved

Thanks Jeff for the great work ! Best, Leonard > 在 2021年8月25日，22:48，Jeff Zhang 写道： > > Hi Flink users, > > We (Zeppelin community) are very excited to announce Zeppelin 0.10.0 is > officially released. In this version, we made several improvements on Flink > interpreter. Here's the main fe

Re: 1.9 to 1.11 Managed Memory Migration Questions

Hi! Why does this ~30% memory reduction happen? I don't know how memory is calculated in Flink 1.9 but this 1.11 memory allocation result is reasonable. This is because managed memory, network memory and JVM overhead memory in 1.11 all has their default sizes or fractions (managed memory 40%, ne

Identify metrics belonging to the "same" task manager in kubernetes

Hi,  We have multiple flink clusters running in kubernetes. We plan to enable prometheus on these clusters. Looks like flink metrics emitted are of the format: "flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Time{host="10_244_2_6",tm_id="10_244_2_6:6122_2e3d7a",} 65.0" 1.  Si

Re: Not able to avoid Dynamic Class Loading

Hi! What Flink version are you using? In current Flink code base FlinkKafkaConsumer does not contain fields related to Avro. Jars in usrlib has a higher priority to be loaded than jars in lib. So if there is another FlinkKafkaConsumer class in your user jar then it might affect class loading and

1.9 to 1.11 Managed Memory Migration Questions

Hi folks, We're about half way complete in migrating our YARN batch processing applications from Flink 1.9 to 1.11, and are currently tackling the memory configuration migrations. Our test application's sink failed with the following exception while writing to HDFS: Caused by: java.lang.OutOf

Re: NullPointerException when using KubernetesHaServicesFactory

Can you check what is the output of those commands $ id $ ls -la $FLINK_HOME/plugins/s3-fs-presto/ jonas eyob schrieb am Mi. 25. Aug. 2021 um 16:17: > The exception is showing up both in TM and JM > > This however seemed only to appear when running on my local kubernetes > setup. > > I'd also

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

Thanks again David, I've spun up a JIRA issue for the ticket while I work on getting things into the proper state. If someone with the appropriate privileges could assign it to me, I'd be appreciative. I'll likely need some assistance at a few po

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

AFAIK there are currently no other sources in Flink that can treat "other sources" / "destination" as data. Most complete generic work on this topic that I'm aware of are Splittable DoFn based IOs in Apache Beam. I think the best module for the contribution would be "elasticsearch-base", because t

Flink KafkaConsumer metrics in DataDog

Hello , We have enabled DataDogHTTPReporter to fetch metrics on flink v1.13.1 running on kubernetes. The metric flink.operator.KafkaConsumer.records_lag_max is not displaying accurate values. It also displays 0 most of the time and when it does fetch a value, it seems to be wrong when I compare th

Not able to avoid Dynamic Class Loading

Hi all, I'm trying to avoid dynamic class loading my user code [0] due to a suspected classloading leak, but when I put my application jar into /lib instead of /usrlib, I run into the following error: ``` The main method caused an error: The implementation of the FlinkKafkaConsumer is not seriali

Re: Dynamic Cluster/Index Routing for Elasticsearch Sink

Hi David, That was perfect and it looks like this is working as I'd expected. I put together some larger integration tests for my specific use-case (multiple Elasticsearch clusters running in TestContainers) and verified that messages were being routed dynamically to the appropriate sinks. I forke

【Announce】Zeppelin 0.10.0 is released, Flink on Zeppelin Improved

Hi Flink users, We (Zeppelin community) are very excited to announce Zeppelin 0.10.0 is officially released. In this version, we made several improvements on Flink interpreter. Here's the main features of Flink on Zeppelin: - Support multiple versions of Flink - Support multiple versions o

Re: NullPointerException when using KubernetesHaServicesFactory

The exception is showing up both in TM and JM This however seemed only to appear when running on my local kubernetes setup. > I'd also recommend setting "kubernetes.namespace" option, unless you're using "default" namespace. Yes, good point - I now see why that was needed. Den ons 25 aug. 2021

Re: [ANNOUNCE] Dropping "CheckpointConfig.setPreferCheckpointForRecovery()"

Hi Stephan, I do not know if anyone is still relying on this but I think it makes sense to drop this feature. So +1 from me. I think it served a valid purpose originally but if we have a good improvement in the pipeline using the savepoints directly that will solve the problem properly. I would c

Query on Metrics | Cumulative Metrics via Prometheus Reporter

Hi, We are using prometheus reporter to fetch metrics (flink 1.12.1 on k8s environment). We were facing an issue since the prometheus reporter has various targets corresponding to each job & task manager hence the different scraping for each target causing syncing issue in metrics count for variou

Disabling autogenerated uid/hash doesn't work when using file source

I set names and uid for all my flink operators and have explicitly disabled auto generation of uid to force developers in my team the same practice. However, when using a file source, there's no option of providing it due to which the job fails to start unless we enable auto generation. Am I doi

Re: NullPointerException when using KubernetesHaServicesFactory

Hi Jonas, Where does the exception pop-up? In job driver, TM, JM? You need to make sure that the plugin folder is setup for all of them, because they all may need to access s3 at some point. Best, D. On Wed, Aug 25, 2021 at 11:54 AM jonas eyob wrote: > Hey Thms, > > tried the s3p:// option as

Re: NullPointerException when using KubernetesHaServicesFactory

Hey Thms, tried the s3p:// option as well - same issue. > Also check if your user that executes the process is able to read the jars. Not exactly sure how to do that? The user "flink" in the docker image is able to read the contents as far I understand. But maybe that's not how I would check it?

Re: Flink SQL: Custom exception handling External

Hi, Thanks for the quick response. The use case is not specific to JDBC (JDBC is just an example) but more for custom error handling in all connectors. How would we go about proposing such a new feature to be added to Flink? On 2021/08/25 09:02:31, Caizhi Weng wrote: > Hi!> > > As far as I

Re: Do we have date_sub function in flink sql?

Hi! Try this ts - interval '1' day where ts is your timestamp or date column. 1095193...@qq.com <1095193...@qq.com> 于2021年8月25日周三 下午5:20写道： > Hi >I want to substract 1 day from current date with Flink sql. Do we have > this function like date_sub()? > > -- > 1

Do we have date_sub function in flink sql?

Hi I want to substract 1 day from current date with Flink sql. Do we have this function like date_sub()? 1095193...@qq.com

Re: NullPointerException when using KubernetesHaServicesFactory

Hey Jonas, you could also try to use the ´s3p://´ scheme to directly specify that presto should be used. Also check if your user that executes the process is able to read the jars. Am Mi., 25. Aug. 2021 um 10:01 Uhr schrieb jonas eyob : > Thanks David for the quick response! > > *face palm* - Tha

Re: Flink SQL: Custom exception handling External

Hi! As far as I know JDBC does not have this error handling mechanism. Also there are very few connectors / formats which support skipping erroneous records (for example the csv format). Which type of exception are you faced with? As JDBC connectors, unlike message queue connectors, rarely (if ev

Flink SQL: Custom exception handling External

Hi, Is there any mechanism for handling of errors produced by Flink SQL? It can be useful for various use cases: 1. Logging exceptions and the erroneous row to a kafka topic 2. Ignoring transient exceptions instead of throwing and failing the entire job If there are no such mechanisms may I propo

Re: NullPointerException when using KubernetesHaServicesFactory

Thanks David for the quick response! *face palm* - Thanks a lot, that seems to have addressed the NullPointerException issue. May I also suggest that this [1] page be updated, since it says the key is " high-availability.cluster-id" This led me to another issue however: "org.apache.flink.core.fs.

Bulk Scheduler timeout when creating several jobs in flink kubernetes HA deployment

Hello, We are struggling a bit with an error in our kubernetes deployment. The deployment is composed of 2 flink job managers and 58 task managers. When deploying the jobs everything is going fine at first, but after the deployment of several jobs (mix of batch and streaming job using the SQL tab

Re: NullPointerException when using KubernetesHaServicesFactory

Hi Jonas, this exception is raised because "kubernetes.cluster-id" [1] is not set. I'd also recommend setting "kubernetes.namespace" option, unless you're using "default" namespace. I've filled FLINK-23961 [2] so we provide more descriptive warning for this issue next time ;) [1] https://ci.apac