Re: Jobmanager Crashes with Kubernetes HA When Restart Kubernetes Master Node

I think your attached exception has been fixed via FLINK-22597[1]. Could you please have a try with the latest version. Moreover, it is not the desired Flink behavior that TaskManager could not retrieve the new JobManager address and re-register successfully. I think you need to share the staled T

Re: yarn ship from s3

Thank You Xintong, I will look for these updates in the near future. Regards, Vijay On Wed, May 26, 2021 at 6:40 PM Xintong Song wrote: > Hi Vijay, > > Currently, Flink only supports shipping files from the local machine where > job is submitted. > > There are tickets [1][2][3] tracking the eff

Re: Time needed to read from Kafka source

Hi, I forgot to mention that we are using Flink 1.12.0. This is a job that has only minimum components. Reading from source and printing it. Profiling was my next step to do. Regarding memory I didn't see any bottlenecks. I guess I will have to do some investigating in the metric part of Flink. BR

Re: Time needed to read from Kafka source

Hi, I forgot to mention that we are running Flink 1.12.0. This is the main function (some parts of codes are abbreviated and this is the main part). As you can see the job was simplified to minimum. Just reading from source and printing. [image: Screenshot 2021-05-26 at 08.05.53.png] And this

multiple streams joining

Hi, Imagine I have one class having 4 fields: ID, A, B, C. There are three data sources providing data in the form of (ID, A), (ID, B), (ID, C) respectively. I want to join these three data sources to get final (ID, A, B, C) without any window. For example, (ID, A) could come one month after (ID,

Re: How can I use different user run flink

Hi igyu: You can submit job use these arguements like this ``` -m yarn-cluster \ -yqu root.realtime \ -ynm “test" \ -yjm 2g \ -ytm 2g \ -n \ -d \ -ys 1 \ -yD security.kerberos.login.principal=xxx...@x.com \ -yD security.kerberos.login.keytab=/tmp/xx.keytab \ ... ``` > On May 27, 2021,

Re: yarn ship from s3

Hi Vijay, Currently, Flink only supports shipping files from the local machine where job is submitted. There are tickets [1][2][3] tracking the efforts that shipping files from remote paths, e.g., http, hdfs, etc. Once the efforts are done, adding s3 as an additional supported schema should be st

How can I use different user run flink

I use CDH 6.3.2 flink-1.12.3 I enabel kerberos I want to use different user with different keytab,because I creat many queue in YARN , different user use different queue. igyu

Re: Jobmanager Crashes with Kubernetes HA When Restart Kubernetes Master Node

Hi Yang, Thanks for getting back to me. By “restart master node”, I mean do “kubctl get nodes” to find the node’s role as master and “ssh” into one of master nodes as ubuntu user. Then run “sudo /sbin/reboot -f” to restart the master node. It looks like The JobManager would cancel the running

Re: KafkaSource metrics

Found https://issues.apache.org/jira/browse/FLINK-22766 From: Alexey Trenikhun Sent: Tuesday, May 25, 2021 3:25 PM To: Ardhani Narasimha ; 陳樺威 ; Flink User Mail List Subject: Re: KafkaSource metrics Looks like when KafkaSource is used instead of FlinkKafkaConsu

Error restarting job from Savepoint

Hello, We are facing an error restarting a job from a savepoint. We believe it is because one of the common classes used across all of our jobs was changed but there was no *serialVersionUID* assigned to the class. There error we are facing is java.lang.Exception: Exception while creating StreamO

Re: yarn ship from s3

Hi Pohl, I tried to ship my property file. Example: *-yarn.ship-files s3://applib/xx/xx/1.0-SNAPSHOT/application.properties \* *Error:* 6:21:37.163 [main] ERROR org.apache.flink.client.cli.CliFrontend - Invalid command line arguments. org.apache.flink.client.cli.CliArgsException: Could not bui

Re: Customer operator in BATCH execution mode

On Wed, May 26, 2021 at 01:03:53PM +0200, Dawid Wysakowicz wrote: > Hi, > > No there is no API in the operator to know which mode it works in. We > aim to have separate operators for both modes if required. You can check > e.g. how we do it in KeyedBroadcastStateTransformationTranslator[1]. Thank

Query related to Minimum scrape interval for Prometheus and fetching metrics of all vertices in a job through Flink Rest API

Hi team, I have two queries as mentioned below: *Query1:* I am using PrometheusReporter to expose metrics to Prometheus Server. What should be the minimum recommended scrape interval to be defined on Prometheus server? Is there any interval in which Flink reports metrics? *Query2:* Is there any

Re: Managing Jobs entirely with Flink Monitoring API

Glad to hear it! Thanks for confirming that it works. Piotrek śr., 26 maj 2021 o 12:59 Barak Ben Nathan napisał(a): > > > Hi Piotrek, > > > > This is exactly what I was searching for. Thanks! > > > > Barak > > > > *From:* Piotr Nowojski > *Sent:* Wednesday, May 26, 2021 9:59 AM > *To:* Barak

Re: Customer operator in BATCH execution mode

Hi, No there is no API in the operator to know which mode it works in. We aim to have separate operators for both modes if required. You can check e.g. how we do it in KeyedBroadcastStateTransformationTranslator[1]. Yes, it should be possible to register a timer for Long.MAX_WATERMARK if you want

Re: Job recovery issues with state restoration

/mnt/data is a local disk, so there shouldn’t be any additional latency. I’ll provide more information when/if this happens again. Peter From: Roman Khachatryan Date: Tuesday, May 25, 2021 at 6:54 PM To: Peter Westermann Cc: user@flink.apache.org Subject: Re: Job recovery issues with state re

Re: [VOTE] Release 1.13.1, release candidate #1

Hi Dawid, +1 (non-binding) Thanks for driving this release. I checked the following things: - downloaded and build source code - verified checksums - double-checked diff of pom files between 1.13.0 and 1.13.1-rc1 - did a visual check of the release blog post - started cluster and ran jobs (WindowJ

Re: Flink 1.12.4 docker image

Just FYI https://hub.docker.com/_/flink is updated now as well. On Wed, May 26, 2021 at 9:57 AM Nikola Hrusov wrote: > Hello Arvid, > > Thank you for your fast response > > Regards > , > Nikola > > > On Tue, May 25, 2021 at 7:11 PM Arvid Heise wrote: > >> Hi Nikola, >> >> https://hub.docker.com

Fwd: Getting error in pod template

-- Forwarded message - From: Priyanka Manickam Date: Tue, 25 May 2021, 21:13 Subject: Fwd: Getting error in pod template To: user , Yang Wang -- Forwarded message - From: Priyanka Manickam Date: Tue, 25 May 2021, 21:11 Subject: Re: Getting error in pod templat

Re: Flink 1.12.4 docker image

Hello Arvid, Thank you for your fast response Regards , Nikola On Tue, May 25, 2021 at 7:11 PM Arvid Heise wrote: > Hi Nikola, > > https://hub.docker.com/r/apache/flink now contains the images. It takes a > few days until https://hub.docker.com/_/flink is updated though. > > Sorry for the has

Re: Flink 1.11.3 NoClassDefFoundError: Could not initialize class

Hi, Maybe before deleting the pods, you could look inside them and inspect your job's jar? What classes does it have inside it? The job's jar should be in a local directory. Or maybe even first inspect the jar before submitting it? Best, Piotrek wt., 25 maj 2021 o 17:40 Georgi Stoyanov napisał(