Re: Flink job can't complete initialisation because of millions of savepoint file reads

Hi Alex, Please see my comment here [1]. [1] https://lists.apache.org/thread/h5mv6ld4l2g4hsjszfdos9f365nh7ctf BR, G On Mon, Sep 2, 2024 at 11:02 AM Alex K. wrote: > We have an issue where a savepoint file containing Kafka topic partitions > offsets is requested millions of times from AWS S3.

Re: Flink JOb Manager jvm metaspace not getting cleared

Hi Yaroslav, Thank you for your mail. I have tried by removing ojdbc 8 jar from application and keeping it in flink lib folder but that doesn’t help. I debugged code and got the root cause that whenever I am making connection with database at that time metaspcae increases by 10mb. On Fri, 23 Au

Flink job unable to checkpoint (timeout) after restart with savepoint with KafkaSink

Hi, We’ve tried to restart with savepoint 2 different jobs:FlinkKafkaProducer -> KafkaSink with a new UID on it and –allowNonRestoredState flag to reset the state of the sink operator.KafkaSink -> KafkaSink with a new UID on it and –allowNonRestoredState flag to reset the state of the sink operator

Flink job can't complete initialisation because of millions of savepoint file reads

We have an issue where a savepoint file containing Kafka topic partitions offsets is requested millions of times from AWS S3. This results in the job crashing and then followed by a restart and crashing again. We have tracked the high number of reads (~3 millions) to Kafka topic partitions (~40k) m

Re: Flink JOb Manager jvm metaspace not getting cleared

Hi Ganesh, Have you tried following this suggestion? https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code On Fri, Aug 23, 2024 at 10:39 AM Ganesh Walse wrote: > Hi All, > > I am using apache flink for

Flink JOb Manager jvm metaspace not getting cleared

Hi All, I am using apache flink for the bounded data where I need to execute 100 of jobs daily. In every job submission my job manager jvm metaspace is increasing by 10Mb and it is not getting cleared anytime so I am getting JVM OOM error after certain number of job submissions. I searched over

Re: Flink job Deployement problem

:22写道： > Hi, i'm trying to deploy flink job but i have this error. How to solve it > please? > > *Thierry FOKOU *| * IT M.A.Sc <http://M.A.Sc> Student* > > Département de génie logiciel et TI > > École de technologie supérieure | Université du Québec > >

Re: Flink job Deployement problem

/java_lambdas/ [2] https://stackoverflow.com/questions/70295970/flink-table-get-type-information De : Fokou Toukam, Thierry Date : jeudi, 6 juin 2024 à 09:04 À : user Objet : Flink job Deployement problem Hi, i'm trying to deploy flink job but i have this error. How to solve it please? [cid:b13

Flink job Deployement problem

Hi, i'm trying to deploy flink job but i have this error. How to solve it please? [cid:b13357ec-00b9-4a15-8d04-1c797a4eced3] Thierry FOKOU | IT M.A.Sc Student Département de génie logiciel et TI École de technologie supérieure | Université du Québec 1100, rue Notre-Dame Ouest Mon

Re: How to start a flink job on a long running yarn cluster from a checkpoint (with arguments)

Hi Sachin, Yes, that's correct. To resume from a savepoint, use the command bin/flink run -s . You can find more details in the Flink documentation on [1]. Additionally, information on how to trigger a savepoint can be found in the section for triggering savepoints [2]. [1] https://nightlies.ap

How to start a flink job on a long running yarn cluster from a checkpoint (with arguments)

Hi, I have a long running yarn cluster and I submit my streaming job using the following command: flink run -m yarn-cluster -yid application_1473169569237_0001 /usr/lib/flink/examples/streaming/WordCount.jar --input file:///input.txt --output file:///output/ Let's say I want to stop this job, mak

Best Practices? Fault Isolation for Processing Large Number of Same-Shaped Input Kafka Topics in a Big Flink Job

Hi everyone, I'm currently prototyping on a project where we need to process a large number of Kafka input topics (say, a couple of hundred), all of which share the same DataType/Schema. Our objective is to run the same Flink SQL on all of the input topics, but I am concerned about doing this in

Re: Flink job performance

How many taskmanagers and server do you have? Can you also share the task managers page of flink dashboard? On Mon, Apr 15, 2024 at 10:58 AM Oscar Perez via user wrote: > Hi community! > > We have an interesting problem with Flink after increasing parallelism in > a certain way. Here is the sum

Re: Flink job performance

almost full, could you try allocating more CPUs and see if the > instability persists? > > Best, > Zhanghao Chen > -- > *From:* Oscar Perez > *Sent:* Monday, April 15, 2024 19:24 > *To:* Zhanghao Chen > *Cc:* Oscar Perez via user > *Subject:*

Re: Flink job performance

instability persists? Best, Zhanghao Chen From: Oscar Perez Sent: Monday, April 15, 2024 19:24 To: Zhanghao Chen Cc: Oscar Perez via user Subject: Re: Flink job performance Hei, ok that is weird. Let me resend them. Regards, Oscar On Mon, 15 Apr 2024 at 14:00, Zhanghao

Re: Flink job performance

Hi, there seems to be sth wrong with the two images attached in the latest email. I cannot open them. Best, Zhanghao Chen From: Oscar Perez via user Sent: Monday, April 15, 2024 15:57 To: Oscar Perez via user ; pi-team ; Hermes Team Subject: Flink job

Re: Flink job performance

thrown after the change? From: Oscar Perez via user Sent: Monday, April 15, 2024 15:57 To: Oscar Perez via user ; pi-team ; Hermes Team Subject: Flink job performance Hi community! We have an interesting problem with Flink after increasing parallelism in a

Re: Flink job unable to restore from savepoint

flink version 1.17 Didn't change any job configuration. We are facing this below issue. https://issues.apache.org/jira/browse/FLINK-23886 On Wed, Mar 27, 2024 at 1:39 AM Hangxiang Yu wrote: > Hi, Prashant. > Which Flink version did you use? > And Did you modify your job logic or configurations

Re: Flink job unable to restore from savepoint

Hi Prashant, Compared to the job that generated savepoint, are there any changes in the new job? For example, data fields were added or deleted, or the type serializer was changed? More detailed job manager logs may help. prashant parbhane 于2024年3月27日周三 14:20写道： > > Hello, > > We have been facin

Re: Flink job unable to restore from savepoint

Hi, Prashant. Which Flink version did you use? And Did you modify your job logic or configurations ? If yes, Could you share changed things ? On Wed, Mar 27, 2024 at 3:37 PM prashant parbhane wrote: > Hello, > > We have been facing this weird issue of not being able to restore from > savepoint,

Flink job unable to restore from savepoint

Hello, We have been facing this weird issue of not being able to restore from savepoint, when we have a significant load on flink jobs. " *org.apache.flink.util.FlinkRuntimeException: Error while deserializing the user key.* *at org.apache.flink.contrib.streaming.state.RocksDBMapState$RocksD

New Relic alerts for Flink job submission failure and recovery

Hi Team, I would like to know the possibilities of configuring the new relic alerts for a Flink job whenever the job is submitted, gets failed and recovers from the failure. In our case, we have configured the Flink environment as a Kubernetes pod running on an EKS cluster and the application

Testing Flink job with bounded input

Hi, I have a question about how to correctly set up a test that will read input from locally provided collection in bounded mode and provide outputs at the end of the computation. My test case looks something like the following: String[] lines = ...; try (StreamExecutionEnvironment env = St

RE: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

Фролов Cc: user@flink.apache.org Subject: Re: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS) ⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠ Hi Vladimir, As I see, your SQL query is reading data from the Kafka topic and pulls all data to the client

Re: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

o be inserted to the topic? Although I would not expect those *Node - 1 disconnected* messages in such a scenario. Alexey On Tue, Dec 19, 2023 at 10:40 AM Вова Фролов wrote: > Hello Flink Community, > > I am texting to you with an issue I have encountered while using Apache > Flink versi

Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

Hello Flink Community, I am texting to you with an issue I have encountered while using Apache Flink version 1.17.1. In my Flink Job, I am using Kafka version 3.6.0 to ingest data from TPC-DS(current tpcds100 target size tpcds1), and then I am executing SQL queries, specifically, the q77

Re: Query on using two sinks for a Flink job (Flink SQL)

day, December 6, 2023 7:21:50 PM > *收件人:* elakiya udhayanan ; user@flink.apache.org < > user@flink.apache.org> > *主题:* Re: Query on using two sinks for a Flink job (Flink SQL) > > Hi Elakiya, > > You can try executing TableEnvironmentImpl#executeInternal for non-insert &

Re: Query on using two sinks for a Flink job (Flink SQL)

1:50 PM 收件人: elakiya udhayanan ; user@flink.apache.org 主题: Re: Query on using two sinks for a Flink job (Flink SQL) Hi Elakiya, You can try executing TableEnvironmentImpl#executeInternal for non-insert statements, then using StatementSet.addInsertSql to add multiple insertion statetments,

Re: Query on using two sinks for a Flink job (Flink SQL)

s-master/docs/dev/table/sql/insert/ >> >> >> >> -- >> Best！ >> Xuyang >> >> >> At 2023-12-06 17:49:17, "elakiya udhayanan" wrote: >> >> Hi Team, >> I would like to know the possibility of having two sinks in a &g

Re: Query on using two sinks for a Flink job (Flink SQL)

elakiya udhayanan" wrote: > > Hi Team, > I would like to know the possibility of having two sinks in a > single Flink job. In my case I am using the Flink SQL based job where I try > to consume from two different Kafka topics using the create table (as > below) DDL and th

Re:Query on using two sinks for a Flink job (Flink SQL)

would like to know the possibility of having two sinks in a single Flink job. In my case I am using the Flink SQL based job where I try to consume from two different Kafka topics using the create table (as below) DDL and then use a join condition to correlate them and at present write it to a

Re: Query on using two sinks for a Flink job (Flink SQL)

Sent: Wednesday, December 6, 2023 17:49 To: user@flink.apache.org Subject: Query on using two sinks for a Flink job (Flink SQL) Hi Team, I would like to know the possibility of having two sinks in a single Flink job. In my case I am using the Flink SQL based job where I try to consume from two

Query on using two sinks for a Flink job (Flink SQL)

Hi Team, I would like to know the possibility of having two sinks in a single Flink job. In my case I am using the Flink SQL based job where I try to consume from two different Kafka topics using the create table (as below) DDL and then use a join condition to correlate them and at present write

Re: Flink Job Failed With Kafka Exception

hile getting metadata but we would like > flink job to continue to consume data from other source topics even if one > of the topic has any issue since failing entire flink application doesn’t > make sense if one if the topic has issue. > > > > Regards, > Madan > > On Nov

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

s. > > Please let me know if there are any other problems. > > Best, > Yu Chen > > > 2023年11月7日 18:11，arjun s 写道： > > > > Hi Chen, > > I attempted to configure the 'source.path.regex-pattern' property in the > table settings as '^cu

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

> > Hi Chen, > I attempted to configure the 'source.path.regex-pattern' property in the > table settings as '^customer.*' to ensure that the Flink job only processes > file names starting with "customer" in the specified directory. However, it >

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

Hi Chen, I attempted to configure the 'source.path.regex-pattern' property in the table settings as '^customer.*' to ensure that the Flink job only processes file names starting with "customer" in the specified directory. However, it appears that this configuration

Re: Flink Job Failed With Kafka Exception

entire flink application is getting failed due to time out while getting metadata but we would like flink job to continue to consume data from other source topics even if one of the topic has any issue since failing entire flink application doesn’t make sense if one if the topic has issue.Regards,Madan

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

iles >> under the directory-- of `path` >> option >> >> >> Best, >> Yu Chen >> >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/filesystem/ >&

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

-- of `path` option > > > Best, > Yu Chen > > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/filesystem/ > > -- > *发件人:* arjun s > *发送时间:* 2023年11月6日 20:50 > *收件人:* user@flin

回复: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

-- of `path` option Best, Yu Chen [1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/filesystem/ 发件人: arjun s 发送时间: 2023年11月6日 20:50 收件人: user@flink.apache.org 主题: Handling Schema Variability and Applying Regex P

Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

Hi team, I'm currently utilizing the Table API function within my Flink job, with the objective of reading records from CSV files located in a source directory. To obtain the file names, I'm creating a table and specifying the schema using the Table API in Flink. Consequently, when

Re: Flink Job Failed With Kafka Exception

Hi, Madan. This error seems like that there are some problems when the consumer tries to read the topic metadata. If you use the same source for these topics, the kafka connector cannot skip one of them. As you say, you need to modify the connector's default behavior. Maybe you should read the cod

Re: Flink Job Failed With Kafka Exception

Hi Madan, Do you mean you want to restart only the failed tasks, rather than restarting the entire pipeline region? As far as I know, currently Flink does not support task-level restart, but requires restarting the pipeline region. Best, Junrui Madan D via user 于2023年10月11日周三 12:37写道： > Hello

Flink Job Failed With Kafka Exception

Hello Team, We are running the Flink pipeline by consuming data from multiple topics, but we recently encountered that if there's one topic having issues with participation, etc., the whole Flink pipeline is failing, which is affecting topics. Is there a way we can make Flink Piplein keep runnin

Flink job reading from s3 path

Hi team, I want to implement a flink job to read avro files from s3 path, and output to a kafka topic. Currently, I am using AvroInputFormat like this: AvroInputFormat avroInputFormat = new AvroInputFormat<>(new Path(S3PathString), Session.class); TypeInformation ty

Flink job submission to Multi-VM Flink Cluster fails!

Hello, I can launch a Flink cluster (version 1.17.x) on my laptop with 1 Job Manager and 3 Task Managers. The cluster starts, jobs can be submitted correctly on the localhost (my laptop). Next I tried to launch this cluster on 4 VMs - 1 Master

Re: Flink Job Failure for version 1.16

Hi, I may have missed something, So could you share more: I have recently migrated from 1.13.6 to 1.16.1, I can see there is a > performance degradation... Are you referring to a decrease in Checkpoint Performance when you mention performance decline? It just happens when you upgrade from 1.13.

Re: Flink Job Failure for version 1.16

Hi Everyone, can someone please shade some light when the Checkpoint Coordinator is suspending Error comes and what should I do to avoid this? it is impacting the production pipeline after the version upgrade. It is related to resource crunch in the pipeline? Thank You On Thu, May 11, 2023 at 10:3

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

/ops/state/checkpoints/#resuming-from-a-retained-checkpoint Best, Weihua On Thu, May 11, 2023 at 1:43 PM amenreet sodhi wrote: > Hey Hang, > > I am deploying my Flink Job in HA application mode, Whenever I redeploy my > job, or deploy an updated version of the job, it's usin

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

Hey Hang, I am deploying my Flink Job in HA application mode, Whenever I redeploy my job, or deploy an updated version of the job, it's using the same job_id. I haven't configured anywhere to use a fixed job id, I think it's doing it by default. Can you share where I can configur

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

Hi Weihua, I am deploying my flink job in HA application mode on a kubernetes cluster. I am using an external nfs mount for storing checkpoints. For some reason, whenever I deploy an updated version of my application, it uses the same job_id for the new job as for the previous job. Thus the flink

Flink Job Failure for version 1.16

I have recently migrated from 1.13.6 to 1.16.1, I can see there is a performance degradation for the Flink pipeline which is using Flink's managed state ListState, MapState, etc. Pipelines are frequently failing with the Exception: 06:59:42.021 [Checkpoint Timer] WARN o.a.f.r.c.CheckpointFailureM

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

he >> new job will fail when it tries to generate a new checkpoint. >> I'd like to suggest you to use different JOB_ID for different jobs, or >> set a different checkpoint dir for a new job. >> >> On Tue, May 9, 2023 at 9:38 PM amenreet sodhi >> wrote: >&

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

you to use different JOB_ID for different jobs, or set > a different checkpoint dir for a new job. > > On Tue, May 9, 2023 at 9:38 PM amenreet sodhi wrote: > >> Hi all, >> >> Is there any way to prevent restart of flink job, or override the >> checkpoint metada

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

to suggest you to use different JOB_ID for different jobs, or set a different checkpoint dir for a new job. On Tue, May 9, 2023 at 9:38 PM amenreet sodhi wrote: > Hi all, > > Is there any way to prevent restart of flink job, or override the > checkpoint metadata, if for some reason the

Flink Job Restarts if the metadata already exists for some Checkpoint

Hi all, Is there any way to prevent restart of flink job, or override the checkpoint metadata, if for some reason there exists a checkpoint by same name. I get the following exception and my job restarts, have been trying to find solution for a very long time but havent found anything useful yet

Re: Flink Job across Data Centers

nters? >> >> I am trying to create a Multi site/Geo Replicated Kafka cluster. I want >> that my Flink job to be closely colocated with my Kafka multi site cluster. >> If the Flink job is bound to a single data center, I believe we will >> observe a lot of client latency by

Re: Flink job manager conditional start of flink jobs

ces say ns1,ns2. I have to make sure only >> one of ns1 or ns2 should run my flink jobs. Say initially ns1 is running >> flink jobs, later planned to move them to ns2. Now when I start in ns2, I >> can make an api call to ns1 jobmanager about running jobs and if no jobs >>

Re: Flink job manager conditional start of flink jobs

y initially ns1 is running flink > jobs, later planned to move them to ns2. Now when I start in ns2, I can > make an api call to ns1 jobmanager about running jobs and if no jobs then > only I should start in ns2. I can introduce this logic inside the flink job > java main method where my

Re: Flink job manager conditional start of flink jobs

move them to ns2. Now when I start in ns2, I can make an api call to ns1 jobmanager about running jobs and if no jobs then only I should start in ns2. I can introduce this logic inside the flink job java main method where my total streaming logic present. So if I identify then, how can I stop the

Re: Flink job manager conditional start of flink jobs

for your email. > I am looking more in terms of running these flinkk jobs in multi names > pace environment and make sure only one namespace flink jobs are running. > So on the Job manager when i try to start a flink job, it has to check if > it's allowed to run in this namespace or

Re: Flink Job across Data Centers

ss data centers? > > I am trying to create a Multi site/Geo Replicated Kafka cluster. I want > that my Flink job to be closely colocated with my Kafka multi site cluster. > If the Flink job is bound to a single data center, I believe we will > observe a lot of client latency by

Flink Job across Data Centers

Hi, Can anyone share any experience on running Flink jobs across data centers? I am trying to create a Multi site/Geo Replicated Kafka cluster. I want that my Flink job to be closely colocated with my Kafka multi site cluster. If the Flink job is bound to a single data center, I believe we will

Re: Flink job manager conditional start of flink jobs

Thanks for your email. I am looking more in terms of running these flinkk jobs in multi names pace environment and make sure only one namespace flink jobs are running. So on the Job manager when i try to start a flink job, it has to check if it's allowed to run in this namespace or no

Re: Flink job manager conditional start of flink jobs

23 at 10:58 AM naga sudhakar wrote: > Hi Team, > Greetings!! > Just wanted to know when job manager or task manager is being restarted, > is there a way to run the existing flink jobs based on a condition? Same > query when I am starting flink job fresh also. > > Plea

Flink job manager conditional start of flink jobs

Hi Team, Greetings!! Just wanted to know when job manager or task manager is being restarted, is there a way to run the existing flink jobs based on a condition? Same query when I am starting flink job fresh also. Please let me know if any more information is required from my side. Thanks

Re: Flink Job Manager Recovery from EKS Node Terminations

First, JobManager does not store any persistent data to local when the Kubernetes HA + S3 used. It means that you do not need to mount a PV for JobMananger deployment. Secondly, node failures or terminations should not cause the CrashLoopBackOff status. One possible reason I could imagine is a bug

Flink Job Manager Recovery from EKS Node Terminations

Hi, Have a query on the Job Manager HA for flink 1.15. We currently run a standalone flink cluster with a single JobManager and multiple TaskManagers, deployed on top of a kubernetes cluster (EKS cluster) in application mode (reactive mode). The Task Managers are deployed as a ReplicaSet and the

Re: How to get failed streaming Flink job log in Flink Native K8s mode?

I think you might need a sidecar container or daemonset to collect the Flink logs and store into a persistent storage. You could find more information here[1]. [1]. https://www.alibabacloud.com/blog/best-practices-of-kubernetes-log-collection_596356 Best, Yang hjw 于2022年12月22日周四 23:28写道： > On

How to get failed streaming Flink job log in Flink Native K8s mode?

On Flink Native K8s mode, the pod of JM and TM will disappear if the streaming job failed.Are there any ways to get the log of the failed Streaming job? I only think of a solution that is to mount job logs to NFS for persistence through pv-pvc defined in pod-template. ENV: Flink version:1.15.0

Re: Query about flink job manager dashboard

ps://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#advanced-options-for-flink-web-ui >>>> >>>> Best regards, >>>> >>>> Martijn >>>> >>>> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar >>>> wrote

Re: Query about flink job manager dashboard

;> >>> Best regards, >>> >>> Martijn >>> >>> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar >>> wrote: >>> >>>> Hi Team, >>>>> Greetings!!! >>>>> I am a software developer using apache fli

Re: Query about flink job manager dashboard

o queries about flink job manager dashboard. Can you please help with below? 1) is it possible to add login mechanism for the flink job manager dash board and have a role based

Re: Query about flink job manager dashboard

Y5I6gI7lex3qiRrQTWoM_EbECDFr62njjFlxqyUFfLFS6GofOfFjNbK9HcXrwyq0w$> >>> >>> Best regards, >>> >>> Martijn >>> >>> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar >>> wrote: >>> >>>> Hi Team, >>>>> Gr

Re: Query about flink job manager dashboard

me. I have two queries about flink job manager dashboard. Can you please help with below? 1) is it possible to add login mechanism for the flink job manager dash board and have a role based mechanism for viewing running jobs, cancelling jo

Re: Query about flink job manager dashboard

sudhakar >> wrote: >> >>> Hi Team, >>>> Greetings!!! >>>> I am a software developer using apache flink and deploying flink jobs >>>> using the same. I have two queries about flink job manager dashboard. Can >>>> you please he

Re: Query about flink job manager dashboard

t;>> I am a software developer using apache flink and deploying flink jobs >>> using the same. I have two queries about flink job manager dashboard. Can >>> you please help with below? >>> >>> 1) is it possible to add login mechanism for the flink job manage

Re: Query about flink job manager dashboard

deploying flink jobs >>> using the same. I have two queries about flink job manager dashboard. Can >>> you please help with below? >>> >>> 1) is it possible to add login mechanism for the flink job manager dash >>> board and have a role based mechanism f

Re: Query about flink job manager dashboard

ov 24, 2022 at 3:44 AM naga sudhakar wrote: > Hi Team, >> Greetings!!! >> I am a software developer using apache flink and deploying flink jobs >> using the same. I have two queries about flink job manager dashboard. Can >> you please help with below? >> >>

Query about flink job manager dashboard

> > Hi Team, > Greetings!!! > I am a software developer using apache flink and deploying flink jobs > using the same. I have two queries about flink job manager dashboard. Can > you please help with below? > > 1) is it possible to add login mechanism for the flink job manag

Query about flink job manager dashboard

> Hi Team, > Greetings!!! > I am a software developer using apache flink and deploying flink jobs > using the same. I have two queries about flink job manager dashboard. Can > you please help with below? > > 1) is it possible to add login mechanism for the flink job manager da

Re: support escaping `#` in flink job spec in Flink-operator

built into flink-core. However, I think we worry that this might be > breaking some applications which rely on the lenient behavior of the > existing parser. > > -Max > > On Tue, Nov 8, 2022 at 12:36 PM liuxiangcao > wrote: > >> Hi Yang, >> >> Do you

Re: support escaping `#` in flink job spec in Flink-operator

ting `#` in FLINK-15358[1] and Flink > job spec not supporting `#` are caused by some common code? or maybe they > are in different code paths? My first guess was they are in different > code paths. The flink-conf is parsed when starting the flink cluster while > job spec is parsed when

Re: support escaping `#` in flink job spec in Flink-operator

Hi Yang, Do you think flink-conf not supporting `#` in FLINK-15358[1] and Flink job spec not supporting `#` are caused by some common code? or maybe they are in different code paths? My first guess was they are in different code paths. The flink-conf is parsed when starting the flink cluster

Re: support escaping `#` in flink job spec in Flink-operator

Hi Gyula, Thanks for getting back. Could you share how to submit job to flinkk8operator in json format? We use the java Fabric8 K8 client, which serializes java FlinkDeployment objects to CustomResource YAML (see the code snippet below). Since `#` is considered a special character denoting comme

Re: support escaping `#` in flink job spec in Flink-operator

This is a known limit of the current Flink options parser. Refer to FLINK-15358[1] for more information. [1]. https://issues.apache.org/jira/browse/FLINK-15358 Best, Yang Gyula Fóra 于2022年11月8日周二 14:41写道： > It is also possible that this is a problem of the Flink native Kubernetes > integration

Re: support escaping `#` in flink job spec in Flink-operator

It is also possible that this is a problem of the Flink native Kubernetes integration, we have to check where exactly it goes wrong before we try to fix it . We simply set the args into a Flink config and pass it to the native deployment logic in the operator. Gyula On Tue, 8 Nov 2022 at 07:37,

Re: support escaping `#` in flink job spec in Flink-operator

Hi! How do you submit your yaml? It’s possible that this is not operator problem. Did you try submitting the deployment in json format instead? If it still doesn't work please open a JIRA ticket with the details to reproduce and what you have tried :) Cheers Gyula On Tue, 8 Nov 2022 at 04:56,

support escaping `#` in flink job spec in Flink-operator

Hi, We have a job that contains `#` as part of mainArgs and it used to work on Ververica. Now we are switching to our own control plane to deploy to flink-operaotor and the job started to fail due to the main args string getting truncated at `#` character when passed to flink application. I believ

Re: Is there an HA solution to run flink job with multiple source

, we would like to process it's data. We use a single flink job, since we find it easier to manage less flink jobs, and that way we also use less resources So far, the idea from Xuyang seems doable to me, I'll explore the idea of subclassing existing Kafka source and making sure that kafka

Re: Is there an HA solution to run flink job with multiple source

opy the implementation of >> existing kafka source and change a little code conveniently. >> >> At 2022-06-01 22:38:39, "Bariša Obradović" wrote: >> >> Hi, >> we are running a flink job with multiple kafka sources connected to >> different ka

Re: Is there an HA solution to run flink job with multiple source

elf. But the good news is that you can copy the implementation of > existing kafka source and change a little code conveniently. > > At 2022-06-01 22:38:39, "Bariša Obradović" wrote: > > Hi, > we are running a flink job with multiple kafka sources connected to > diff

Re:Is there an HA solution to run flink job with multiple source

existing kafka source and change a little code conveniently. At 2022-06-01 22:38:39, "Bariša Obradović" wrote: Hi, we are running a flink job with multiple kafka sources connected to different kafka servers. The problem we are facing is when one of the kafka's is down

Is there an HA solution to run flink job with multiple source

Hi, we are running a flink job with multiple kafka sources connected to different kafka servers. The problem we are facing is when one of the kafka's is down, the flink job starts restarting. Is there anyway for flink to pause processing of the kafka which is down, and yet continue proce

Re: Flink Job Manager unable to recognize Task Manager Available slots

/3a97d1d50f663027ae81efe0f0aa rmr /flink/default/leader/3a97d1d50f663027ae81efe0f0aa This should result in your JobManager recovering from the faulty job. Regards, Hong From: "s_penakalap...@yahoo.com" Date: Tuesday, 24 May 2022 at 18:40 To: User Subject: RE: [EXTERNAL]Flink Job Manager

Re: Flink Job Manager unable to recognize Task Manager Available slots

Hi Team, Any inputs please badly stuck. Regards,Sunitha On Sunday, May 22, 2022, 12:34:22 AM GMT+5:30, s_penakalap...@yahoo.com wrote: Hi All, Help please! We have standalone Flink service installed in individual VM and clubed to form a cluster with HA and checkpoint in place. When can

Flink Job Manager unable to recognize Task Manager Available slots

Hi All, Help please! We have standalone Flink service installed in individual VM and clubed to form a cluster with HA and checkpoint in place. When cancelling Job, Flink cluster went down and its unable to start up normally as Job manager is continuously going down with the below error: 2022-05-

答复: Flink Job Execution issue at Yarn

mum-allocation-mb` and `yarn.nodemanager.resource.memory-mb`. Maybe it is better to post the JM/TM logs if any of them exists to provide more information. Best, Biao Geng 发件人: Anitha Thankappan 日期: 星期三, 2022年5月18日 下午8:26 收件人: user@flink.apache.org 主题: Flink Job Execution issue at Yarn Hi, We

Flink Job Execution issue at Yarn

Hi, We are using below command to submit a flink application job to GCP dataproc cluster using Yarn. *flink run-application -t yarn-application .jar* Our Cluster have 1 master node with 64 GB and 10 worker nodes of 32 GB. The flink configurations given are: *jobmanager.memory.process.size: 400

  1   2   3   4   5   6   7   8   >