Flink 1.16.3 parallelism issue

Hi Team, I am currently running Flink 1.16.3 with Apache Beam to run our batch jobs. We are trying to process the data with an input size of over 20 million, but unfortunately, it is failing. we want Flink to handle the parallelism based on input load automatically but even with -1 configuration,

Re: Issue while migrating to flink 1.20.0 with Java 17 & Scala 2.12

Hi All , Facing the same issue. Has anyone else faced the same issue. Is Java 17 still on experimental support in Flink 1.20 ? Thanks, Prasanna. On Tue, Nov 19, 2024 at 3:36 PM Nitin Saini wrote: > Hello Flink Community, > > I am in the process of migrating my Flink jobs fro

Issue while migrating to flink 1.20.0 with Java 17 & Scala 2.12

reads, I found that Java 17 is reportedly incompatible with Scala 2.12, and it is recommended to use Scala 2.13 instead. If this is indeed the issue, is there a way to resolve it? Unfortunately, I couldn't find any Flink 1.20.0 images that use Scala 2.13. I would greatly appreciate any assist

Hikari data source and jdbc connection issue

Hi All, I am using Hikari data source with ojdbc 8 to connect with database in flink application. I have series of jobs which other java application submit to my jobmanager. And on each job submission I will connect to database after that my job manager and task manager metaspace increases by 10m

Hikari data source and jdbc connection issue

Hi All, I am using Hikari data source with ojdbc 8 to connect with database in flink application. I have series of jobs which other java application submit to my jobmanager. And on each job submission I will connect to database after that my job manager and task manager metaspace increases by 10m

[Help needed] Flink Statefun Runtime on Kubernetes - OutOfMemoryError Issue

(ThreadExecutorMap.java:74) [statefun-flink-distribution.jar:3.2.0] at java.lang.Thread.run(Unknown Source) [?:?] ``` I am seeking advice on how to address this issue. Specifically, I would like to know: How can I increase memory allocation in the Statefun manifest? What might be causing this crash

Re: Issue with protobuf kyro serializer for UnmodifiableLazyStringList

at 8:14 AM Sebastian Zapata wrote: > Hi everyone wanted to check if somebody else has found an issue like > this?, and could maybe have some pointer or has encountered this before > > my kyro serializer is failing to serialize > com.google.protobuf.UnmodifiableLazyStringList >

Issue with protobuf kyro serializer for UnmodifiableLazyStringList

Hi everyone wanted to check if somebody else has found an issue like this?, and could maybe have some pointer or has encountered this before my kyro serializer is failing to serialize com.google.protobuf.UnmodifiableLazyStringList I am tyring to read a protobuf message with the data streaming

Re: [External] Re: [External] Re:Re: Backpressure issue with Flink Sql Job

gt;>> Hi all, >>>> >>>> We are facing backpressure in the flink sql job from the sink and the >>>> backpressure only comes from a single task. This causes the checkpoint to >>>> fail despite enabling unaligned checkpoints and using debloating bu

Re: [External] Re:Re: Backpressure issue with Flink Sql Job

ite enabling unaligned checkpoints and using debloating buffers. >>> We enabled flamegraph and the task spends most of the time doing rocksdb >>> get and put. The sql job does a left join over two streams with a >>> parallelism of 20. The total data the topics have is 540Gb

Re: [External] Re:Re: Backpressure issue with Flink Sql Job

oughly 60Gb in the second topic. We are running 20 taskmanagers with 1 >> slot each with each taskmanager having 72G mem and 9 cpu. >> Can you provide any help on how to go about fixing the pipeline? We are >> using Flink 1.17.2. The issue is similar to this stackoverflow thread >> <https://stackoverflow.com/questions/77762119/flink-sql-job-stops-with-backpressure-after-a-week-of-execution>, >> instead of week it starts facing back pressure as soon as the lag comes >> down to 4-5%. >> >> [image: image.png] >> >

Re:Re: Backpressure issue with Flink Sql Job

running 20 taskmanagers with 1 slot each with each taskmanager having 72G mem and 9 cpu. Can you provide any help on how to go about fixing the pipeline? We are using Flink 1.17.2. The issue is similar to this stackoverflow thread, instead of week it starts facing back pressure as soon as the lag

Re: Backpressure issue with Flink Sql Job

opic > and roughly 60Gb in the second topic. We are running 20 taskmanagers with 1 > slot each with each taskmanager having 72G mem and 9 cpu. > Can you provide any help on how to go about fixing the pipeline? We are > using Flink 1.17.2. The issue is similar to this stackoverflow thr

Backpressure issue with Flink Sql Job

on how to go about fixing the pipeline? We are using Flink 1.17.2. The issue is similar to this stackoverflow thread <https://stackoverflow.com/questions/77762119/flink-sql-job-stops-with-backpressure-after-a-week-of-execution>, instead of week it starts facing back pressure as soon as the lag

Runtime issue while using statefun-datastream v3.3.0

Hi all, We are trying to leverage statefun datastream features. After adding *org.apache.flink:statefun-flink-datastream:3.3.0* in our gradle file, we are experiencing a runtime error like this, *Caused by: java.lang.NoSuchMethodError: ‘com.google.protobuf.Descriptors$FileDescriptor com.google.pro

Issue with presto recovering savepoint files in Flink 1.19

Hi all, We have just upgraded to Flink 1.19 and we are experiencing some issue when the job tries to restore from *some* savepoints, not all. In these cases, the failure manifests when the job is unable to create a checkpoint after starting from savepoint, saying *Failure reason: Not all required

Re: TTL issue with large RocksDB keyed state

on filter for TTL? > > 2. I'm wondering if FIFO compaction is a solution for above. To move to that > that we will need to first take a canonical savepoint then redeploy with > RocksDB/FIFO. That should work but will doing that "reset the clock" for the > TTL? Given it'

TTL issue with large RocksDB keyed state

That should work but will doing that "reset the clock" for the TTL? Given it's nature though, I am leaning to this as our only option. 3. Rescaling is a problem because of this issue: https://issues.apache.org/jira/browse/FLINK-34050. The fix for this is not yet released. Because of t

Re: Flink autoscaler with AWS ASG: checkpoint access issue

Hello, After digging into the 403 issue a bit, I figured out that after the scale-up event, the flink-s3-fs-presto uses the node-profile instead of IRSA (Iam Role for Service Account) on some of the newly created TM pods. 1. Anyone else experienced this as well? 2. Verified that this is an issue

Flink autoscaler with AWS ASG: checkpoint access issue

rest of the TM pods are scheduled on these new nodes. Issue After the scale-up, the TM pods scheduled on the existing nodes with available resources successfully read the checkpoint from S3 however the TM pods scheduled on the new nodes added by ASG run into 403 (access denied) while reading the same

Re: Re: Looking for help with Job Initialisation issue

:38写道： > > > > > > Some more context: Our job graph has 5 different Tasks/operators/flink > functions of which we are seeing this issue every time in a particular > operator > > > We’re using Unaligned checkpoints. With aligned checkpoint we don’t > see this issu

RE: Re: Looking for help with Job Initialisation issue

ad dump to see where the > operator is getting stuck. > > Best, > Biao Geng > > Abhi Sagar Khatri via user 于2024年4月30日周二 19:38写道： > > > > Some more context: Our job graph has 5 different Tasks/operators/flink functions of which we are seeing this issue every time

Re: Flink submit to cluster issue

Hi Phil, >>> Is it a library missing from the Flink cluster? What would cause that error? In most cases, it implies the collision of jar dependencies. flink-sql-connector-kafka jar should be enough to run the job and you may need to check your classpath in the jobmanager to see if any other jars t

Flink submit to cluster issue

I am running a Flink job locally using python -m job.py and it runs fine. The job is: calcul_count = t_env.execute_sql(""" SELECT username, COUNT(action) AS a_count FROM kafka_logs GROUP BY username """) with calcul_count.collect() as results: for row in results: print(row) When i try to s

Re: Issue in PrefetchCount

Hi ajay, When you have 3 parallelisms you will have 3 independent clients. If you want to keep prefetch count 3 you need to set setRequestedChannelMax as 1 and setParallelism 3. So All 3 clients can have one connection. Talat On Tue, May 7, 2024 at 5:52 AM ajay pandey wrote: > Hi Flink Team, >

Re: Looking for help with Job Initialisation issue

周二 19:38写道： > > Some more context: Our job graph has 5 different Tasks/operators/flink > functions of which we are seeing this issue every time in a particular > operator > We’re using Unaligned checkpoints. With aligned checkpoint we don’t see this > issue but the checkpoin

Re: Looking for help with Job Initialisation issue

Some more context: Our job graph has 5 different Tasks/operators/flink functions of which we are seeing this issue every time in a particular operator We’re using Unaligned checkpoints. With aligned checkpoint we don’t see this issue but the checkpoint duration in that case is very high and causes

Looking for help with Job Initialisation issue

Hi Flink folks, Our team has been working on a Flink service. After completing the service development, we moved on to the Job Stabilisation exercises at the production load. During high load, we see that if the job restarts (mostly due to the "org.apache.flink.util.FlinkExpectedException: The Task

Re:Re: Found an issue when using Flink 1.19's AsyncScalarFunction

ns.text` to the OWNER_CLASSPATH list, the issue can be resolved. I'll create a JIRA about it. On Mon, Apr 15, 2024 at 12:14 PM Xiaolong Wang wrote: Sure, the AsyncScalarFunction's code looks like : public class AsyncHashCodeFunction extends AsyncScalarFunction { private int factor = 0

Re:Found an issue when using Flink 1.19's AsyncScalarFunction

s bug by this way, I think we should open an bug issue for it. -- Best！ Xuyang At 2024-04-09 18:11:27, "Xiaolong Wang" wrote: Hi, I found a ClassNotFound exception when using Flink 1.19's AsyncScalarFunction. Stack trace: Caused b

Found an issue when using Flink 1.19's AsyncScalarFunction

[?:?] > > at > org.apache.flink.table.planner.delegation.StreamPlanner.$anonfun$translateToPlan$1(StreamPlanner.scala:85) > ~[?:?] > Environment: flink image: flink:1.19.0-scala_2.12-java11 Tried solutions: I tried to package the needed dependency `commons-io-1.10.0.jar` into both user jar and the classpath and the issue remained. Would someone please help resolve this ?

Re: emitValueWithRetract issue

Hi Adam, Thanks for reporting this issue. The bug you identified has been fixed in the release-1.18 and master branches, which will be released in v1.18.2 and v1.19.0. You can now use the `emitUpdateWithRetract` method with the expected behavior. More details can be found here[1]. [1] https

RE: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

Фролов Cc: user@flink.apache.org Subject: Re: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS) ⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠ Hi Vladimir, As I see, your SQL query is reading data from the Kafka topic and pulls all data to the client

Re: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

o be inserted to the topic? Although I would not expect those *Node - 1 disconnected* messages in such a scenario. Alexey On Tue, Dec 19, 2023 at 10:40 AM Вова Фролов wrote: > Hello Flink Community, > > I am texting to you with an issue I have encountered while using Apache > Flink versi

Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

Hello Flink Community, I am texting to you with an issue I have encountered while using Apache Flink version 1.17.1. In my Flink Job, I am using Kafka version 3.6.0 to ingest data from TPC-DS(current tpcds100 target size tpcds1), and then I am executing SQL queries, specifically, the q77

Re: oomkill issue

ap > > 1009.svg > ``` > > Otherwise, if the OOMKill no longer occurs, but the GC overhead limit > exceeded, then you should dump heap memory to find out what objects are > taking up so much of the memory. > Here is the command for you. > ``` > jmap -dump:live,fo

Re: 回复: oomkill issue

emory. > Here is the command for you. > ``` > jmap -dump:live,format=b,file=/tmp/heap.hprof > ``` > > [1] Using jemalloc to Optimize Memory Allocation — Sentieon Appnotes > 202308.01 documentation <https://support.sentieon.com/appnotes/jemalloc/> > > Best, > Yu Chen &

Re:回复: oomkill issue

e command for you. ``` jmap -dump:live,format=b,file=/tmp/heap.hprof ``` [1] Using jemalloc to Optimize Memory Allocation — Sentieon Appnotes 202308.01 documentation Best, Yu Chen 发件人: prashant parbhane 发送时间: 2023年11月28日 1:42 收件人: user@flink.apache.org 主题: oomkill issue Hello, We hav

回复: oomkill issue

e.org 主题: oomkill issue Hello, We have been facing this oomkill issue, where task managers are getting restarted with this error. I am seeing memory consumption increasing in a linear manner, i have given memory and CPU as high as possible but still facing the same issue. We are using rocksdb for

oomkill issue

Hello, We have been facing this oomkill issue, where task managers are getting restarted with this error. I am seeing memory consumption increasing in a linear manner, i have given memory and CPU as high as possible but still facing the same issue. We are using rocksdb for the state backend, is

FLINK-24035, how can this issue be repeated?

We encountered similar problems in production, and we want to integrate FLINK-24035 to solve them, but we don't know how to repeat the problem.

Re: Needs help debugging an issue

The additional exceptions with the same error but on different files Pyflink lib error : java.lang.RuntimeException: An error occurred while copying the file. at org.apache.flink.api.common.cache.DistributedCache.getFile( DistributedCache.java:158) at org.apache.flink.python.env.PythonDependencyI

Needs help debugging an issue

Hi, We are using flink-1.17.0 table API and RocksDB as backend to provide a service to our users to run sql queries. The tables are created using the avro schema and we also provide users to attach python udf as a plugin. This plugin is downloaded at the time of building the table and we update th

Re: Issue with flink-kubernetes-operator not updating execution.savepoint.path after savepoint deletion

ading our operator version to the HEAD commit of the release-1.6 > branch ( > https://github.com/apache/flink-kubernetes-operator/pkgs/container/flink-kubernetes-operator/127962962?tag=3f0dc2e), > we are still seeing this same issue. > > Here's the log message on the last savep

Re: Issue with flink-kubernetes-operator not updating execution.savepoint.path after savepoint deletion

Hi Gyula, After upgrading our operator version to the HEAD commit of the release-1.6 branch ( https://github.com/apache/flink-kubernetes-operator/pkgs/container/flink-kubernetes-operator/127962962?tag=3f0dc2e), we are still seeing this same issue. Here's the log message on the last save

Flink version upgrade issues (schema register forbidden error + incorrect kafka offset issue)

Hi team, we encountered the schema forbidden issue during deployment with the changes of flink version upgrade (1.16.1 -> 1.17.1), and also encountered incorrect reset kafka offset issue during restoring our flink job from this change. Hope to get some help here. *1 - schema forbidden issue*

Re: File Source Watermark Issue

> > > > Can someone help me here? > > > > Regards, > > Kirti Dhar > > > > From: Kirti Dhar Upadhyay K > Sent: 10 October 2023 15:52 > To: user@flink.apache.org > Subject: File Source Watermark Issue > > > > Hi Team, > > > >

RE: File Source Watermark Issue

Hi Community, Can someone help me here? Regards, Kirti Dhar From: Kirti Dhar Upadhyay K Sent: 10 October 2023 15:52 To: user@flink.apache.org Subject: File Source Watermark Issue Hi Team, I am using Flink File Source with window aggregator as process function, and stuck with a weird issues

File Source Watermark Issue

Hi Team, I am using Flink File Source with window aggregator as process function, and stuck with a weird issues. File source doesn't seem emitting/progressing the watermarks, whereas if I put a delay (say 100ms) while extracting timestamp from event, it is working fine. A bit same thing I found

Re: Issue with flink-kubernetes-operator not updating execution.savepoint.path after savepoint deletion

06:02, Nathan Moderwell < nathan.moderw...@robinhood.com> wrote: > Small update on this. I see that the issue is that we use `upgradeMode: > savepoint`, but have not configured the operator to retain savepoints for > long enough (the previous operator we used never deleted s

Re: Issue with flink-kubernetes-operator not updating execution.savepoint.path after savepoint deletion

Small update on this. I see that the issue is that we use `upgradeMode: savepoint`, but have not configured the operator to retain savepoints for long enough (the previous operator we used never deleted savepoints so we didn't run into this). I am reconfiguring to use `upgradeMode: last-state

Issue with flink-kubernetes-operator not updating execution.savepoint.path after savepoint deletion

Hi flink-kubernetes-operator maintainers, We have recently migrated to the official operator and seeing a new issue where our FlinkDeployments can fail and crashloop looking for a non-existent savepoint. On further inspection, the job is attempting to restart from the savepoint specified in

回复: InaccessibleObjectException issue

: Re: InaccessibleObjectException issue   H Bill, I would recommend you to try that example with JDK 11.  Support of JDK 17 is coming in Flink 1.18  https://issues.apache.org/jira/browse/FLINK-15736 Best regards, Alexey On Tue, Aug 29, 2023 at 11:50 AM Xu Bill wrote: Hello, I'm new to A

Re: InaccessibleObjectException issue

H Bill, I would recommend you to try that example with JDK 11. Support of JDK 17 is coming in Flink 1.18 https://issues.apache.org/jira/browse/FLINK-15736 Best regards, Alexey On Tue, Aug 29, 2023 at 11:50 AM Xu Bill wrote: > Hello, > > I'm new to Apache Flink. > I followed the instrunctions t

InaccessibleObjectException issue

Hello, I'm new to Apache Flink. I followed the instrunctions to run examples after installation (as described here: https://nightlies.apache.org/flink/flink-docs-stable/docs/try-flink/local_installation/#submitting-a-flink-job) and also tried running an example java app (as described here: ht

State Memory Leak Issue in Flink CEP v1.17: Discussion & Proposed Solution

Hi, We have encountered a state memory leak issue while working with the Flink CEP (Complex Event Processing) in version 1.17. Based on code, it seems like the issue is persistent across all versions of CEP library. I've already raised a JIRA issue detailing the problem we've face

Re: [Issue] Failure on stopping Flink(1.17) application

It seems it is not removed until 1.17 version) Does someone know how to avoid this kind of issue? Thanks. 2023년 8월 20일 (일) 오후 3:36, liu ron 님이 작성: > Hi, > > I think you can check the client side and JobManager side log to get more > info. > > Best, > Ron > > Dennis

Re: [Issue] Failure on stopping Flink(1.17) application

t; org.apache.flink.runtime.checkpoint.CheckpointException: > org.apache.flink.runtime.checkpoint.CheckpointException: Task name with > subtask : Source: kafka-source (1/2)#0 Failure reason: Task has failed. > > org.apache.flink.runtime.taskmanager.Task.declineCheckpoint(Task.java:1395) > > > org.apache.flink.runtime.taskmanager.Task.lambda$triggerCheckpointBarrier$3(Task.java:1338) > ... > ``` > > Was there some similar issue? > > Regards > > >

[Issue] Failure on stopping Flink(1.17) application

Task name with subtask : Source: kafka-source (1/2)#0 Failure reason: Task has failed. org.apache.flink.runtime.taskmanager.Task.declineCheckpoint(Task.java:1395) org.apache.flink.runtime.taskmanager.Task.lambda$triggerCheckpointBarrier$3(Task.java:1338) ... ``` Was there some similar issue? Regards

Re: [Issue] Repeatedly receiving same message from Kafka

Hello, Thank you. I think it is a problem caused by Kafka configuration, not Flink. I'll take a look and let you know if there's an issue in Flink. BR, Jung 2023년 8월 15일 (화) 오후 9:40, Hector Rios 님이 작성: > Hi there > > It would be helpful if you could include the code for

Re: [Issue] Repeatedly receiving same message from Kafka

situation. Also, can you look in the Flink UI for this job and see if checkpoints are in fact being taken? Hope that helps -Hector On Tue, Aug 15, 2023 at 11:36 AM Dennis Jung wrote: > Sorry, I've forgot putting title, so sending again. > > 2023년 8월 15일 (화) 오후 6:27, Dennis Jung 님이

[Issue] Repeatedly receiving same message from Kafka

Sorry, I've forgot putting title, so sending again. 2023년 8월 15일 (화) 오후 6:27, Dennis Jung 님이 작성: > (this is issue from Flink 1.14) > > Hello, > > I've set up following logic to consume messages from kafka, and produce > them to another kafka broker. F

Re: Flink plain avrò deserilization issue

Attaching client code here again as it got blocked previously by the firewall. Thanks, GIrish From: Girish Narayanan Date: Thursday, 20 July 2023 at 12:06 PM To: user@flink.apache.org Subject: Flink plain avrò deserilization issue Hi Team/Users, We are trying to send plain AVRO messages to a

Flink plain avrò deserilization issue

One or more of the following files ( client code.zip ) violates IBM policy and all attachment(s) have been removed from the message. ** Hi Team/Users, We are trying to send plain AVRO messages to a Kafka topic but they are not

Flink 1.15.1 issue we need help on

Hi, we are trying to determine how fix the following exception. This is an issue that is repeatedly happening for us. We have tried looking online for some solutions. One thread suggested setting idleTimeout but this doesn’t seem supported in Flink Source code. https://github.com/netty/netty

Re: Issue with flink 1.16 and hive dialect

ng out. > 1: > About Hive dialect issue, may be you're using JDK11? > There's a known issue in FLINK-27450[1]. The main reason that Hive > dosen't fully support JDK11. More specific to your case, it has been > tracked in HIVE-21584[2]. > Flink has upgrade the Hi

Re: Issue with flink 1.16 and hive dialect

Hi, Ram. Thanks for reaching out. 1: About Hive dialect issue, may be you're using JDK11? There's a known issue in FLINK-27450[1]. The main reason that Hive dosen't fully support JDK11. More specific to your case, it has been tracked in HIVE-21584[2]. Flink has upgrade the H

Need help to debug the issue 413 Request Entity Too Large

Hi Flink experts, I have been facing this issue for quite some days as I am not able to submit/run SQL Queries using REST CLIENT OR FLINK SQL CLIENT via SQL gateway. Here's my Setup Cluster Set up via k8s operator to deploy Session Cluster, Service gateway and ingress files. I am able t

Issue with flink 1.16 and hive dialect

Hi All, I am not sure if this was already discussed in this forum. In our set up with 1.16.0 flink we have ensured that the setup has all the necessary things for Hive catalog to work. The flink dialect works fine functionally (with some issues will come to that later). But when i follow the ste

Re: Issue with Incremental window aggregation using Aggregate function.

Any thoughts on this? On Fri, Apr 21, 2023 at 4:10 PM Sumanta Majumdar wrote: > Hi, > > Currently we have a streaming use case where we have a flink application > which runs on a session cluster which is responsible for reading data from > Kafka source which is basically table transaction events

Re: Issue when Running the flink-kuberenetes-operator sql-runner (The file STDOUT does not exist on the TaskExecutor)

Thanks Hang and Gyula for the quick response! That makes sense, I can resolve manually for now. On Thu, May 4, 2023 at 8:37 PM Hang Ruan wrote: > Hi, Nathan, > > I think the previous discussion[1] is helpful for you. > > Best, > Hang > > [1] https://lists.apache.org/thread/cgwsr6f1l3202ktwcvtyxt

Re: Issue when Running the flink-kuberenetes-operator sql-runner (The file STDOUT does not exist on the TaskExecutor)

Hi, Nathan, I think the previous discussion[1] is helpful for you. Best, Hang [1] https://lists.apache.org/thread/cgwsr6f1l3202ktwcvtyxtdsgj2vyms1 Nathan Moderwell 于2023年5月5日周五 03:54写道： > Hi, > > I'm running the sql-runner example in the flink-kuberenetes-operator repo > and hitting an error.

Issue when Running the flink-kuberenetes-operator sql-runner (The file STDOUT does not exist on the TaskExecutor)

Hi, I'm running the sql-runner example in the flink-kuberenetes-operator repo and hitting an error. I've tried building the image from different releases of the repo and that does not help. The pods run (status stays at Running) and it is able to build the execution graph successfully, however I s

Re: Flink Kubernetes Operator Scale Issue

Hi! It’s currently not possible to run the operator in parallel by simply adding more replicas. However there are different things you can do to scale both vertically and horizontally. First of all you can run multiple operators each watching different set of namespaces to partition the load. Th

Flink Kubernetes Operator Scale Issue

Hi All, We are using Flink Kubernetes Operator on our production. We have 3k+ jobs in standalone mode. But after 2.5k jobs operator getting slow. Now when we submit a job it takes 10+ minutes to the job runs. Does anyone use similar scale or more job ? Now we run as a single pod. Does operator su

Issue with Incremental window aggregation using Aggregate function.

Hi, Currently we have a streaming use case where we have a flink application which runs on a session cluster which is responsible for reading data from Kafka source which is basically table transaction events getting ingested into the upstream kafka topic which is converted to a row and then dedup

Re: Facing issue when using S3 in Flink 1.17

Thanks a lot, Gabor. Do you know by any chance when this fix will be released? Thanks in advance. On Thu, Apr 20, 2023 at 9:06 PM Gabor Somogyi wrote: > Hi Sriram, > > This has been fixed in https://issues.apache.org/jira/browse/FLINK-31839 > > G > > > On Thu, Apr 20, 2023 at 4:57 PM Sriram Gane

Re: Facing issue when using S3 in Flink 1.17

Hi Sriram, This has been fixed in https://issues.apache.org/jira/browse/FLINK-31839 G On Thu, Apr 20, 2023 at 4:57 PM Sriram Ganesh wrote: > Hi Team, > > I am using S3 as FileSystem to write data from Flink. I am getting the > below error in Flink 1.17. The same code works in Flink 1.16. Coul

Facing issue when using S3 in Flink 1.17

Hi Team, I am using S3 as FileSystem to write data from Flink. I am getting the below error in Flink 1.17. The same code works in Flink 1.16. Could someone please help? Caused by: java.lang.IllegalStateException: Delegation token provider with service name {} has multiple implementations [s3]

Re: emitValueWithRetract issue

hi Adam As far as I know, there is currently no similar API available, but I believe that this feature was accidentally removed and we should add it back. I have created a Jira to track the progress of this feature. https://issues.apache.org/jira/browse/FLINK-31788 On Tue, Apr 11, 2023 at 12:10

Re: emitValueWithRetract issue

Many thanks for the sanity check, Feng. It’s a shame this well-documented feature was silently removed. emitValue() creates an unreasonable amount of unnecessary and disruptive chatter on the changelog stream, as evidenced by putting a print table after the flatAggregate. Lots of -D/+I RowData pai

Re: emitValueWithRetract issue

hi Adam I have checked the code and indeed this feature is not available in the latest version of Flink code. This feature was originally implemented in the old planner: https://github.com/apache/flink/pull/8550/files However, this logic was not

emitValueWithRetract issue

The TableAggregateFunction javadocs indicate that either "emitValue" or "emitUpdateWithRetract" is required. But if I implement my TableAggregateFunction with "emitUpdateWithRetract", I get a validation error. If I implement both methods it works, but emitUpdateWithRetract is not used. Peering in

Re: Flink 1.17 upgrade issue when using azure storage account for checkpoints/savepoints

Hi, This is tracked under https://issues.apache.org/jira/browse/FLINK-31612 and a fix has been merged and will be made available when the first patch version for Flink 1.16.1 will be released. Best regards, Martijn On Sat, Mar 25, 2023 at 9:37 AM ChangZhuo Chen (陳昌倬) wrote: > On Sat, Mar 25,

Re: Issue with the flink version 1.10.1

Hi Kiran According to your description, it sounds like a configuration issue prevented the job from being submitted successfully. You can check whether there are any error messages from the submission log of yarn or your submission system. You can also compare the configuration differences

Re: Issue with the flink version 1.10.1

Hi, Kiran To be honest, both 1.10 and 1.9 are pretty old version, it’s hard to fix and release a bugfix version for 1.10.1 even the community help troubleshoot your issue. So, could you try latest versions like Flink 1.16.1 or 1.17.0? Best, Leonard > On Mar 27, 2023, at 8:28 PM, Kiran Ku

Re: Issue with the flink version 1.10.1

Hi Kiran, it's really hard to come up with an answer based on your description. Usually, it helps to share some logs with the exact error that's appearing and a clear description on what you're observing and what you're expecting. A plain "no jobs are running" is too general to come up with a concl

Issue with the flink version 1.10.1

When I submit a job using flink version 1.10.1 , it is not upadating the jobs that are running and completed successfully in the Web UI of YARN resource manager . But When I use flink version 1.9.3 it is working fine and I am able to see the jobs that are running and completed in YARN resource man

Re: Flink 1.17 upgrade issue when using azure storage account for checkpoints/savepoints

On Sat, Mar 25, 2023 at 02:01:24PM +0530, Jessy Ping wrote: > Root cause: Caused by: java.util.concurrent.CompletionException: > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not found We have similar error

Flink 1.17 upgrade issue when using azure storage account for checkpoints/savepoints

Hi Team. The application failed to start after upgrading from Flink 1.16.1 to 1.17.0 in both kubernetes and docker. I didn't make any changes in flink configurations related to savepoints and checkpoints. Root cause: Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeExcepti

Re: Issue with de-serializing CompiledPlan and UNNEST_ROWS in Table API

Hi Daniel, Thanks for reporting this issue. According to the FLIP [1], this should be a bug, and I've created a Jira ticket [2] to track this. > We will introduce a declarative concept to `BuiltInFunctionDefinitions` > and `FlinkSqlOperatorTable` that maintain a function name +

Re: Issue with de-serializing CompiledPlan and UNNEST_ROWS in Table API

: "Daniel Henneberger" 收件人: "User" 发送时间: 星期三, 2023年 2 月 22日 上午 5:35:02 主题: Issue with de-serializing CompiledPlan and UNNEST_ROWS in Table API Dear Apache Flink community, I could use some help with a serialization issue I'm having while using the Table API. Sp

Issue with de-serializing CompiledPlan and UNNEST_ROWS in Table API

Dear Apache Flink community, I could use some help with a serialization issue I'm having while using the Table API. Specifically, I'm trying to deserialize a serialized CompiledPlan, but I'm running into trouble with the UNNEST_ROWS operation. It seems that the CompilePlan des

Re: SNI issue

Hi, I didn't face this issue, and I'm guessing it might have something to do with the configuration of SSL[1], have you configured the "security.ssl.rest.enabled" option? [1] https://cnightlies.apache.org/flink/flink-docs-master/docs/deployment/security/security-ssl/#configuri

SNI issue

Hello, When using the job manager API with an https proxy that uses SNI in front to route the traffic, I get an issue because the flink cli doesn't use the SNI when calling in https the API. Did other user face this issue ? Regards

Re: Facing Issue in running Python Flink Program in flink cluster (Version=1.15.2)

-libraries harshit.varsh...@iktara.ai 于2022年11月21日周一 20:51写道： > Dear Team, > > I am facing a issue while running pyflink program in flink cluster as it > stop running while reading the machine learning model > > > > This is the error : > > > > ./bin/flink run

Facing Issue in running Python Flink Program in flink cluster (Version=1.15.2)

Dear Team, I am facing a issue while running pyflink program in flink cluster as it stop running while reading the machine learning model This is the error : ./bin/flink run --python /home/desktop/ProjectFiles/test_new.py Job has been submitted with JobID

Re: Strange issue with exactly once checkpoints and the kafka sink

As noted in the SO, it's a bit confusing to me how the `checkpointing.mode` delivery guarantees with the ones for the different sinks, and in particular with the kafka one. Based on the error I had, I understand that if I use `EXACTLY_ONCE` for the checkpoints and I indicate nothing in the kafka s

Re: Dependency resolution issue with apache-flink 1.16.0 python package.

Hi Yogi, I think the problem comes from poetry depending on the metadata in PyPI. This problem has been reported in https://issues.apache.org/jira/browse/FLINK-29817 and I will fix it in 1.16.1. Best, Xingbo Yogi Devendra 于2022年11月17日周四 06:21写道： > Dear community/maintainers, > > Thanks for the

Dependency resolution issue with apache-flink 1.16.0 python package.

Dear community/maintainers, Thanks for the lovely platform of Apache Flink. I got following error when add apache-flink 1.16.0 dependency in my python project. Given below is the stack trace for further investigation. When I tried using lower version (1.15.2) for the same; I was able to move forw

  1   2   3   4   5   6   7   8   9   10   >