Re: The problem of getting data by Rest API

The WebUI also retrieves all data from the REST API, which should be updated with a minimum interval of 10 seconds. On 5/7/2021 3:57 AM, penguin. wrote: On the Web UI page, we can see that the relevant data is updated every 3S, such as the read-bytes of each operator. But when I get data throug

Re: [ANNOUNCE] Apache Flink 1.13.0 released

Thanks, Dawid & Guowei for the great work, thanks to everyone involved. Best, Yangze Guo On Thu, May 6, 2021 at 5:51 PM Rui Li wrote: > > Thanks to Dawid and Guowei for the great work! > > On Thu, May 6, 2021 at 4:48 PM Zhu Zhu wrote: >> >> Thanks Dawid and Guowei for being the release managers

Re: The problem of getting data by Rest API

To be more precise, the update of the data is scheduled at most once every 10 seconds, but it can of course happen that the result of said update arrives in a different interval. As in, this would be possible: T00: Issue update 1 T10: Issue update 2 T12: Receive update1 T14: Receive update2 O

How to comsume and send data with two different kerberos cetified kafka in one flink task?

Hi By setting security.kerberos.* configure, we can connect one kerberos certified Kafka in Flink sql task. How to consume and produce with two different kerberos cetified Kafka in one flink sql task? Kafka allow multiple SASL authenticated Java clients in a single JVM process. https://cwik

Watermark time zone issue

Hi My friends: My watermark added 8 more hours to the timestamp displayed on the flink web. What is the reason for this? Actually looking at the data, it is correct. I don't know where the problem occurred? Is it because of the time zone? Flink 1.11.1 Best Wishes!!! forideal

Re: Watermark time zone issue

Hi, forideal It’s not because the time zone issue, the watermark value is timestamp in UTC mills, you should convert it to UTC timestamp and then compare with your data. Best, Leonard > 在 2021年5月7日，18:28，forideal 写道： > > Hi My friends: > My watermark added 8 more hours to the timestamp d

How to increase the number of task managers?

Hi community, According to the [docs]( https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/ ): > taskmanager.numberOfTaskSlots: The number of slots that a TaskManager offers *(default: 1)*. Each slot can take one task or pipeline. Having multiple slots in a TaskMan

Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

Hi , I am surfacing when submitting flink from intellij IDE . what cloud the issues. Do need to change the scala version flink 1.11.3 scala 2.11 Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V at scala.concurrent.java8.FuturesConvertersImpl$CF.(

Re: How to increase the number of task managers?

Hey num of TMs = parallelism / num of slots parallelism.default is another config you should consider. Read also https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/parallel/ [https://my-email-signature.link/signature.gif?u=1088647&e=151783011&v=daa41c995e0f03eb0cd3

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

It looks like you have different scala versions on the classpath. Please check that all your dependencies use the same scala version. On 5/7/2021 1:25 PM, Ragini Manjaiah wrote: Hi , I am surfacing when submitting flink from intellij  IDE . what cloud the issues. Do need to change the scala ve

FlieSystem Connector's Success File Was Submitted Late

Hi My friends: I use FlieSystem in Flink SQL, and I found that my success file was submitted late, probably dozens of minutes late. Here I provide some information： 1.Flink version is 1.11.1. 2.Source DDL create table test （ `timestamp bigint`, event_time as _

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

The scala version is same across the pom file . 2.11 On Fri, May 7, 2021 at 5:06 PM Chesnay Schepler wrote: > It looks like you have different scala versions on the classpath. Please > check that all your dependencies use the same scala version. > > On 5/7/2021 1:25 PM, Ragini Manjaiah wrote: >

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

Can you show us the dependency tree of your project? (If you are using maven, run "mvn dependency:tree") On 5/7/2021 2:15 PM, Ragini Manjaiah wrote: The scala version is same across the pom file . 2.11 On Fri, May 7, 2021 at 5:06 PM Chesnay Schepler > wrote: It

Re:FlieSystem Connector's Success File Was Submitted Late

I found the reason： Late data processing: The record will be written into its partition when a record is supposed to be written into a partition that has already been committed, and then the committing of this partition will be triggered again. So, I see that the success file is slower to up

Re: Read kafka offsets from checkpoint - state processor

Anyone who has tried this or can help on this. Thanks. On Thu, May 6, 2021 at 10:34 AM bat man wrote: > Hi Users, > > Is there a way that Flink 1.9 the checkpointed data can be read using the > state processor api. > Docs [1] says - When reading operator state, users specify the operator > uid,

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

Hi , Please find the details [INFO] X:XXSNAPSHOT [INFO] +- org.apache.flink:flink-java:jar:1.11.3:compile [INFO] | +- org.apache.flink:flink-core:jar:1.11.3:compile [INFO] | | +- org.apache.flink:flink-annotations:jar:1.11.3:compile [INFO] | | +- org.apache.flink:flink-metrics-core:j

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

I see a several scala 2.12 dependencies in there. [INFO] |  +- org.apache.kafka:kafka_2.12:jar:5.5.1-ccs:compile [INFO] +- io.confluent:kafka-schema-registry:jar:5.5.1:compile [INFO] |  |  \- com.kjetland:mbknor-jackson-jsonschema_2.12:jar:1.0.39:compile On 5/7/2021 2:47 PM, Ragini Manjaiah w

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

hi , true.. but iam using where scala.version is 2.11 . wondering from where this 2.12 is added org.apache.kafka kafka_${scala.version} ${kafka.version} On Fri, May 7, 2021 at 6:24 PM Chesnay Schepler wrote: > I see a several scala 2.12 dependencies in there. > > [INFO] | +- org.ap

Re: FlieSystem Connector's Success File Was Submitted Late

Hi, forideal I also encountered this problem and opened an issue[1], you can have a look. Best, Leonard [1] https://issues.apache.org/jira/browse/FLINK-22472 > 在 2021年5月7日，20:31，forideal 写道： > > I found the reason： > >Late data processing: The record will be written into its partition w

Re: Flink :Exception in thread "main" java.lang.NoSuchMethodError: scala.Function1.$init$(Lscala/Function1;)V

This is where the 2.21 dependency comes from: [INFO] +- io.confluent:kafka-schema-registry:jar:5.5.1:compile [INFO] |  +- org.apache.kafka:kafka_2.12:jar:5.5.1-ccs:compile This is the entry added by your dependency: [INFO] +- org.apache.kafka:kafka_2.11:jar:0.10.2.1:compile On 5/7/2021 3:15 PM,

Enabling Checkpointing using FsStatebackend

Hi Team, We have recently enabled checking pointing using FsStateBackend where we are trying to use S3 bucket as the persistent storage but after enabling it we are running into issues while submitting the job into the cluster. Can you please let us know if we are missing anything ? Below is th

Re: Task Local Recovery with mountable disks in the cloud

Hi Till, Thanks for getting back to me. Apologies for my delayed response. Thanks for confirming that the slot ID (Allocation ID) is indeed necessary today for task local recovery to kick in, and thanks for your insights on how to make this work. We are interested in exploring this disaggregat

Viewing the offsets stored in a Savepoint

Hello, Someone else asked this question on Stackoverflow and we would also find it very helpful: https://stackoverflow.com/questions/66256168/querying-kafka-offsets-from-flink-savepoint Is there a way to check the external savepoint data for what Kafka offset it is going to resume from? We produce

how to split a column value into multiple rows in flink sql?

Hi For example , a table like this: A B C -- a1 b1c1,c2,c3 --- how to split c1,c2,c3 into multiple rows like this in flink sql function: A B C a1 b1 c1 a1 b1 c2 a1 b1 c3 Thank you 1095193...@qq.com

Savepoint/checkpoint confusion

Hello, I got confused about usage of savepoints and checkpoints in different scenarios. I understand that checkpoints' main purpose is fault tolerance, they are more lightweight and don't support changing job graph, parallelism or state backend when restoring from them, as mentioned in the latest 1

Re: Questions about implementing a flink source

Hi Arvid, thank you so much for the detailed reply! A few replies / questions inline. Somewhat relatedly, I'm also wondering where this connector should live. I saw that there's already a pubsub connector in https://github.com/apache/flink/tree/bf40fa5b8cd5b3a8876f37ab4aa7bf06e8b9f666/flink-connec

Re: [External] : Re: StopWithSavepoint() method doesn't work in Java based flink native k8s operator

Hi Austin, Yang, Matthias, I am following up to see if you guys have any idea regarding this problem. I also created an issue here to describe the problem. [1] After looking into the source code[1], I believe for native k8s, three configuration files should be added to the flink-config- configm

Re: Question regarding cpu limit config in Flink standalone mode

Hi Xintong, Thanks for answering my question! After discussing with my teammates, we decide to rely on k8s pod and an external control plane to restrict the CPU usage of a job. Thanks again for your help! Best, Fan From: Xintong Song Sent: Thursday, May 6, 202

Re: Table name for table created fromDataStream

Hi Leonard, Yes that would be one solution. But why is it necessary to create a temporaryView from already created table ? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Enabling Checkpointing using FsStatebackend

Hi, I think the checkpointing is not the root cause of your job failure. As the log describes, your job failed caused by the authorization issue of Kafka. "Caused by: org.apache.kafka.common.errors.TransactionalIdAuthorizationException: Transactional Id authorization failed." Best, Yangze Guo On

Re: How to increase the number of task managers?

Hi, > I wonder if I can tune the number of task managers? Is there a corresponding > config? With K8S/Yarn resource provider, the task managers are allocated on demand. So, the number of them are depends on the max parallelism and the slot sharing group topology of your job. In standalone mode,

Re: Table name for table created fromDataStream

> 在 2021年5月8日，08:00，tbud 写道： > > Hi Leonard, > Yes that would be one solution. But why is it necessary to create a > temporaryView from already created table ? The name “Table” is quite misleading here, the table API object Table actually represents a relational query (e.g. Table table = m

Re: How to increase the number of task managers?

Hi Yangze, Thanks for the answer! That helps. Best, Yik San On Sat, May 8, 2021 at 10:15 AM Yangze Guo wrote: > Hi, > > > I wonder if I can tune the number of task managers? Is there a > corresponding config? > > With K8S/Yarn resource provider, the task managers are allocated on > demand. So,

What does enableObjectReuse exactly do?

I wrote a streaming job with scala, using only immutable case class. Is it safe to enable object reuse? Will it get benefits from enabling object reuse? I reached to documents but they cover neither streaming cases nor immutable data structures.

Re: [External] : Re: StopWithSavepoint() method doesn't work in Java based flink native k8s operator

Since your problem is about the flink-native-k8s-operator, let's move the discussion there. Best, Yang Fuyao Li 于2021年5月8日周六 上午5:41写道： > Hi Austin, Yang, Matthias, > > > > I am following up to see if you guys have any idea regarding this problem. > > > > I also created an issue here to describe

I call Pandas UDF N times, do I have to initiate the UDF N times?

Hi community, I am using PyFlink and Pandas UDF in my job. The job executes a SQL like this: ``` SELECT LABEL_ENCODE(a), LABEL_ENCODE(b), LABEL_ENCODE(c) ... ``` And my LABEL_ENCODE UDF is defined below: ``` class LabelEncode(ScalarFunction): def open(self, function_context): logging.inf

How to recovery from last count when using CUMULATE window after restart flink-sql job?

Hi I have tried cumalate window function in Flink-1.13 sql to accumulate data from Kafka. When I restart a cumulate window sql job, last count state is not considered and the count state accumulates from 1. Any solutions can help recovery from last count state when restarting Flink-sql job?

Re: How to recovery from last count when using CUMULATE window after restart flink-sql job?

Hi, please use user mailing list only to discuss these issues. Best, Kurt On Sat, May 8, 2021 at 1:05 PM 1095193...@qq.com <1095193...@qq.com> wrote: > Hi >I have tried cumalate window function in Flink-1.13 sql to accumulate > data from Kafka. When I restart a cumulate window sql job, las

Re: I call Pandas UDF N times, do I have to initiate the UDF N times?

Hi Yik San, Is it acceptable to rewrite the UDF a bit to accept multiple parameters and then rewrite the program as following: ``` SELECT LABEL_ENCODE(a, b, c) ... ``` Regards, Dian > 2021年5月8日 上午11:56，Yik San Chan 写道： > > Hi community, > > I am using PyFlink and Pandas UDF in my job. > >

Re: I call Pandas UDF N times, do I have to initiate the UDF N times?

Hi Dian, Thanks for pointing that out, it is a work around that I have also considered. I wonder if there is a framework level optimization on this, so that a UDF is only initiated once, no matter how many times it is called? Thank you! Best, Yik San On Sat, May 8, 2021 at 1:32 PM Dian Fu wro

Re: I call Pandas UDF N times, do I have to initiate the UDF N times?

There is still no such optimization at framework level. However, I think this maybe a good point that we could optimize. Would you like to create a ticket for this? Regards, Dian > 2021年5月8日 下午2:27，Yik San Chan 写道： > > Hi Dian, > > Thanks for pointing that out, it is a work around that I hav

Re: I call Pandas UDF N times, do I have to initiate the UDF N times?

Hi Dian, Thanks for the confirmation, I have created a ticket https://issues.apache.org/jira/browse/FLINK-22605 Best, Yik San On Sat, May 8, 2021 at 2:32 PM Dian Fu wrote: > There is still no such optimization at framework level. However, I think > this maybe a good point that we could optimiz

Re: I call Pandas UDF N times, do I have to initiate the UDF N times?

Thanks a lot~ > 2021年5月8日 下午2:41，Yik San Chan 写道： > > Hi Dian, > > Thanks for the confirmation, I have created a ticket > https://issues.apache.org/jira/browse/FLINK-22605 > > > Best, > Yik San > > On Sat, May 8, 2021 at 2:32 PM Dian Fu <