[state.checkpoints.num-retained ]The default value does not take effect

hi, The configration is valid only when I specify -Dstate.cache. num-retained=n by command line interface. If I do not specify this configuration, the default value does not take effect , is it a bug ? my flink version : flink-1.13.3

Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

Wouldn't it be better to ask the Iceberg maintainers to support dynamic schema change? On Fri, Dec 17, 2021 at 3:03 AM Dong Lin wrote: > Hi Ayush, > > Your use-case should be supported. Sorry, we don't have a good way to > support this in Flink 1.14. > > I am going to propose a FLIP to fix it i

Re: [ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

I guess this is related to publishers everywhere are updating their artifacts in response to the log4shell vulnerability[1]. All we can do and need to do is wait. ☕️ Best, Leonard [1] https://issues.sonatype.org/browse/OSSRH-76300 > 2021年12月17

Re: [ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

Not found in https://repo1.maven.org/maven2/org/apache/flink/flink-table-api-java/ I guess too many people sent versions, resulting in maven central repository synchronization being slower. Best, Jingsong On Fri, Dec 17, 2021 at 2:00 PM casel.chen wrote: > > I can NOT find flink 1.13.5 rel

Re:[ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

I can NOT find flink 1.13.5 related jar in maven central repository, did you upload them onto there already? Thanks! At 2021-12-17 01:26:19, "Chesnay Schepler" wrote: >The Apache Flink community has released emergency bugfix versions of >Apache Flink for the 1.11, 1.12, 1.13 and

Re: Flink fails to load class from configured classpath using PipelineOptions

The config option "pipeline.jars" is used to specify the user jar, which contains the main class. I think what you need is "pipeline.classpaths". /** * A list of URLs that are added to the classpath of each user code classloader of the program. * Paths must specify a protocol (e.g. file://) and

Re: unexpected result of interval join when using sql

Hi! Thanks for raising this issue. This is unfortunately a bug. I've created a JIRA ticket [1] and you can track the progress of this issue there. [1] https://issues.apache.org/jira/browse/FLINK-25357 Schwalbe Matthias 于2021年12月16日周四 14:51写道： > Probably an oversight … did you actually mean to

Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

Hi Ayush, Your use-case should be supported. Sorry, we don't have a good way to support this in Flink 1.14. I am going to propose a FLIP to fix it in Flink 1.15. Thanks, Dong On Thu, Dec 9, 2021 at 7:11 PM Ayush Chauhan wrote: > My usecase is that as soon as the avro message version is chan

Re: Antlr usage in FLink

Hi! Are you talking about the java code splitter module? Flink will change user SQL into several Java classes and run them in distributed JVMs. However due to some constraints in JVM, a single Java method can't be longer than 64KB (in byte code). This constraint is easily exceeded with a very com

Re: Unified Source Interface in flink 1.12

Hi! It is possible for Flink 1.12. A major example is the Hive source [1]. [1] https://issues.apache.org/jira/browse/FLINK-19888 Krzysztof Chmielewski 于2021年12月17日周五 06:56写道： > Hi, > I know that FLIP-27 [1] was released in version 1.12 and I know that > currently (version 1.14) we can easily u

Unified Source Interface in flink 1.12

Hi, I know that FLIP-27 [1] was released in version 1.12 and I know that currently (version 1.14) we can easily use a custom source connector that implements new unified source interface as a corner stone for Table Source Connector in SQL Api. My question is, does version 1.12 also allows for usin

Antlr usage in FLink

Hi, I was cruising through Flink's source code and I have noticed that one of the modules contains a lexer and parser g4 files for Java. I'm fairly familiar with Antlr4 btw. and I was wondering for what Flink uses Antlr4 with Java g4 files. Regards, Krzysztof Chmielewski

Flink fails to load class from configured classpath using PipelineOptions

I am developing a Java application which uses UDFs on Flink 1.14. It uses PipelineOptions.JARS config to add jar files, containing UDF classes, dynamically to the user classpath in the main method; However the application fails to load UDF class from configured jar files at job launch time with and

Re: [EXTERNAL] Re: Periodic Job Failure

Worth a shot - might explain the variability thanks for sending this over! Get Outlook for iOS ___ Julian Cardarelli CEO T (800) 961-1549 ejul...@thentia.com LinkedIn DISCLAIMER ​ ​Neither Thentia Corporation, nor its directors, officers, sh

Re: [EXTERNAL] Re: Periodic Job Failure

A shot in the dark but could it be this: https://mux.com/blog/5-years-of-flink-at-mux/ ? > The JVM will cache DNS entries forever by default. This is undesirable in Kubernetes deployments where there’s an expectation that DNS entries can and do change frequently as pod deployments move between

Re: [EXTERNAL] Re: Periodic Job Failure

So it connects to http rest based micro services and they are outside a Kubernetes HA setup for flink. All of a sudden and it’s arbitrary not consistent it could be 10 days it could be 28 days, the calls stop going out on this one job but not others. Recycling it brings it back. But the job an

[ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

The Apache Flink community has released emergency bugfix versions of Apache Flink for the 1.11, 1.12, 1.13 and 1.14 series. These releases include a version upgrade for Log4j to address [CVE-2021-44228](https://nvd.nist.gov/vuln/detail/CVE-2021-44228) and [CVE-2021-45046](https://nvd.nist.gov/

Re: How do I determine which hardware device and software has log4j zero-day security vulnerability?

I think this is meant for the Apache log4j mailing list [1]. [1] https://logging.apache.org/log4j/2.x/mail-lists.html On Thu, Dec 16, 2021 at 4:07 PM David Morávek wrote: > Hi Turritopsis, > > I fail to see any relation to Apache Flink. Can you please elaborate on > how Flink fits into it? > >

Re: Information request: Reactive mode and Rescaling

Hi Morgan, Regarding your first question, if the Kafka connector is configured to use exactly-once semantics it will check the offsets of partitions when recovering from the checkpoint, so there will be no data loss or duplication. I'm not quite sure I understood the second part of the first quest

Re: How do I determine which hardware device and software has log4j zero-day security vulnerability?

Hi Turritopsis, I fail to see any relation to Apache Flink. Can you please elaborate on how Flink fits into it? Best, D. On Thu, Dec 16, 2021 at 3:52 PM Turritopsis Dohrnii Teo En Ming < ceo.teo.en.m...@gmail.com> wrote: > Subject: How do I determine which hardware device and software has > log

How do I determine which hardware device and software has log4j zero-day security vulnerability?

Subject: How do I determine which hardware device and software has log4j zero-day security vulnerability? Good day from Singapore, I am working for a Systems Integrator (SI) in Singapore. We have several clients writing in, requesting us to identify log4j zero-day security vulnerability in their

Re: pyFlink + asyncio

Hi Михаил, >From looking at https://nightlies.apache.org/flink/flink-docs-master/api/python//pyflink.datastream.html there is currently no AsyncFunction / RichAsyncFunction implementation in pyFlink, so you are bound to synchronously interacting. Best regards, Alexander On Thu, Dec 16, 2021 at 1

Re: Confusion about rebalance bytes sent metric in Flink UI

Hi Tao, Could you prepare a minimalistic example that would reproduce this issue? Also what Flink version are you using? Best, Piotrek czw., 16 gru 2021 o 09:44 tao xiao napisał(a): > >Your upstream is not inflating the record size? > No, this is a simply dedup function > > On Thu, Dec 16, 202

pyFlink + asyncio

Hi team!https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/Is it possible to use this for pyFlink?Or another asynchronous enrichment of an unordered data stream?

Re: CVE-2021-44228 - Log4j2 vulnerability

We will announce the releases when the binaries are available. On 16/12/2021 05:37, Parag Somani wrote: Thank you Chesnay for expediting this fix...! Can you suggest, when can I get binaries for 1.14.2 flink version? On Thu, Dec 16, 2021 at 5:52 AM Chesnay Schepler wrote: We will push

Re: using flink retract stream and rockdb, too many intermediateresult of values cause checkpoint too heavy to finish

Hi Arvid Heise, Thanks for your reply! It's not classical sensor aggregation. The reason for not using window join is the very long time gap between patient's behaviors. There is a long gap of days even months between the appointment of doctor and the visit, and between tests and betwee

Svar: WindowOperator TestHarness

Hi Timo, And thank you for the detailed answer. We chose to go for the second alternative using the following: import org.apache.flink.streaming.api.transformations.OneInputTransformation import org.apache.flink.streaming.api.windowing.assigners.SlidingEventTimeWindows val env = StreamExecutionE

Re: Confusion about rebalance bytes sent metric in Flink UI

>Your upstream is not inflating the record size? No, this is a simply dedup function On Thu, Dec 16, 2021 at 2:49 PM Arvid Heise wrote: > Ah yes I see it now as well. Yes you are right, each record should be > replicated 9 times to send to one of the instances each. Your upstream is > not inflat

unexpected result when using sql interval join ability

Hi Flink 1.14.0 Scala 2.12 I'm using flink sql interval join ability, here is my table schema and sql create table `queue_3_ads_ccops_perf_o_ebs_volume_capacity` ( `dtEventTime` timestamp(3), `dtEventTimeStamp` bigint, `sourceid` string, `cluster_name` string, `poolname` string, `storage_pool

Re: unaligned checkpoint for job with large start delay

Hi Mason, In Flink 1.14 we have also changed the timeout behavior from checking against the alignment duration, to simply checking how old is the checkpoint barrier (so it would also account for the start delay) [1]. It was done in order to solve problems as you are describing. Unfortunately we ca

unexpected result of interval join when using sql

Hi Flink 1.14.0 Scala 2.12 I'm using flink sql interval join ability, here is my table schema and sql create table `queue_3_ads_ccops_perf_o_ebs_volume_capacity` ( `dtEventTime` timestamp(3), `dtEventTimeStamp` bigint, `sourceid` string, `cluster_name` string, `poolname` string, `storage_pool