Re: how to propagate watermarks across multiple jobs

Great :) Just one more note. Currently FlinkKafkaShuffle has a critical bug [1] that probably will prevent you from using it directly. I hope it will be fixed in some next release. In the meantime you can just inspire your solution with the source code. Best, Piotrek [1] https://issues.apache.o

Re: Scaling Higher than 10k Nodes

Hi Joey, Sorry for not responding to your question sooner. As you can imagine there are not many users running Flink at such scale. As far as I know, Alibaba is running the largest/one of the largest clusters, I'm asking for someone who is familiar with those deployments to take a look at this con

Re: [Flink-SQL] FLOORing OR CEILing a DATE or TIMESTAMP to WEEK uses Thursdays as week start

Thanks a lot for the added context and pointers Julian and Leonard, I've fixed it by going down to the arithmetics as suggested in one of the Calcite discussions. The changes proposed by FLIP-126 definitely look good. I'll check its details further. Best Regards, On Thu, 4 Mar 2021 at 04:18, Le

Re: Processing-time temporal join is not supported yet.

Hi, Eric > what will be the best workaround to enrich stream of data from a kafka topics > with statical data based on id? Currently you can put your statical data in Hive/JDBC/HBase which supports lookup the data in full table env as a workaround,. You can also write a UDF which caches the s3 f

Re: [Flink-SQL] FLOORing OR CEILing a DATE or TIMESTAMP to WEEK uses Thursdays as week start

Hi, Sebastián Ramírez Magrí (Sorry for wrong name in above mail) Flink follows old version calcite’s behaviour which lead to the wrong behavior. snd Julian is right that calcite has corrected FLOOR and CEIL functions in CALCITE-3412, Flink has upgraded calcite to 1.26 version which contains th

Re: [Flink-SQL] FLOORing OR CEILing a DATE or TIMESTAMP to WEEK uses Thursdays as week start

Hi, Jaffe Flink follows old version calcite’s behaviour which lead to the wrong behavior. snd Julian is right that calcite has corrected FLOOR and CEIL functions in CALCITE-3412, Flink has upgraded calcite to 1.26 version which contains the patch, what we need is only to adapt it in Flink code

Defining GlobalJobParameters in Flink Unit Testing Harnesses

Hi all, Early today I had asked a few questions regarding the use of the many testing constructs available within Flink and believe that I have things in a good direction at present. I did run into a specific case that either may not be supported, or just isn't documented well enough for me to det

pipeline.auto-watermark-interval vs setAutoWatermarkInterval

I'm hoping to have my confusion clarified regarding the settings, 1. pipeline.auto-watermark-interval https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/common/ExecutionConfig.html#setAutoWatermarkInterval-long- 2. setAutoWatermarkInterval https://ci.apache.org/p

Re: how to propagate watermarks across multiple jobs

Yes, you are right and thank you. I take a brief look at what FlinkKafkaShuffle is doing, it seems what I need and I will have a try. >

Re: [External] : Re: Need help with JDBC Broken Pipeline Issue after some idle time

Hi Qinghui, I agree. I am trying to found internal and resources on the internet to fix the issue. Idle Time Limits might be a reason. But after configuring those p

Re: 1.12.2 docker image

they should be released in a day or two. On 3/3/2021 11:18 PM, Bohinski, Kevin wrote: Hi, Are there plans to provide a docker image for 1.12.2? Best kevin

1.12.2 docker image

Hi, Are there plans to provide a docker image for 1.12.2? Best kevin

Re: Flink Zookeeper leader change v 1.9.X

1) This could occur due to a number of reasons, like processes crashing, network issues between ZK and Flink, or the JobManager being stuck in some blocking operation for a long time. You will need to take a look at the ZK/Flink logs to narrow things down. 2) For FLINK-14091 the issue was not

Re: Stop vs Cancel with savepoint

Your understanding of cancel vs stop(-with-savepoint) is correct. I agree that we should update the REST API documentation and have a section outlining the problems with cancel-with-savepoint. Would you like to open a ticket yourself? On 3/3/2021 11:16 AM, Thomas Eckestad wrote: Hi! Cancel w

Re: Unit Testing State Stores in KeyedProcessFunctions

Thanks Chesnay, I agree that output testing is more practical and far less brittle, I was just curious if support was there for it. I have a specific use case where I’m managing my own windows and may schedule something to be emitted but after some processing time delay so it could potentially

Re: Compile Error

The flink-clients dependency is correct. We will need additional information to debug the Job execution failures, because these can happen due to all kind of reasons. Things like the full stacktrace, or exceptions from the logging output. Additionally, I would recommend to base your project on

Antw: [EXT] Re: Running Apache Flink on Android

Hey, Thanks for your answer :) For my Master's thesis,I want to test and evaluate the use of CEP technologies for detecting Complex Patterns in Android sensor data (Floating Phone Data). Apache Flink offers a CEP library, so I thought it would be an interesting option. The data sources would be th

Re: Unit Testing State Stores in KeyedProcessFunctions

I do not believe this to be possible. Given that the state will likely in some form affect the behavior of the function (usually in regards to what it outputs), it may be a better idea to test for that. (I suppose you'd want tests like that anyway) On 3/3/2021 8:10 PM, Rion Williams wrote: H

Re: Flink upgrade causes operator to lose state

It is currently not possible to upgrade table API / SQL applications via savepoints. This thread may provide some more insights: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-and-checkpoints-and-savepoints-td40749.html On 3/3/2021 6:53 PM, soumoks wrote: Hi,

Re: Running Apache Flink on Android

Hi, The question would be, why do you want to do it? I think it might be possible, but probably nobody has ever tested it. Flink is a distributed system, so running it on an Android phone doesn't make much sense. I would suggest you first make your app/example work outside of Android. To make sur

Re: Flink KafkaProducer flushing on savepoints

Hi, What Flink version and which FlinkKafkaProducer version are you using? `FlinkKafkaProducerBase` is no longer used in the latest version. I would guess some older versions, and FlinkKafkaProducer010 or later (no longer supported). I would suggest either to use the universal FlinkKafkaProducer

Re: Job downgrade

If I copy class A into version 1+ it works. But it is the problem from CD perspective - I want to introduce feature which required new state: 1st I need make version 1+ with class B, but no other changes, then version 2 with class B and logic changes, upgrade job and if job doesn’t do what expec

Re: Job downgrade

Hi, I'm not sure what's the reason behind this. Probably classes are somehow attached to the state and this would explain why you are experiencing this issue. I've asked someone else from the community to chip in, but in the meantime, can not you just prepare a new "version 1" of the job, with jus

Re: how to propagate watermarks across multiple jobs

Hi, Can not you write the watermark as a special event to the "mid-topic"? In the "new job2" you would parse this event and use it to assign watermark before `xxxWindow2`? I believe this is what FlinkKafkaShuffle is doing [1], you could look at its code for inspiration. Piotrek [1] https://ci.ap

Re: Flink Metrics

Hi, 1) Do you want to output those metrics as Flink metrics? Or output those "metrics"/counters as values to some external system (like Kafka)? The problem discussed in [1], was that the metrics (Counters) were not fitting in memory, so David suggested to hold them on Flink's state and treat the m

Unit Testing State Stores in KeyedProcessFunctions

Hi all! Is it possible to apply assertions against the underlying state stores within a KeyedProcessFunction using the existing KeyedOneInputStreamOperatorTestHarness class within unit tests? Basically I wanted to ensure that if I passed in two elements each with unique keys that I would be able t

Re: Independence of task parallelism

Hi Jan, As far as I remember, Flink doesn't handle very well cases like (1-2-1-1-1) and two Task Managers. There are no guarantees how the operators/subtasks are going to be scheduled, but most likely it will be as you mentioned/observed. First task manager will be handling all of the operators, w

Re: Allocating tasks to specific TaskManagers

Hi Hyejo, I don't think it's possible. May I ask why do you want to do this? Best, Piotrek pon., 1 mar 2021 o 21:02 황혜조 napisał(a): > Hi, > > I am looking for a way to allocate each created subTask to a specific > TaskManager. > Is there any way to force assigning tasks to specific taskManager

Flink upgrade causes operator to lose state

Hi, We are upgrading several applications from Flink 1.9.1 to 1.11.2. Some of the applications written with Table API are not able start from savepoint after the upgrade and fail with the following error. Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint s3://

Compile Error

Hi, I am running a code (Example Program) from [1], and followed the [2] for the dependencies. I used this in the pom.xml: http://maven.apache.org/POM/4.0.0 "* xmlns:xsi=*"http://www.w3.org/2001/XMLSchema-instance "* xs

Re: Savepoint documentation

Thanks a million :) Le mer. 3 mars 2021 à 11:15, David Anderson a écrit : > > Out of curiosity, does it mean that savepoint created by flink 1.11 > cannot be recovered by a job running with flink 1.10 or older versions (so > downgrade is impossible)? > > That's correct. See the mailing list thre

Re: Flink 1.12 Compatibility with hbase-shaded-client 2.1 in application jar

The issue is resolved. org.apache.hbase exclusion was missing on my application pom while creating the uber jar. diff --git a/map/engine/pom.xml b/map/engine/pom.xml index 8337be031d1..8eceb721fa7 100644 --- a/map/engine/pom.xml +++ b/map/engine/pom.xml @@ -203,6 +203,7 @@

Re: Handling Data Separation / Watermarking from Kafka in Flink

When bounded Flink sources reach the end of their input, a special watermark with the value Watermark.MAX_WATERMARK is emitted that will take care of flushing all windows. One approach is to use a DeserializationSchema or KafkaDeserializationSchema with an implementation of isEndOfStream that retu

Stop vs Cancel with savepoint

Hi! Cancel with savepoint is marked as deprecated in the cli-documentation. It is not marked as deprecated in the REST-API documentation though? Is that a mistake? At least some recommendation regarding stop vs cancel would be appropriate to include in the API doc, or? As I understand, stop wi

Re: Savepoint documentation

> Out of curiosity, does it mean that savepoint created by flink 1.11 cannot be recovered by a job running with flink 1.10 or older versions (so downgrade is impossible)? That's correct. See the mailing list thread on Backwards Compatibility of Savepoints [1]. [1] http://apache-flink-mailing-list

Re: BroadcastState dropped when data deleted in Kafka

I created a new descriptor and rulestream used it in the second process function and this works fine. public static final MapStateDescriptor rulesDescriptor = new MapStateDescriptor<>( "rules", BasicTypeInfo.INT_TYPE_INFO, TypeInformation.of(Rule.class)); public static fin

Re: Processing-time temporal join is not supported yet.

Hi Leonard, Thx for your reply, Not problem to help on the JIRA topic, In my situation, in a full sql env, what will be the best workaround to enrich stream of data from a kafka topics with statical data based on id? i know how to do t in stream. eric Le sam. 27 févr. 2021 à 05:15, Leonard Xu a é

Re: Python DataStream API Questions -- Java/Scala Interoperability?

Hi Kevin, Thank you for your questions. Currently, users are not able to defined custom source/sinks in Python. This is a greate feature that can unify the end to end PyFlink application development in Python and is a large topic that we have no plan to support at present. As you have noticed tha

Re: Flink application kept restarting

I see. Thank you for the explanation. Best regards Rainie On Wed, Mar 3, 2021 at 12:24 AM Matthias Pohl wrote: > Hi Rainie, > in general buffer pools being destroyed usually mean that some other > exception occurred that caused the task to fail and in the process of > failure handling the opera

Re: Flink application kept restarting

Hi Rainie, in general buffer pools being destroyed usually mean that some other exception occurred that caused the task to fail and in the process of failure handling the operator-related network buffer is destroyed. That causes the "java.lang.RuntimeException: Buffer pool is destroyed." in your ca