Re: OneInputStreamOperatorTestHarness.snapshot doesn't include timers?

No problem :) You motivated me to do a fix for that, since I stumbled across this bug/issue myself before and also took me some time in the debugger to find the cause. Piotrek > On 16 Aug 2018, at 20:05, Ken Krugler wrote: > > Hi Piotr, > > Thanks, and darn it that’s something I should have

Re: Need a clarification about removing a stateful operator

The state won't exist in the snapshot. On 17.08.2018 04:38, Tony Wei wrote: Hi all, I'm confused about the description in documentation. [1] * *Removing a stateful operator:*The state of the removed operator is lost unless another operator takes it over. When starting the upgraded

Re: processWindowFunction

Hi Vino thank you for the information, actually I am using a trigger alert and processWindowFunction to send my results, but when my window slides or ends it sends again the objects and I an getting duplicated data El jue., 16 ago. 2018 a las 22:05, vino yang () escribió: > Hi Antonio, > > What r

Re: Kafka connector issue

It looks like it is some issue with backpressure as the same behavior happens with the client library as a custom source. Michael > On Aug 16, 2018, at 6:59 PM, TechnoMage wrote: > > I have seen this in the past and running into it again. > > I have a kafka consumer that is not getting all th

Need a clarification about removing a stateful operator

Hi all, I'm confused about the description in documentation. [1] - *Removing a stateful operator:* The state of the removed operator is lost unless another operator takes it over. When starting the upgraded application, you have to explicitly agree to discard the state. Does that

Looking for flink code example using flink-jpmml library over DataStream

Hi, We are planning to use flink to run jpmml models using flink-jpmml library from (radicalbit) over DataStream in Flink. Is there any code example which we can refer to kick start the process ? Thanks,

Re: processWindowFunction

Hi Antonio, What results do not you want to get when creating each window? Examples of the use of ProcessWindowFunction are included in many test files in Flink's project, such as SideOutputITCase.scala or WindowTranslationTest.scala. For more information on ProcessWindowFunction, you can refer t

Re: watermark does not progress

Hi Jo, Thanks for letting us know. Best, Hequn On Fri, Aug 17, 2018 at 2:12 AM, John O wrote: > Just wanted to post an update on this. > > > > Problem was my dataset. I was using a kafka topic with multiple partitions > but only generated data for a single key. This meant that in a > paralleli

Re: Flink CLI does not return after submitting yarn job in detached mode

Hi Madhav, Can you set the log level to DEBUG in the log4j-client configuration file? Then post the log. I can try to locate it through the log. Thanks, vino. makelkar 于2018年8月17日周五 上午1:27写道： > Hi Vino, >We should not have to specify class name using -c option to run > job in detac

Kafka connector issue

I have seen this in the past and running into it again. I have a kafka consumer that is not getting all the records from the topic. Kafka conforms there are 300k messages in each partition, and flink only sees a total of 8000 records in the source. Kafka is 2.0, flink is 1.4.2 connector is Fli

processWindowFunction

Hello I am implementing a data stream where I use sliding windows but I am stuck because I need to set values to my object based on some if statements in my process function and send the object to the next step but I don't want results every time a window is creating if anyone has a good example

InvalidTypesException: Type of TypeVariable 'K' in 'class X' could not be determined

Hello, I have some code which compiles correctly (Flink 1.4) under Java 8. It uses generic types. While it compiles correctly, the execution fails with the error: org.apache.flink.api.common.functions.InvalidTypesException: Type of TypeVariable 'K' in 'class X' could not be determined. This is m

Running Flink in multiple AWS availability zones

Hi all, I'm looking to learn if/how others are running Flink jobs in such a way that they can survive failure of a single Amazon AWS availability zone. If you're currently doing this I would love a reply detailing your setup. Thanks! -Jamie

RE: watermark does not progress

Just wanted to post an update on this. Problem was my dataset. I was using a kafka topic with multiple partitions but only generated data for a single key. This meant that in a parallelism>1 environment, some sources will never get any data and watermark. After “keyby”, the next processor will

Re: OneInputStreamOperatorTestHarness.snapshot doesn't include timers?

Hi Piotr, Thanks, and darn it that’s something I should have noticed. — Ken > On Aug 16, 2018, at 4:37 AM, Piotr Nowojski wrote: > > Hi, > > You made a small mistake when restoring from state using test harness, that I > myself have also done in the past. Problem is with an ordering of thos

Re: Flink CLI does not return after submitting yarn job in detached mode

Hi Vino, We should not have to specify class name using -c option to run job in detached mode. I tried that this morning but it also didn't work. flink CLI always starts in interactive mode, and somehow ignores option -yd specified in yarn-cluster mode. Can someone verify thi

Re: Scala 2.12 Support

Hi Aaron, we just released Flink 1.6 and the discussion for the roadmap of 1.7 will begin soon. I guess the Jira issue will also updated then. I would recommend to watch it for now. Regards, Timo Am 16.08.18 um 17:08 schrieb Aaron Levin: Hi Piotr, Thanks for the update. Glad to hear it's

Re: Scala 2.12 Support

Hi Piotr, Thanks for the update. Glad to hear it's high on the priority list! I'm looking forward to the 1.7 update! It may be worth having someone more official from the Flink team give an update on that ticket. It wasn't clear if the 1.7 comment from that user was just a reference to the fact t

Re: How to submit flink job on yarn by java code

Hi, Is this path accessible on the container? If not, use some distributed file system, nfs or -yt —yarnship option of the cli. Please also take a look at https://lists.apache.org/thread.html/%3CCAF=1nJ8GONoqux7czxpUxAf7L3p=-E_ePSTHk0uWa=GRyG=2...@mail.gmail.com%3E

Re: How to compare two window ?

Hi, Could you rephrase your question? Maybe by posting some code examples? Piotrek > On 16 Aug 2018, at 08:26, 苗元君 wrote: > > Hi, Flink guys, > U really to a quick release, it's fantastic ! > > I'v got a situation , > window 1 is time driven, slice is 1min, trigger is 1 count > window 2 is

Re: How to compare two window ?

Hi miaoyuan, > First question I didn't quite catch your meaning. There are some documents about using window in sql[1] or table-api[2] and it is worth to take a look. > Second question Group by window without other keys results a AllWindowedStream. Sql like bellow: > SELECT COUNT(*) > , TUMBLE_S

Re: Standalone cluster instability

Hi, I’m not aware of such rules of thumb. Memory consumption is highly application and workload specific. It depends on how much things you allocate in your user code and how much memory do you keep on state (in case of heap state backend). Basically just as with most java applications, you hav

Re: OneInputStreamOperatorTestHarness.snapshot doesn't include timers?

Hi, You made a small mistake when restoring from state using test harness, that I myself have also done in the past. Problem is with an ordering of those calls: result.open(); if (savedState != null) { result.initializeState(savedState); } Open is supposed to

Re: Standalone cluster instability

Thank you for your help Piotrek. I think it was a combination of a. other processes taking up available memory and b. flink processes consuming all the memory allocated to them, that resulted in kernel running out of memory. Are there any heuristics or best practices which you (or anyone in the c

Re: Flink SQL does not support rename after cast type

Hi Hequn, Thanks a lot for your anwswer! The question is clear now. Best Henry > 在 2018年8月14日，下午1:24，Hequn Cheng > 写道： > > Hi Henry, > > Flink does support rename column after casting. > > The exception is not caused by cast. It is caused by mixing of type

Re: Scala 2.12 Support

Hi, Scala 2.12 support is high on our priority list and we hope to have it included for the 1.7 release (as you can see in the ticket itself), which should happen later this year. Piotrek > On 15 Aug 2018, at 17:59, Aaron Levin wrote: > > Hello! > > I'm wondering if there is anywhere I can

Re: How to submit flink job on yarn by java code

Sorry, I don't know why the code and error are not visible. The error is : The program finished with the following exception: /org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster. at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(