Re: Window is not working in streaming

Hi Taher, Thanks for the quick response but, if you tell the problem in my code then it would be a great help. Thanks, Abhijeet Kumar Software Development Engineer, Sentienz Solutions Pvt Ltd Cognitive Data Platform - Perceive the Data ! abhijeet.ku...@sentienz.com

Re: Window is not working in streaming

Which version of Flink are you using? The method was deprecated in your code, as per the new syntax this is the correct way to extract timestamp with periodic watermarks. On Mon 26 Nov, 2018, 1:45 PM Abhijeet Kumar Hi Taher, > > Thanks for the quick response but, if you tell the problem in my cod

Re: Flink Exception - assigned slot container was removed

Thanks for the suggestion Qi. I tried increasing slot.idle.timeout to 360 but it seems to still have encountered the issue. Does this mean if a slot or "flink worker" has not processed items for 1 hour, that it will be removed? Would any other flink configuration properties help for this? s

Re: 回复：Flink job failing due to "Container is running beyond physical memory limits" error.

I am also experiencing this error message "Container is running beyond physical memory limits". In my case, I am using Flink 1.5.2 with 10 task managers, with 40 slots for each task manager. The memory assigned during flink cluster creation is 1024MB per task manager. The checkpoint is using Roc

Re: Flink Exception - assigned slot container was removed

In addition, after the Flink job has failed from the above exception, the Flink job is unable to recover from previous checkpoint. Is this the expected behavior? How can the job be recovered successfully from this? ‐‐‐ Original Message ‐‐‐ On Monday, November 26, 2018 12:30 AM, Flink Dev

Re: 回复：Flink job failing due to "Container is running beyond physical memory limits" error.

Also, after the Flink job has failed from the above error, the Flink job is unable to recover from previous checkpoint. Is this the expected behavior? How can the job be recovered successfully from this? ‐‐‐ Original Message ‐‐‐ On Monday, November 26, 2018 12:35 AM, Flink Developer wr

Re: understadning kafka connector - rebalance

Ok, thanks for the clarification. but if I use it with keyed state so the partition is by the key. rebalancing will not shuffle this partitioning ? e.g .addSource(source) .rebalance .keyBy(_.id) .mapWithState(...) On Mon, Nov 26, 2018 at 8:32 AM Taher Koitawala wrote: > Hi Avi

回复：回复：Flink job failing due to "Container is running beyond physical memory limits" error.

It may work aournd by increasing the task manager memory size. The recover failure is up to serveral issues, whether it had successful checkpoint before, the states are available and what is the failover strategy? Best， Zhijiang --

Re: understadning kafka connector - rebalance

You can use rebalance before keyBy because rebalance returns DataStream. The API does not allow rebalance on keyedStreamed which is returned after keyBy so you are safe. On Mon 26 Nov, 2018, 2:25 PM Avi Levi Ok, thanks for the clarification. but if I use it with keyed state so the > partition is

Re: CEP Dynamic Patterns

Hi Steve, No this feature has not been contributed yet. Best, Fabian Am Fr., 23. Nov. 2018 um 20:58 Uhr schrieb Steve Bistline < srbistline.t...@gmail.com>: > Have dynamic patterns been introduced yet? > > Steve >

RE: CEP Dynamic Patterns

Hi What do you mean by “dynamic pattern” ? How it differs from “static” pattern ? BR Esa From: Fabian Hueske Sent: Monday, November 26, 2018 12:01 PM To: srbistline.t...@gmail.com Cc: user Subject: Re: CEP Dynamic Patterns Hi Steve, No this feature has not been contributed yet. Best, Fabia

Re: Joining more than 2 streams

Hi, Yes, your reasoning is correct. If you use two binary joins, the data of the first two streams will be buffered twice. Unioning all three streams and joining them in a custom ProcessFunction would reduce the amount of required state. Best, Fabian Am Sa., 24. Nov. 2018 um 14:08 Uhr schrieb Ga

Re: understadning kafka connector - rebalance

Hi, DataStream x = ... x.rebalance().keyBy() is not a good idea. It will first distribute the records round-robin (over the network) and subsequently partition them by hash. The first shuffle is unnecessary. It does not have any effect because it is undone by the second partitioning. Btw. any m

Re: Flink Exception - assigned slot container was removed

This is weird. Could you paste your entire exception trace here? > On Nov 26, 2018, at 4:37 PM, Flink Developer > wrote: > > In addition, after the Flink job has failed from the above exception, the > Flink job is unable to recover from previous checkpoint. Is this the expected > behavior? Ho

Re: understadning kafka connector - rebalance

Thanks, that makes sense ! On Mon, Nov 26, 2018 at 1:06 PM Fabian Hueske wrote: > Hi, > > DataStream x = ... > x.rebalance().keyBy() > > is not a good idea. > > It will first distribute the records round-robin (over the network) and > subsequently partition them by hash. > The first shuffle is u

Flink join stream where one stream is coming 5 minutes late

Hello Team, I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether on

Re: Flink join stream where one stream is coming 5 minutes late

May I ask why you want to have 2 differences between window time? What's the use case? On Mon 26 Nov, 2018, 5:53 PM Abhijeet Kumar Hello Team, > > I've to join two stream where one stream is coming late. So, I planned > doing it by creating two windows, for first window the size will be 5 > minut

Re: Tentative release date for 1.6.3

Hi vino, We want to upgrade flink version to 1.6 I searched but couldn't find, is there a release date to v1.7? Thanks! Alon -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Tentative release date for 1.6.3

+1. Do not see 1.7 release details anywhere. On Mon, Nov 26, 2018, 8:02 AM galantaa Hi vino, > We want to upgrade flink version to 1.6 > > I searched but couldn't find, is there a release date to v1.7? > > Thanks! > Alon > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336

Re: Flink join stream where one stream is coming 5 minutes late

I already mentioned above, one is coming late by 5 mins. Maybe my approach isn't correct, and I asked to correct me if I'm wrong. > On 26-Nov-2018, at 5:56 PM, Taher Koitawala wrote: > > May I ask why you want to have 2 differences between window time? What's the > use case? > > On Mon 26 No

Re: Flink join stream where one stream is coming 5 minutes late

Hi Abhijeet, If you want to perform window-join in the DataStream API, the window configurations on both sides must be exactly the same. For your case, maybe you can try adding a 5 mins delay on event times (and watermarks) of the faster stream. Hope that helps. Best, Xingcan > On Nov 26,

Re: Flink join stream where one stream is coming 5 minutes late

Thanks for the response, Xingcan. I got your point and the whole idea. > On 27-Nov-2018, at 12:15 AM, Xingcan Cui wrote: > > Hi Abhijeet, > > If you want to perform window-join in the DataStream API, the window > configurations on both sides must be exactly the same. > > For your case, maybe

Re: how to override s3 key config in flink job

Did you try "-Dkey=value"? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: your advice please regarding state

General approach#1 is ok, but you may have to use some hash based key selector if you have a heavy data skew. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: how to override s3 key config in flink job

Hi yinhua, I didn't try this yet, but I didn't see this option in both flink cli tool and rest api either. Could you please provide more details about how to use this option to submit flink application? BTW, we are using standalone session cluster, not yarn session cluster. And I need to submit d

Re: Store Predicate or any lambda in MapState

Any further help on this? Jayant Ameta On Wed, Nov 21, 2018 at 4:37 PM Jayant Ameta wrote: > Here are the error logs. > > First error log was encountered when getting the values from the MapState. > > java.lang.ClassNotFoundException: > com.test.MatcherFactory$$Lambda$879.1452224137 > a

Re: where can I see logs from code

The code running in your main method will output to flink cli log, others like map function etc will output to task manager log. Are you saying that you only see flink code in http://SERVERADD/#/taskmanager/TM_ID/log? It might be useful if you elaborate your environment. -- Sent from: http://ap

Re: how to override s3 key config in flink job

Which flink version are you using. I know how it works in yarn, but not very clear with standalone mode. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: how to override s3 key config in flink job

Hi yinhua, Our flink version is 1.6.0. Best, Tony Wei yinhua.dai 於 2018年11月27日 週二 下午2:32寫道： > Which flink version are you using. > I know how it works in yarn, but not very clear with standalone mode. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com