Re: Data Transfer between TM should be encrypted

2016-09-14 Thread vinay patil
Hi Vijay, Did you raise the PR for this task, I don't mind testing it out as well. Regards, Vinay Patil On Tue, Aug 30, 2016 at 6:28 PM, Vinay Patil wrote: > Hi Vijay, > > That's a good news for me. Eagerly waiting for this change so that I can > integrate and test it before going live. > > Re

Re: Fw: Flink Cluster Load Distribution Question

2016-09-14 Thread amir bahmanyari
Hi Aljoscha,Experimenting on  relatively smaller file , everything fixed except KafkaIO()  vs. TextIO(), I get 50% better runtime performance in the Flink Cluster when reading tuples by TextIO().I understand the NW involvement in reading from Kafka topic etc.,  but 50% is significant.Also, I exp

Re: Fw: Flink Cluster Load Distribution Question

2016-09-14 Thread amir bahmanyari
Hi Aljoscha,The JM logs is also attached. Seems like everything is ok, assigned...to all nodes...Not sure why I dont get performance? :-(Thanks+regards,Amir- From: Aljoscha Krettek To: user@flink.apache.org; amir bahmanyari Sent: Wednesday, September 14, 2016 1:48 AM Subject: Re: Fw:

Re: Fw: Flink Cluster Load Distribution Question

2016-09-14 Thread amir bahmanyari
Hi Aljoscha,Thanks for your response. Its the same job but I am reading through TextIO() instead of a Kafka topic.I thought that would make a difference. It doesnt. Same slowness in Flink Cluster.I had sent you the code with reading from KafkaIO().Nothing different except commenting out the Kafk

CEP two transitions to the same state

2016-09-14 Thread Frank Dekervel
Hello, I'm trying to model a FSM using the flink CEP patterns. However, there is something i can't figure out as all the documentation examples are linear (either you go to the single possible next state, either no match). Suppose that two transitions lead from one state to two different states.

RemoteEnv connect failed

2016-09-14 Thread Dayong
Hi folks, I need to run a java app to submit a job to remote flink cluster. I am testing with the code at https://gist.github.com/datafibers/4b842ebc5b3c9e754ceaf78695e7567e and my comments. Thanks, Will

Re: SQL for Flink

2016-09-14 Thread Deepak Sharma
Thanks Greg . I will start picking some of them. Thanks Deepak On 14 Sep 2016 6:31 pm, "Greg Hogan" wrote: > Hi Deepak, > > There are many open tickets for Flink's SQL API. Documentation is at > https://ci.apache.org/projects/flink/flink-docs-master/dev/table_api.html. > > https://issues.apache

Re: SQL for Flink

2016-09-14 Thread Greg Hogan
Hi Deepak, There are many open tickets for Flink's SQL API. Documentation is at https://ci.apache.org/projects/flink/flink-docs-master/dev/table_api.html. https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20%22Table%20AP

Re: Streaming issue help needed

2016-09-14 Thread Márton Balassi
Dear Vaidya, This seems weird, me guess is that somehow that Time and AbstractTime implementations are not from the same Flink version. According to your Maven build you should be using Flink 0.10.2. Since then there have been changes to windowing, are you tied to that version or would it be feas

Problem with CEPPatternOperator when taskmanager is killed

2016-09-14 Thread jaxbihani
*Problem :* I have created a PatternStream with "custom type" and added an event pattern. This works fine in both local and cluster setup. But when I tried to take one of the taskmanager down (on which task was executing), flink tries to restart a job but restart fails with the exception : "Could

Re: SQL for Flink

2016-09-14 Thread Deepak Sharma
+1 Yes.I agree to having SQL for Flink. I can take up some tasks as well once this starts. Thanks Deepak On Wed, Sep 14, 2016 at 3:47 PM, Radu Tudoran wrote: > Hi, > > > > As a follow up to multiple discussions that happened during Flink Forward > about how SQL should be supported by Flink, I w

SQL for Flink

2016-09-14 Thread Radu Tudoran
Hi, As a follow up to multiple discussions that happened during Flink Forward about how SQL should be supported by Flink, I was thinking to make a couple of proposals. Disclaimer: I do not claim I have managed to synthesized all the discussions and probably a great deal of things are still miss

Re: Tumbling window rich functionality

2016-09-14 Thread Aljoscha Krettek
Hi, WindowFunction.apply() will be called once for each window so you should be able to do the setup/teardown in there. open() and close() are called at the start of processing, end of processing, respectively. Cheers, Aljoscha On Wed, 14 Sep 2016 at 09:04 Swapnil Chougule wrote: > Hi Team, > >

Re: Why tuples are not ignored after watermark?

2016-09-14 Thread Aljoscha Krettek
Hi, the problem might be that your timestamp/watermark assigner is run in parallel and that only one parallel instance of those operators emits the watermark because only one of those parallel instances sees the element with _3 == 9000. For the watermark to advance at an operator it needs to advanc

Re: Fw: Flink Cluster Load Distribution Question

2016-09-14 Thread Aljoscha Krettek
Hi, this is a different job from the Kafka Job that you have running, right? Could you maybe post the code for that as well? Cheers, Aljoscha On Tue, 13 Sep 2016 at 20:14 amir bahmanyari wrote: > Hi Robert, > Sure, I am forwarding it to user. Sorry about that. I followed the > "robot's" instru

Re: Firing windows multiple times

2016-09-14 Thread Aljoscha Krettek
Hi, yes AJ that observation is correct. Let's see what Shannon has to say about this but it might be that all "higher-level" aggregates will have to be based on the first level and can then update at the speed of that aggregate. Cheers, Aljoscha On Mon, 12 Sep 2016 at 05:03 aj.h wrote: > In the

Tumbling window rich functionality

2016-09-14 Thread Swapnil Chougule
Hi Team, I am using tumbling window functionality having window size 5 minutes. I want to perform setup & teardown functionality for each window. I tried using RichWindowFunction but it didn't work for me. Can anybody tell me how can I do it ? Attaching code snippet what I tried impressions.map(