Re: Jobmanager drops upon submitting a jar

you mean by “modernizing” the FlinkKafkaConsumer implementation? Could you explain a bit more? Thanks :) Chiming in just to check if there’s any issue we need to be aware of .. Cheers, Gordon On 14 April 2017 at 5:35:43 AM, amir bahmanyari (amirto...@yahoo.com.invalid) wrote: It ends up to be

Why do I need this dependency?

Hi colleagues,I build my app (fat jar) with no errors with these dependencies:       org.apache.flink   flink-connector-kafka-0.8_2.10   1.2.0  org.apache.flink   flink-java  1.2.0  org.apache.flink   flink-streaming-java_2.10  1.2.0  org.apache.flink   flink-clients_2.10  

Re: Jobmanager drops upon submitting a jar

consumer object. Any suggestions on modernizing the FlinkKafkaConsumer implementation?Thanks+regards From: wenlong.lwl To: dev@flink.apache.org; amir bahmanyari Sent: Wednesday, April 12, 2017 8:01 PM Subject: Re: Jobmanager drops upon submitting a jar Hi, amir, I think you could

Re: Jobmanager drops upon submitting a jar

:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) From: wenlong.lwl To: dev@flink.apache.org; amir bahmanyari Sent: Wednesday, April 12, 2017 8:01 PM Subject: Re: Jobman

Jobmanager drops upon submitting a jar

Hi Colleagues,I have a simple test job when I submit it to the Flink cluster the JM seems to disconnect.Its a one node cluster implemented in a VirtualBox Centos 7 VM.Flink starts fine and everything else look fine. Following is stack trace.I appreciate a feedback.Cheers 17/04/12 15:53:04 IN

Re: Flink limitations under Beam

Thanks Aljoscha,How sever of performance difference are we talking about here?  examples of "Iterations/feedback edges" and how they get applied?Thanks so much Aljoscha. From: Aljoscha Krettek To: dev@flink.apache.org; amir bahmanyari Sent: Tuesday, April 4, 2017 1:59 AM S

Flink limitations under Beam

Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner? Thanks+regards

Re: Machine Learning on Flink - Next steps

Great points to start:    - Online learning   - Offline learning with the streaming API Thanks + have a great weekend. From: Katherin Eri To: dev@flink.apache.org Sent: Friday, March 3, 2017 7:41 AM Subject: Re: Machine Learning on Flink - Next steps Thank you, Theodore. Shortly s

Re: Flink survey by data Artisans

From: Kostas Tzoumas To: "dev@flink.apache.org" ; u...@flink.apache.org Sent: Friday, November 18, 2016 7:28 AM Subject: Flink survey by data Artisans Hi everyone! The Apache Flink community has evolved quickly over the past 2+ years, and there are now many production Flink deplo

Re: [FLINK-3035] Redis as State Backend

scha...Cheers From: Ovidiu Cristian Marcu To: "dev@flink.apache.org" ; amir bahmanyari Sent: Monday, November 7, 2016 3:56 AM Subject: RE: [FLINK-3035] Redis as State Backend Thanks, how big was your state (GBs)? Can you share your benchmark/s? Best, Ovidiu -Original Mes

Re: [FLINK-3035] Redis as State Backend

FYI.I was using Redis as a state backend in my benchmarking Beam.It proved to be a bottleneck. Perhaps due to high frequency of updating state components.I replaced it with Java 8 ConcurrentHashmaps and it settled down tremendously.Amir- From: Ovidiu-Cristian MARCU To: dev@flink.apache.

Re: Flink Metrics

Hi colleagues,Is there a link that described Flink Matrices & provides example on how to utilize it pls?I really appreciate it...Cheers From: Till Rohrmann To: u...@flink.apache.org Cc: dev@flink.apache.org Sent: Monday, October 17, 2016 12:52 AM Subject: Re: Flink Metrics Hi Govin

Re: Use specific Task Manager for heavy computations

Looks like what I am currently doing, or at least close.No need to copy the big file on every node.Copy on one node. Read the data, and send it to a Kafka cluster using KafkaProducer() object.Use KafkaIO() (in case its a Beam app). Deploy to the node where the JM is running.It will be executed i

Re: Get Flink ExecutionGraph Programmatically

My only 2 cents is that when I started to turn the mem pre-allocation param, to true & #slots & #buffersI started to get all kinds of Akka & Disassociated exceptions thrown by the JM regarding the TMs...So yes, since I am also not well aware of Akka internals...I went back to my previous con

Re: Performance and Latency Chart for Flink

ptions.  So I am going incremental with that value. Have it set at 8192 (twice as much as before 4096).Thanks From: Greg Hogan To: dev@flink.apache.org; amir bahmanyari Sent: Monday, September 19, 2016 1:28 PM Subject: Re: Performance and Latency Chart for Flink My thought would

Re: Performance and Latency Chart for Flink

To: dev@flink.apache.org; amir bahmanyari Sent: Monday, September 19, 2016 12:43 PM Subject: Re: Performance and Latency Chart for Flink You will need to add the configuration parameters to your flink-conf.yaml. I believe the intent is that all configuration parameters should be listed

Re: Performance and Latency Chart for Flink

roughly 300 MiBytes for network buffers." ?Thats roughly twice as much. Am i Missing something here?I still need your help to set the accurate number for my  - taskmanager.network.numberOfBuffers = 4096. Thanks for your response Greg.Amir- From: amir bahmanyari To: "dev@

Re: Performance and Latency Chart for Flink

here 16 is number of tasks per TM, 4 is # of TMs & 4 is there in the formula.What would you set it to? Once I have that number, I will set  "taskmanager.memory.preallocate" to true & will give it another shot.Thanks Greg From: Greg Hogan To: dev@flink.apache.org; ami

Re: Performance and Latency Chart for Flink

o under Beam APIs.Which was the initial intent anyways.If you have suggestions to make improvements in the above case, I am all ears & greatly appreciate it.Cheers,Amir- From: "Chawla,Sumit" To: dev@flink.apache.org; amir bahmanyari Sent: Sunday, September 18, 2016 2:07 PM Su

Re: Performance and Latency Chart for Flink

Hi Fabian,FYI. This is report on other engines we did the same type of bench-marking.Also explains what Linear Road bench-marking is.Thanks for your help. http://www.slideshare.net/RedisLabs/walmart-ibm-revisit-the-linear-road-benchmark https://github.com/IBMStreams/benchmarks  https://www.datato

Re: Performance and Latency Chart for Flink

FYI, we, at a well known IT department, have been actively measuring Beam Flink Runner performance using MIT's Linear Road to stress the Flink Cluster servers.The results, thus far does not even come close to the previous streaming engines we have bench-marked.Our optimistic assumption was, when

Re: Flink Cluster Load Distribution Question

Hi Robert,I forwarded my Qs to the users forum...No response ...Is there anyway I can get some answers to my questions pls?I really really appreciate your help...CheersAmir- From: Robert Metzger To: "dev@flink.apache.org" ; amir bahmanyari Sent: Tuesday, September 13, 20

Flink Cluster Load Distribution Question

Hi Colleagues,Just joined this forum.I have done everything possible to get a 4 nodes Flink cluster to work peoperly & run a Beam app.It always generates system-output logs (*.out) in only one node. Its so slow for 4 nodes being there.Seems like the load is not distributed amongst all 4