Re: sharebuffer prune code

2018-05-28 Thread Shailesh Jain
Thanks Dawid. I'll rebase against your branch and test it. Would revert back if I hit the issue again. Regards, Shailesh On Sun, May 27, 2018 at 5:54 PM, Dawid Wysakowicz wrote: > The logic for SharedBuffer and in result for prunning will be changed in > FLINK-9418 [1]. We plan to make it backw

Re: latency critical job

2018-05-28 Thread makeyang
Rong Rong: my flink version is 1.4.2 since we are using the docker env which is sharing disk-io, based on our observation, disk-io spike cased by other process in the same physical machine can lead to long time operator processing. -- Sent from: http://apache-flink-user-mailing-list-arc

Re: latency critical job

2018-05-28 Thread makeyang
Timo: thanks for u suggestion -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: [ANNOUNCE] Apache Flink 1.5.0 release

2018-05-28 Thread Bowen Li
Congratulations, everyone! On Mon, May 28, 2018 at 1:15 AM, Fabian Hueske wrote: > Thank you Till for serving as a release manager for Flink 1.5! > > 2018-05-25 19:46 GMT+02:00 Till Rohrmann : > > > Quick update: I had to update the date of the release blog post which > also > > changed the URL.

Re: Odd job failure

2018-05-28 Thread Elias Levy
On Mon, May 28, 2018 at 1:48 AM, Piotr Nowojski wrote: > Most likely suspect is the standard java problem of some dependency > convergence issue. Please check if you are not pulling in multiple Kafka > versions into your class path. Especially your job shouldn’t pull any Kafka > library except of

Debugging window processing: can I output window start/end times, prove correctness?

2018-05-28 Thread chrisr123
I am learning the tumbling and rolling window API and I was wondering what API calls people use to determine if their events are being assigned to windows as they expect? For example, is there a way to print out the window start and and times for windows as they are being processed, and what event

Re: Clarification in TumblingProcessing TimeWindow Documentation

2018-05-28 Thread Dhruv Kumar
Sure thanks a lot! Bowen and Fabian. I will create the JIRA and also submit a PR with a better comment. -- Dhruv Kumar PhD Candidate Department of Computer Science and Engineering University of Minnesota www.dhruvkumar.me > On May 28, 2018, at 03:5

[ANNOUNCE] Final Reminder - Call for Presentations - Flink Forward Berlin 2018

2018-05-28 Thread Fabian Hueske
Hi all, This is the final reminder for the call for presentations for Flink Forward Berlin 2018. *The call closes in 7 days* (June 4, 11:59 PM CEST). Submit your talk and get to present your Apache Flink and stream processing ideas, experiences, use cases, and best practices on September 4-5 in

Re: Job execution fails when parallelism is increased beyond 1

2018-05-28 Thread HarshithBolar
I'm using Flink 1.4.2 -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Job execution fails when parallelism is increased beyond 1

2018-05-28 Thread Chesnay Schepler
Could you tell us which Flink version you are using? On 28.05.2018 14:01, HarshithBolar wrote: I'm submitting a Flink job to a cluster that has three Task Managers via the Flink dashboard. When I set `Parallelism` to 1 (which is default), everything runs as expected. But when I increase `Paralle

Job execution fails when parallelism is increased beyond 1

2018-05-28 Thread HarshithBolar
I'm submitting a Flink job to a cluster that has three Task Managers via the Flink dashboard. When I set `Parallelism` to 1 (which is default), everything runs as expected. But when I increase `Parallelism` to anything more than 1, the job fails with the exception, /java.io.FileNotFoundExcepti

ML in Streaming API

2018-05-28 Thread Thodoris Bitsakis
Hello and thanks for the subscription! I am using Streaming API to develop a ML algorithm and i would like your opinions regarding the following issues: 1) The input is read from a big size file with d-dimensional points, and i want to perform a parallel count window. In each parallel count wind

Re: Writing Table API results to a csv file

2018-05-28 Thread Chris Ruegger
Fabian, Jorn: Yes, that was indeed it. When I added the env.execute("MyApp") it worked. Thank you for your help. -Chris On Mon, May 28, 2018 at 5:03 AM, Fabian Hueske wrote: > Hi, > > Jörn is probably right. > > In contrast to print(), which immediately triggers an execution, > writeToSink()

Re: Writing Table API results to a csv file

2018-05-28 Thread Fabian Hueske
Hi, Jörn is probably right. In contrast to print(), which immediately triggers an execution, writeToSink() just appends a sink operator and requires to explicitly trigger the execution. The INFO messages of the TypeExtractor are "just" telling you, that Row cannot be used as a POJO type, but tha

Re: Clarification in TumblingProcessing TimeWindow Documentation

2018-05-28 Thread Fabian Hueske
I agree, this should be fixed. Thanks for noticing, Dhruv. Would you mind creating a JIRA for this? Thank you, Fabian 2018-05-28 8:39 GMT+02:00 Bowen Li : > Hi Dhruv, > > I can see it's confusing, and it does seem the comment should be improved. > You can find concrete explanation of tumbling w

Re: Large number of sources in Flink Job

2018-05-28 Thread Fabian Hueske
Hi Chirag, There have been some issue with very large execution graphs. You might need to adjust the default configuration and configure larger Akka buffers and/or timeouts. Also, 2000 sources means that you run at least 2000 threads at once. The FileInputFormat (and most of its sub-classes) in

Re: Odd job failure

2018-05-28 Thread Piotr Nowojski
Hi, I think that’s unlikely to happen. As far as I know, the only way to actually unload the classes in JVM is when their class loader is garbage collected, which means all the references in the code to it must vanish. In other words, it should never happen that class is not found while anyone

Re: [ANNOUNCE] Apache Flink 1.5.0 release

2018-05-28 Thread Fabian Hueske
Thank you Till for serving as a release manager for Flink 1.5! 2018-05-25 19:46 GMT+02:00 Till Rohrmann : > Quick update: I had to update the date of the release blog post which also > changed the URL. It can now be found here: > > http://flink.apache.org/news/2018/05/25/release-1.5.0.html > > Ch