Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-20 Thread Robert Metzger
I like option 1 the most ("move to flink-core"), however, it would scatter the type extractor / type information classes accross multiple projects. Why are we not moving the entire type extractor system into "flink-core"? There were some thoughts in the past to restructure the maven module layout:

Re: Iteration stats logging

2015-06-20 Thread Robert Metzger
Are you running a fixed number of iterations or do you use a dynamic termination criterion? For fixed iterations, you can get the id of the current iteration ... which allows you to find out when you are running the last iterations. Would it be feasible for you to just log these statistics to the

Re: execute() and collect()/print()/count()

2015-06-20 Thread Robert Metzger
We could also add a link to the documentation into the exception that explains the behavior. On Fri, Jun 19, 2015 at 5:52 AM, Chiwan Park wrote: > +1 for ignoring execute() call with warning. > > But I'm concerned for how the user catches the error in program without > any data sinks. > > By the

Re: [jira] [Created] (FLINK-2252) Reduce local input split assignment log level from INFO to DEBUG

2015-06-20 Thread Robert Metzger
I don't know whether its a good idea to move a JIRA discussion to the dev@ list. I agree with Ufuk. The logs are the only way of finding out whether there is something happening or not. As I said in the JIRA, I think its totally fine to log this at INFO level. If a production user has identified t

Re: [jira] [Created] (FLINK-2252) Reduce local input split assignment log level from INFO to DEBUG

2015-06-20 Thread Ufuk Celebi
Recently I was running a job and the only way to tell how far it was and to make sure that it was still running were the JM logs with the current split assignments. I like Fabians suggestion. But as long as we don't have good runtime monitoring in place there is a valid use case for the logs (whic

Re: [jira] [Commented] (FLINK-2252) Reduce local input split assignment log level from INFO to DEBUG

2015-06-20 Thread Fabian Hueske
The ratio could be directly logged. That would be easier that estimating the ratio from potentially hundreds of log statements of several data sources. On Jun 20, 2015 22:49, "Robert Metzger (JIRA)" wrote: > > [ > https://issues.apache.org/jira/browse/FLINK-2252?page=com.atlassian.jira.plugin

Re: Note #2 on the list

2015-06-20 Thread Márton Balassi
Thanks, Ted - great news. :) @Stephan: Streaming is mentioned, but not really prominent. "... could be just the ticket for fusing streaming analytics with historical analytics." On Sat, Jun 20, 2015 at 8:23 PM, Ted Dunning wrote: > On Sat, Jun 20, 2015 at 12:52 AM, Stephan Ewen wrote: > > > T

Re: Note #2 on the list

2015-06-20 Thread Ted Dunning
On Sat, Jun 20, 2015 at 12:52 AM, Stephan Ewen wrote: > There is no comment section, otherwise > I would have added that... > Email to the author. Better coverage next time.

Re: Note #2 on the list

2015-06-20 Thread Stephan Ewen
Thanks you for forwarding this, Ted! I also found this today and was very happy to see Flink being up in the list. They labeled Flink as being mainly strong in iterations. While it is true that Flink is strong in iterations, it is also particularly strong in streaming, which was not mentioned. Th