Re: Scheduling in BATCH execution mode?

[Niklas Semmler]

Hello Ufuk,

thanks that makes very much sense :).

Best,
Niklas

On 09.09.2015 23:22, Ufuk Celebi wrote:



On 09 Sep 2015, at 19:31, Niklas Semmler  wrote:

Hello Ufuk,

thanks for you amazingly quick reply.

I have seen the markFinished in Execution.java, but if I get it right, this is 
simply used to stop a task. The ScheduleOrUpdateConsumers message in the 
pipeline case on the other hand is notifying the consumers that a pipelined 
partition is ready and can now be consumed. Can you give me a hint on how the 
receiver is notified of a consumable partition in the batch case?


Yes, this transitions the state of the respective execution to FINISHED. But if 
you look closely there is a call “finishAllBlockingPartitions”, which schedules 
the receivers, iff it the last execution finishes the  result.

You can think about the intermediate data as follows:

- On the job graph level you have: (Operator) -> [Result] -> (Operator), e.g. (map) 
-> [map result] -> (reduce)

- At runtime when there are multiple parallel tasks, you have:
   * [Operator subtask 0] -> [Result partition 0] -> (Operator subtask 0)
   * [Operator subtask 1] -> [Result partition 1] -> (Operator subtask 1)

Now, if the exchange is blocking (in batch execution mode), the result is 
finished, iff all subtasks producing it have finished.

The question now is: when is the result finished? It is finished, iff all 
subtasks producing it have finished. And that’s where the markFinished comes 
into play. The last to finish triggers the scheduling. This is 
non-deterministic, i.e. either subtask 0 or 1 can be the last to finish it. 
This is kept track of via a simple counter. The task decrementing it to 0 
triggers the scheduling.

(If the result is pipelined, the first data point triggers the scheduling 
already via the code paths you have looked at.)

Does this help?

– Ufuk



--
PhD Student / Research Assistant
INET, TU Berlin
Room 4.029
Marchstr 23
10587 Berlin
Tel: +49 30 314 75739

[jira] [Created] (FLINK-2651) Failing NettyServerLowAndHighWatermarkTest

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-2651:
--

 Summary: Failing NettyServerLowAndHighWatermarkTest
 Key: FLINK-2651
 URL: https://issues.apache.org/jira/browse/FLINK-2651
 Project: Flink
  Issue Type: Bug
Affects Versions: master
Reporter: Ufuk Celebi
Assignee: Ufuk Celebi


https://travis-ci.org/aljoscha/flink/jobs/79610050



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2652) Failing PartitionRequestClientFactoryTest

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-2652:
--

 Summary: Failing PartitionRequestClientFactoryTest
 Key: FLINK-2652
 URL: https://issues.apache.org/jira/browse/FLINK-2652
 Project: Flink
  Issue Type: Bug
Reporter: Ufuk Celebi
Priority: Minor


PartitionRequestClientFactoryTest fails when running {{mvn 
-Dhadoop.version=2.6.0 clean verify}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Releasing 0.10.0-milestone1

[Fabian Hueske]

I can take the role of the release manager and push the 0.10 milestone
release forward.

I added a list of pending fixes for 0.10 to the wiki:
https://cwiki.apache.org/confluence/display/FLINK/0.10+Release
Please extend the list (or respond to this thread) if you have more fixes
that should go into the release.

>From the currently open issues only FLINK-2632 (Web Client does not respect
the class loader of submitted jobs) is unassigned.
Anybody around to pick this one up?

Cheers, Fabian

2015-09-09 12:00 GMT+02:00 Till Rohrmann :

> I forgot to mention that there is also a bug with the `StreamFold` operator
> which we might consider fixing for the milestone release. I've opened a PR
> for it.
>
> https://issues.apache.org/jira/browse/FLINK-2631
> https://github.com/apache/flink/pull/1101
>
>
> On Wed, Sep 9, 2015 at 10:58 AM, Gyula Fóra  wrote:
>
> > This sounds good +1 from me as well :)
> > Till Rohrmann  ezt írta (időpont: 2015. szept.
> > 9.,
> > Sze, 10:40):
> >
> > > +1 for a milestone release with the TypeInformation issues fixed. I'm
> > > working on it.
> > >
> > > On Tue, Sep 8, 2015 at 9:32 PM, Stephan Ewen  wrote:
> > >
> > > > Great!
> > > >
> > > > I'd like to push one more commit later today.
> > > > A fix for https://issues.apache.org/jira/browse/FLINK-2632 would
> also
> > be
> > > > highly appreciated by some users.
> > > >
> > > > Anyone volunteering as release manager (for creating release
> candidates
> > > and
> > > > uploading them)?
> > > >
> > > >
> > > > On Tue, Sep 8, 2015 at 6:11 PM, Kostas Tzoumas 
> > > > wrote:
> > > >
> > > > > +1 for a milestone release
> > > > >
> > > > > On Tue, Sep 8, 2015 at 5:43 PM, Robert Metzger <
> rmetz...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 for a "milestone1" release. We have a lot of good features in
> > > master
> > > > > > that people can benefit from.
> > > > > >
> > > > > > On Tue, Sep 8, 2015 at 5:10 PM, Maximilian Michels <
> m...@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > +1 for releasing a milestone release soon to encourage people
> to
> > > try
> > > > > > > out the new features.
> > > > > > >
> > > > > > > There is this bug:
> > > https://issues.apache.org/jira/browse/FLINK-2632
> > > > > > > which affects the Web Client's error and results display for
> > jobs.
> > > > > > > Would be nice to fix it but IMHO it is not critical for the
> > > milestone
> > > > > > > release.
> > > > > > >
> > > > > > > On Tue, Sep 8, 2015 at 1:00 PM, Ufuk Celebi 
> > > wrote:
> > > > > > > >
> > > > > > > >> On 08 Sep 2015, at 12:01, Stephan Ewen 
> > > wrote:
> > > > > > > >>
> > > > > > > >> Hi all!
> > > > > > > >>
> > > > > > > >> Some day back we talked about releasing an 0.10.0-milestone1
> > > > > release.
> > > > > > > The
> > > > > > > >> master has advanced quite a bit (especially due to
> > > > high-availability
> > > > > > > code).
> > > > > > > >>
> > > > > > > >> I cherry picked the important additions to the
> > > > > > release-0.10.0-milestone1
> > > > > > > >> branch (fixes and Kafka consumer/producer rework).
> > > > > > > >>
> > > > > > > >> How about releasing the branch now as an intermediate
> version
> > > for
> > > > > > > people to
> > > > > > > >> try out while we stabilize the windows and HA code for the
> > 0.10
> > > > > > release?
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Thanks for cping the important changes. I’ve checked and
> there
> > is
> > > > > > > nothing I would add at this point.
> > > > > > > >
> > > > > > > > Can anybody else took a look at it? Other than that, I think
> > it’s
> > > > > good
> > > > > > > to go.
> > > > > > > >
> > > > > > > > – Ufuk
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Releasing 0.10.0-milestone1

[Robert Metzger]

Thank you very much for taking care of the release management!

I'm not sure whether it makes sense to block a "preview" release on so many
bug fixes. It feels like 80% of our commits are bugfixes or stability
improvements anyways, so we would wait infinitely to release the milestone
if we wait for all of them.

For example for the class loader issue in the web client issue, Max (who
initially brought the issue to our attention) also said: "Would be nice to
fix it but IMHO it is not critical for the milestone release."


I would vote for merging Till's https://github.com/apache/flink/pull/1101
and then start with the RC0 vote.
If there are more bug fixes available and the RC0 vote fails (which is
likely ;) ), we can include them as well.



On Thu, Sep 10, 2015 at 11:04 AM, Fabian Hueske  wrote:

> I can take the role of the release manager and push the 0.10 milestone
> release forward.
>
> I added a list of pending fixes for 0.10 to the wiki:
> https://cwiki.apache.org/confluence/display/FLINK/0.10+Release
> Please extend the list (or respond to this thread) if you have more fixes
> that should go into the release.
>
> From the currently open issues only FLINK-2632 (Web Client does not respect
> the class loader of submitted jobs) is unassigned.
> Anybody around to pick this one up?
>
> Cheers, Fabian
>
> 2015-09-09 12:00 GMT+02:00 Till Rohrmann :
>
> > I forgot to mention that there is also a bug with the `StreamFold`
> operator
> > which we might consider fixing for the milestone release. I've opened a
> PR
> > for it.
> >
> > https://issues.apache.org/jira/browse/FLINK-2631
> > https://github.com/apache/flink/pull/1101
> >
> >
> > On Wed, Sep 9, 2015 at 10:58 AM, Gyula Fóra  wrote:
> >
> > > This sounds good +1 from me as well :)
> > > Till Rohrmann  ezt írta (időpont: 2015.
> szept.
> > > 9.,
> > > Sze, 10:40):
> > >
> > > > +1 for a milestone release with the TypeInformation issues fixed. I'm
> > > > working on it.
> > > >
> > > > On Tue, Sep 8, 2015 at 9:32 PM, Stephan Ewen 
> wrote:
> > > >
> > > > > Great!
> > > > >
> > > > > I'd like to push one more commit later today.
> > > > > A fix for https://issues.apache.org/jira/browse/FLINK-2632 would
> > also
> > > be
> > > > > highly appreciated by some users.
> > > > >
> > > > > Anyone volunteering as release manager (for creating release
> > candidates
> > > > and
> > > > > uploading them)?
> > > > >
> > > > >
> > > > > On Tue, Sep 8, 2015 at 6:11 PM, Kostas Tzoumas <
> ktzou...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 for a milestone release
> > > > > >
> > > > > > On Tue, Sep 8, 2015 at 5:43 PM, Robert Metzger <
> > rmetz...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 for a "milestone1" release. We have a lot of good features
> in
> > > > master
> > > > > > > that people can benefit from.
> > > > > > >
> > > > > > > On Tue, Sep 8, 2015 at 5:10 PM, Maximilian Michels <
> > m...@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1 for releasing a milestone release soon to encourage people
> > to
> > > > try
> > > > > > > > out the new features.
> > > > > > > >
> > > > > > > > There is this bug:
> > > > https://issues.apache.org/jira/browse/FLINK-2632
> > > > > > > > which affects the Web Client's error and results display for
> > > jobs.
> > > > > > > > Would be nice to fix it but IMHO it is not critical for the
> > > > milestone
> > > > > > > > release.
> > > > > > > >
> > > > > > > > On Tue, Sep 8, 2015 at 1:00 PM, Ufuk Celebi 
> > > > wrote:
> > > > > > > > >
> > > > > > > > >> On 08 Sep 2015, at 12:01, Stephan Ewen 
> > > > wrote:
> > > > > > > > >>
> > > > > > > > >> Hi all!
> > > > > > > > >>
> > > > > > > > >> Some day back we talked about releasing an
> 0.10.0-milestone1
> > > > > > release.
> > > > > > > > The
> > > > > > > > >> master has advanced quite a bit (especially due to
> > > > > high-availability
> > > > > > > > code).
> > > > > > > > >>
> > > > > > > > >> I cherry picked the important additions to the
> > > > > > > release-0.10.0-milestone1
> > > > > > > > >> branch (fixes and Kafka consumer/producer rework).
> > > > > > > > >>
> > > > > > > > >> How about releasing the branch now as an intermediate
> > version
> > > > for
> > > > > > > > people to
> > > > > > > > >> try out while we stabilize the windows and HA code for the
> > > 0.10
> > > > > > > release?
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > Thanks for cping the important changes. I’ve checked and
> > there
> > > is
> > > > > > > > nothing I would add at this point.
> > > > > > > > >
> > > > > > > > > Can anybody else took a look at it? Other than that, I
> think
> > > it’s
> > > > > > good
> > > > > > > > to go.
> > > > > > > > >
> > > > > > > > > – Ufuk
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Releasing 0.10.0-milestone1

[Fabian Hueske]

All these bugs reported by users and its just four of them of which 2 are
already fixed.
It should be possible to fix them in a day.

2015-09-10 11:19 GMT+02:00 Robert Metzger :

> Thank you very much for taking care of the release management!
>
> I'm not sure whether it makes sense to block a "preview" release on so many
> bug fixes. It feels like 80% of our commits are bugfixes or stability
> improvements anyways, so we would wait infinitely to release the milestone
> if we wait for all of them.
>
> For example for the class loader issue in the web client issue, Max (who
> initially brought the issue to our attention) also said: "Would be nice to
> fix it but IMHO it is not critical for the milestone release."
>
>
> I would vote for merging Till's https://github.com/apache/flink/pull/1101
> and then start with the RC0 vote.
> If there are more bug fixes available and the RC0 vote fails (which is
> likely ;) ), we can include them as well.
>
>
>
> On Thu, Sep 10, 2015 at 11:04 AM, Fabian Hueske  wrote:
>
> > I can take the role of the release manager and push the 0.10 milestone
> > release forward.
> >
> > I added a list of pending fixes for 0.10 to the wiki:
> > https://cwiki.apache.org/confluence/display/FLINK/0.10+Release
> > Please extend the list (or respond to this thread) if you have more fixes
> > that should go into the release.
> >
> > From the currently open issues only FLINK-2632 (Web Client does not
> respect
> > the class loader of submitted jobs) is unassigned.
> > Anybody around to pick this one up?
> >
> > Cheers, Fabian
> >
> > 2015-09-09 12:00 GMT+02:00 Till Rohrmann :
> >
> > > I forgot to mention that there is also a bug with the `StreamFold`
> > operator
> > > which we might consider fixing for the milestone release. I've opened a
> > PR
> > > for it.
> > >
> > > https://issues.apache.org/jira/browse/FLINK-2631
> > > https://github.com/apache/flink/pull/1101
> > >
> > >
> > > On Wed, Sep 9, 2015 at 10:58 AM, Gyula Fóra  wrote:
> > >
> > > > This sounds good +1 from me as well :)
> > > > Till Rohrmann  ezt írta (időpont: 2015.
> > szept.
> > > > 9.,
> > > > Sze, 10:40):
> > > >
> > > > > +1 for a milestone release with the TypeInformation issues fixed.
> I'm
> > > > > working on it.
> > > > >
> > > > > On Tue, Sep 8, 2015 at 9:32 PM, Stephan Ewen 
> > wrote:
> > > > >
> > > > > > Great!
> > > > > >
> > > > > > I'd like to push one more commit later today.
> > > > > > A fix for https://issues.apache.org/jira/browse/FLINK-2632 would
> > > also
> > > > be
> > > > > > highly appreciated by some users.
> > > > > >
> > > > > > Anyone volunteering as release manager (for creating release
> > > candidates
> > > > > and
> > > > > > uploading them)?
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 8, 2015 at 6:11 PM, Kostas Tzoumas <
> > ktzou...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 for a milestone release
> > > > > > >
> > > > > > > On Tue, Sep 8, 2015 at 5:43 PM, Robert Metzger <
> > > rmetz...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 for a "milestone1" release. We have a lot of good features
> > in
> > > > > master
> > > > > > > > that people can benefit from.
> > > > > > > >
> > > > > > > > On Tue, Sep 8, 2015 at 5:10 PM, Maximilian Michels <
> > > m...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 for releasing a milestone release soon to encourage
> people
> > > to
> > > > > try
> > > > > > > > > out the new features.
> > > > > > > > >
> > > > > > > > > There is this bug:
> > > > > https://issues.apache.org/jira/browse/FLINK-2632
> > > > > > > > > which affects the Web Client's error and results display
> for
> > > > jobs.
> > > > > > > > > Would be nice to fix it but IMHO it is not critical for the
> > > > > milestone
> > > > > > > > > release.
> > > > > > > > >
> > > > > > > > > On Tue, Sep 8, 2015 at 1:00 PM, Ufuk Celebi <
> u...@apache.org>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> On 08 Sep 2015, at 12:01, Stephan Ewen <
> se...@apache.org>
> > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> Hi all!
> > > > > > > > > >>
> > > > > > > > > >> Some day back we talked about releasing an
> > 0.10.0-milestone1
> > > > > > > release.
> > > > > > > > > The
> > > > > > > > > >> master has advanced quite a bit (especially due to
> > > > > > high-availability
> > > > > > > > > code).
> > > > > > > > > >>
> > > > > > > > > >> I cherry picked the important additions to the
> > > > > > > > release-0.10.0-milestone1
> > > > > > > > > >> branch (fixes and Kafka consumer/producer rework).
> > > > > > > > > >>
> > > > > > > > > >> How about releasing the branch now as an intermediate
> > > version
> > > > > for
> > > > > > > > > people to
> > > > > > > > > >> try out while we stabilize the windows and HA code for
> the
> > > > 0.10
> > > > > > > > release?
> > > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > >
> > > > > > > > > > Thanks for cping the important changes. I’ve checked and
> > > 

Re: Build failure with maven-junction-plugin

[Matthias J. Sax]

Hi,

I am picking up on this old thread, because I have the same error now. I
just created a new branch from a freshly rebased master branch.

mvn -DskipTests clean install

> Failed to execute goal com.pyx4j:maven-junction-plugin:1.0.3:link (default) 
> on project flink-dist: Can't create junction source 
> [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT]
>  -> [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/../build-target] -> 
> [Help 1]

This email thread did not show a fix/reason for this issue.


-Matthias

On 06/03/2015 05:24 PM, Theodore Vasiloudis wrote:
> mvn clean package -Dmaven.javadoc.skip=true -DskipTests
> 
> Called from the flink root.
> 
> On Wed, Jun 3, 2015 at 5:21 PM, Robert Metzger  wrote:
> 
>> The WARNING is okay.
>>
>> How did you call maven, and from which directory (flink root or
>> flink-dist?)
>>
>> On Wed, Jun 3, 2015 at 5:10 PM, Theodore Vasiloudis <
>> theodoros.vasilou...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I'm having some problems with building the project. The error I get is
>> for
>>> the flink-dist module is:
>>>
>>> [ERROR] Failed to execute goal com.pyx4j:maven-junction-plugin:1.0.3:link
 (default) on project flink-dist: Can't create junction source

>>>
>> [/long/dir/here//flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT]
 -> [/long/dir/here/build-target/flink-dist/../build-target] -> [Help 1]

>>>
>>> Before that I get this warning in maven-assembly-plugin:2.4:single (bin)
>> @
>>> flink-dist ---:
>>>
>>> [WARNING] Assembly file:
 long/dir/here/flink-dist/target/flink-0.9-SNAPSHOT-bin is not a regular
 file (it may be a directory). It cannot be attached to the project
>> build
 for installation or deployment.

>>>
>>> Has anyone else encountered this? Any ideas on how it can be fixed?
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Build-failure-with-maven-junction-plugin-tp6056.html
>>> Sent from the Apache Flink Mailing List archive. mailing list archive at
>>> Nabble.com.
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool

[Behrouz Derakhshan]

Hi,

So my understanding was that the changes are only meant for flink-examples
package. But each package has its own set of examples.
And all of them has to be changed.
Is that OK?

@Ufuk: I agree, I create a ticket for adding Javadocs.

BR,
Behrouz


On Wed, Sep 9, 2015 at 3:53 PM, Maximilian Michels  wrote:

> It would be nice to support both non-positional and positional
> arguments. Like in
>
> > posarg1 posarg2 --nonpos1 nonpos1value --nonpos2 nonpos2value
>
> The arguments should also be named but should be expected at a fixed
> position counting from the left ignoring non-positional arguments.
>
> For the time being, it would also be ok with me if we ported all
> examples to non-positional arguments.
>
> On Fri, Sep 4, 2015 at 2:46 PM, Behrouz Derakhshan
>  wrote:
> > Yes, I was referring mostly to blog posts and other websites and was
> > wondering if breaking them is an issue or not.
> > I have already created a subtask to add support for positional arguments
> (
> > FLINK-2621 ), so the
> > examples would be backward compatible.
> > The problem with that is, we have to detect from the arguments to the
> > program, if they are positional or key/value and parse them accordingly.
> > But if everyone is OK with completely switching to ParameterTool and
> > breaking the support for the old way of executing the examples, then my
> job
> > would be also a lot easier.
> >
> >
> >
> > On Fri, Sep 4, 2015 at 2:34 PM, Robert Metzger 
> wrote:
> >
> >> If you are referring to this training material (
> >>
> >>
> https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataArtisans/flinkTraining/exercises/dataStreamJava/rideCleansing/RideCleansing.java
> >> ),
> >> some of the examples are actually already using the ParameterTool.
> >>
> >> The problem are probably websites / blogposts etc. that show how to use
> the
> >> Flink examples. But I think its fine to break these. All example jars
> >> contain the version number. If the way we pass arguments to the examples
> >> changes between 0.9 and 0.10, that should be fine.
> >>
> >> I think using the ParameterTool for the examples will improve the
> >> readability of the examples a lot. Right now, all examples have a
> >> (copy-pasted) parseParameters() method, which is doing very simplistic
> >> parameter parsing.
> >>
> >> The PT tool also allows to show the input parameters in the web
> interface.
> >>
> >> So I'm voting for doing a breaking change and using parameters such as
> >> "--input hdfs:/// --output hdfs:/// --iterations 15".
> >>
> >> On Fri, Sep 4, 2015 at 1:05 PM, Behrouz Derakhshan <
> >> behrouz.derakhs...@gmail.com> wrote:
> >>
> >> > Will do.
> >> >
> >> > Thanks,
> >> > Behrouz
> >> >
> >> > On Fri, Sep 4, 2015 at 11:29 AM, Maximilian Michels 
> >> > wrote:
> >> >
> >> > > Hi Behrouz,
> >> > >
> >> > > I would create a new sub-task under the original issue that
> introduce
> >> > > the ParameterTool: https://issues.apache.org/jira/browse/FLINK-1525
> >> > >
> >> > > Cheers,
> >> > > Max
> >> > >
> >> > > On Fri, Sep 4, 2015 at 11:17 AM, Behrouz Derakhshan
> >> > >  wrote:
> >> > > > Hi Max,
> >> > > >
> >> > > > What you said makes sense, for "ParameterTool doesn't seem to
> support
> >> > > > positional arguments :) but we could fix that." should we create a
> >> > > separate
> >> > > > ticket or should it also be part of FLINK-2021 ?
> >> > > >
> >> > > > BR,
> >> > > > Behrouz
> >> > > >
> >> > > >
> >> > > > On Fri, Sep 4, 2015 at 10:55 AM, Maximilian Michels <
> m...@apache.org>
> >> > > wrote:
> >> > > >
> >> > > >> Hi Behrouz,
> >> > > >>
> >> > > >> Thanks for starting the discussion. If I understand your question
> >> > > >> correctly, you are asking if it breaks the training or other
> >> external
> >> > > >> material if we convert the Flink examples to make use of the
> >> > > >> ParameterTool?
> >> > > >>
> >> > > >> We could make the changes such that the examples will accept the
> >> same
> >> > > >> parameters but use the ParameterTool internally to verify the
> >> > > >> parameters and print usage information. I think most examples
> simply
> >> > > >> use positional arguments and we could keep it that way. The only
> >> > > >> problem is that the ParameterTool doesn't seem to support
> positional
> >> > > >> arguments :) but we could fix that.
> >> > > >>
> >> > > >> Cheers,
> >> > > >> Max
> >> > > >>
> >> > > >> On Thu, Sep 3, 2015 at 5:50 PM, Behrouz Derakhshan
> >> > > >>  wrote:
> >> > > >> > Hi,
> >> > > >> >
> >> > > >> > I had at look at this ticket FLINK-2021
> >> > > >> > , there
> isn't
> >> > much
> >> > > to
> >> > > >> do
> >> > > >> > from a technical stand point and it kinda makes sense to use
> the
> >> new
> >> > > >> > "ParameterTool", since it is being used in most of the other
> part
> >> of
> >> > > the
> >> > > >> > code base.
> >> > > >> > The only question is do we reall

Re: Build failure with maven-junction-plugin

[Matthias J. Sax]

I could resolve this by manually deleting the link
"/home/mjsax/workspace_flink_asf/flink-asf/build-target"

But I still don't understand why this is necessary...

On 09/10/2015 12:14 PM, Matthias J. Sax wrote:
> Hi,
> 
> I am picking up on this old thread, because I have the same error now. I
> just created a new branch from a freshly rebased master branch.
> 
> mvn -DskipTests clean install
> 
>> Failed to execute goal com.pyx4j:maven-junction-plugin:1.0.3:link (default) 
>> on project flink-dist: Can't create junction source 
>> [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT]
>>  -> [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/../build-target] 
>> -> [Help 1]
> 
> This email thread did not show a fix/reason for this issue.
> 
> 
> -Matthias
> 
> On 06/03/2015 05:24 PM, Theodore Vasiloudis wrote:
>> mvn clean package -Dmaven.javadoc.skip=true -DskipTests
>>
>> Called from the flink root.
>>
>> On Wed, Jun 3, 2015 at 5:21 PM, Robert Metzger  wrote:
>>
>>> The WARNING is okay.
>>>
>>> How did you call maven, and from which directory (flink root or
>>> flink-dist?)
>>>
>>> On Wed, Jun 3, 2015 at 5:10 PM, Theodore Vasiloudis <
>>> theodoros.vasilou...@gmail.com> wrote:
>>>
 Hello,

 I'm having some problems with building the project. The error I get is
>>> for
 the flink-dist module is:

 [ERROR] Failed to execute goal com.pyx4j:maven-junction-plugin:1.0.3:link
> (default) on project flink-dist: Can't create junction source
>

>>> [/long/dir/here//flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT]
> -> [/long/dir/here/build-target/flink-dist/../build-target] -> [Help 1]
>

 Before that I get this warning in maven-assembly-plugin:2.4:single (bin)
>>> @
 flink-dist ---:

 [WARNING] Assembly file:
> long/dir/here/flink-dist/target/flink-0.9-SNAPSHOT-bin is not a regular
> file (it may be a directory). It cannot be attached to the project
>>> build
> for installation or deployment.
>

 Has anyone else encountered this? Any ideas on how it can be fixed?




 --
 View this message in context:

>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Build-failure-with-maven-junction-plugin-tp6056.html
 Sent from the Apache Flink Mailing List archive. mailing list archive at
 Nabble.com.

>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: Releasing 0.10.0-milestone1

[Matthias J. Sax]

I justed opened a PR for FLINK-2632: "Web Client does not respect the
class loader of submitted jobs"

https://github.com/apache/flink/pull/1114

-Matthias


On 09/10/2015 11:49 AM, Fabian Hueske wrote:
> All these bugs reported by users and its just four of them of which 2 are
> already fixed.
> It should be possible to fix them in a day.
> 
> 2015-09-10 11:19 GMT+02:00 Robert Metzger :
> 
>> Thank you very much for taking care of the release management!
>>
>> I'm not sure whether it makes sense to block a "preview" release on so many
>> bug fixes. It feels like 80% of our commits are bugfixes or stability
>> improvements anyways, so we would wait infinitely to release the milestone
>> if we wait for all of them.
>>
>> For example for the class loader issue in the web client issue, Max (who
>> initially brought the issue to our attention) also said: "Would be nice to
>> fix it but IMHO it is not critical for the milestone release."
>>
>>
>> I would vote for merging Till's https://github.com/apache/flink/pull/1101
>> and then start with the RC0 vote.
>> If there are more bug fixes available and the RC0 vote fails (which is
>> likely ;) ), we can include them as well.
>>
>>
>>
>> On Thu, Sep 10, 2015 at 11:04 AM, Fabian Hueske  wrote:
>>
>>> I can take the role of the release manager and push the 0.10 milestone
>>> release forward.
>>>
>>> I added a list of pending fixes for 0.10 to the wiki:
>>> https://cwiki.apache.org/confluence/display/FLINK/0.10+Release
>>> Please extend the list (or respond to this thread) if you have more fixes
>>> that should go into the release.
>>>
>>> From the currently open issues only FLINK-2632 (Web Client does not
>> respect
>>> the class loader of submitted jobs) is unassigned.
>>> Anybody around to pick this one up?
>>>
>>> Cheers, Fabian
>>>
>>> 2015-09-09 12:00 GMT+02:00 Till Rohrmann :
>>>
 I forgot to mention that there is also a bug with the `StreamFold`
>>> operator
 which we might consider fixing for the milestone release. I've opened a
>>> PR
 for it.

 https://issues.apache.org/jira/browse/FLINK-2631
 https://github.com/apache/flink/pull/1101


 On Wed, Sep 9, 2015 at 10:58 AM, Gyula Fóra  wrote:

> This sounds good +1 from me as well :)
> Till Rohrmann  ezt írta (időpont: 2015.
>>> szept.
> 9.,
> Sze, 10:40):
>
>> +1 for a milestone release with the TypeInformation issues fixed.
>> I'm
>> working on it.
>>
>> On Tue, Sep 8, 2015 at 9:32 PM, Stephan Ewen 
>>> wrote:
>>
>>> Great!
>>>
>>> I'd like to push one more commit later today.
>>> A fix for https://issues.apache.org/jira/browse/FLINK-2632 would
 also
> be
>>> highly appreciated by some users.
>>>
>>> Anyone volunteering as release manager (for creating release
 candidates
>> and
>>> uploading them)?
>>>
>>>
>>> On Tue, Sep 8, 2015 at 6:11 PM, Kostas Tzoumas <
>>> ktzou...@apache.org>
>>> wrote:
>>>
 +1 for a milestone release

 On Tue, Sep 8, 2015 at 5:43 PM, Robert Metzger <
 rmetz...@apache.org>
 wrote:

> +1 for a "milestone1" release. We have a lot of good features
>>> in
>> master
> that people can benefit from.
>
> On Tue, Sep 8, 2015 at 5:10 PM, Maximilian Michels <
 m...@apache.org
>>
 wrote:
>
>> +1 for releasing a milestone release soon to encourage
>> people
 to
>> try
>> out the new features.
>>
>> There is this bug:
>> https://issues.apache.org/jira/browse/FLINK-2632
>> which affects the Web Client's error and results display
>> for
> jobs.
>> Would be nice to fix it but IMHO it is not critical for the
>> milestone
>> release.
>>
>> On Tue, Sep 8, 2015 at 1:00 PM, Ufuk Celebi <
>> u...@apache.org>
>> wrote:
>>>
 On 08 Sep 2015, at 12:01, Stephan Ewen <
>> se...@apache.org>
>> wrote:

 Hi all!

 Some day back we talked about releasing an
>>> 0.10.0-milestone1
 release.
>> The
 master has advanced quite a bit (especially due to
>>> high-availability
>> code).

 I cherry picked the important additions to the
> release-0.10.0-milestone1
 branch (fixes and Kafka consumer/producer rework).

 How about releasing the branch now as an intermediate
 version
>> for
>> people to
 try out while we stabilize the windows and HA code for
>> the
> 0.10
> release?
>>>
>>> +1
>>>
>>> Thanks for cping the important changes. I’ve checked and
 there
> is
>> nothing I would add at this point.
>>>
>>> Can anybody else took a look at it? Other than that, I
>>> think
> it’s
 good
>>

Re: Build failure with maven-junction-plugin

[Stephan Ewen]

Did you do a "clean" together with the "install"? Then it should work.

The problem occurred when you switch between versions where the link is set
(0.9+) and versions prior to the link (< 0.9) ...

Stephan


On Thu, Sep 10, 2015 at 1:37 PM, Matthias J. Sax  wrote:

> I could resolve this by manually deleting the link
> "/home/mjsax/workspace_flink_asf/flink-asf/build-target"
>
> But I still don't understand why this is necessary...
>
> On 09/10/2015 12:14 PM, Matthias J. Sax wrote:
> > Hi,
> >
> > I am picking up on this old thread, because I have the same error now. I
> > just created a new branch from a freshly rebased master branch.
> >
> > mvn -DskipTests clean install
> >
> >> Failed to execute goal com.pyx4j:maven-junction-plugin:1.0.3:link
> (default) on project flink-dist: Can't create junction source
> [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT]
> -> [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/../build-target]
> -> [Help 1]
> >
> > This email thread did not show a fix/reason for this issue.
> >
> >
> > -Matthias
> >
> > On 06/03/2015 05:24 PM, Theodore Vasiloudis wrote:
> >> mvn clean package -Dmaven.javadoc.skip=true -DskipTests
> >>
> >> Called from the flink root.
> >>
> >> On Wed, Jun 3, 2015 at 5:21 PM, Robert Metzger 
> wrote:
> >>
> >>> The WARNING is okay.
> >>>
> >>> How did you call maven, and from which directory (flink root or
> >>> flink-dist?)
> >>>
> >>> On Wed, Jun 3, 2015 at 5:10 PM, Theodore Vasiloudis <
> >>> theodoros.vasilou...@gmail.com> wrote:
> >>>
>  Hello,
> 
>  I'm having some problems with building the project. The error I get is
> >>> for
>  the flink-dist module is:
> 
>  [ERROR] Failed to execute goal
> com.pyx4j:maven-junction-plugin:1.0.3:link
> > (default) on project flink-dist: Can't create junction source
> >
> 
> >>>
> [/long/dir/here//flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT]
> > -> [/long/dir/here/build-target/flink-dist/../build-target] -> [Help
> 1]
> >
> 
>  Before that I get this warning in maven-assembly-plugin:2.4:single
> (bin)
> >>> @
>  flink-dist ---:
> 
>  [WARNING] Assembly file:
> > long/dir/here/flink-dist/target/flink-0.9-SNAPSHOT-bin is not a
> regular
> > file (it may be a directory). It cannot be attached to the project
> >>> build
> > for installation or deployment.
> >
> 
>  Has anyone else encountered this? Any ideas on how it can be fixed?
> 
> 
> 
> 
>  --
>  View this message in context:
> 
> >>>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Build-failure-with-maven-junction-plugin-tp6056.html
>  Sent from the Apache Flink Mailing List archive. mailing list archive
> at
>  Nabble.com.
> 
> >>>
> >>
> >
>
>

Re: Build failure with maven-junction-plugin

[Maximilian Michels]

The junction plugin could not create the link "build-target" to the
build directory in flink-dist. Maybe this is a permission problem. You
could have turned on the Maven debug mode to see the underlying
exception.

On Thu, Sep 10, 2015 at 1:37 PM, Matthias J. Sax  wrote:
> I could resolve this by manually deleting the link
> "/home/mjsax/workspace_flink_asf/flink-asf/build-target"
>
> But I still don't understand why this is necessary...
>
> On 09/10/2015 12:14 PM, Matthias J. Sax wrote:
>> Hi,
>>
>> I am picking up on this old thread, because I have the same error now. I
>> just created a new branch from a freshly rebased master branch.
>>
>> mvn -DskipTests clean install
>>
>>> Failed to execute goal com.pyx4j:maven-junction-plugin:1.0.3:link (default) 
>>> on project flink-dist: Can't create junction source 
>>> [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT]
>>>  -> [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/../build-target] 
>>> -> [Help 1]
>>
>> This email thread did not show a fix/reason for this issue.
>>
>>
>> -Matthias
>>
>> On 06/03/2015 05:24 PM, Theodore Vasiloudis wrote:
>>> mvn clean package -Dmaven.javadoc.skip=true -DskipTests
>>>
>>> Called from the flink root.
>>>
>>> On Wed, Jun 3, 2015 at 5:21 PM, Robert Metzger  wrote:
>>>
 The WARNING is okay.

 How did you call maven, and from which directory (flink root or
 flink-dist?)

 On Wed, Jun 3, 2015 at 5:10 PM, Theodore Vasiloudis <
 theodoros.vasilou...@gmail.com> wrote:

> Hello,
>
> I'm having some problems with building the project. The error I get is
 for
> the flink-dist module is:
>
> [ERROR] Failed to execute goal com.pyx4j:maven-junction-plugin:1.0.3:link
>> (default) on project flink-dist: Can't create junction source
>>
>
 [/long/dir/here//flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT]
>> -> [/long/dir/here/build-target/flink-dist/../build-target] -> [Help 1]
>>
>
> Before that I get this warning in maven-assembly-plugin:2.4:single (bin)
 @
> flink-dist ---:
>
> [WARNING] Assembly file:
>> long/dir/here/flink-dist/target/flink-0.9-SNAPSHOT-bin is not a regular
>> file (it may be a directory). It cannot be attached to the project
 build
>> for installation or deployment.
>>
>
> Has anyone else encountered this? Any ideas on how it can be fixed?
>
>
>
>
> --
> View this message in context:
>
 http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Build-failure-with-maven-junction-plugin-tp6056.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

>>>
>>
>

Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool

[Maximilian Michels]

I think the primary concern was flink-examples but if you're on it,
you can also modify the other examples.

On Thu, Sep 10, 2015 at 12:43 PM, Behrouz Derakhshan
 wrote:
> Hi,
>
> So my understanding was that the changes are only meant for flink-examples
> package. But each package has its own set of examples.
> And all of them has to be changed.
> Is that OK?
>
> @Ufuk: I agree, I create a ticket for adding Javadocs.
>
> BR,
> Behrouz
>
>
> On Wed, Sep 9, 2015 at 3:53 PM, Maximilian Michels  wrote:
>
>> It would be nice to support both non-positional and positional
>> arguments. Like in
>>
>> > posarg1 posarg2 --nonpos1 nonpos1value --nonpos2 nonpos2value
>>
>> The arguments should also be named but should be expected at a fixed
>> position counting from the left ignoring non-positional arguments.
>>
>> For the time being, it would also be ok with me if we ported all
>> examples to non-positional arguments.
>>
>> On Fri, Sep 4, 2015 at 2:46 PM, Behrouz Derakhshan
>>  wrote:
>> > Yes, I was referring mostly to blog posts and other websites and was
>> > wondering if breaking them is an issue or not.
>> > I have already created a subtask to add support for positional arguments
>> (
>> > FLINK-2621 ), so the
>> > examples would be backward compatible.
>> > The problem with that is, we have to detect from the arguments to the
>> > program, if they are positional or key/value and parse them accordingly.
>> > But if everyone is OK with completely switching to ParameterTool and
>> > breaking the support for the old way of executing the examples, then my
>> job
>> > would be also a lot easier.
>> >
>> >
>> >
>> > On Fri, Sep 4, 2015 at 2:34 PM, Robert Metzger 
>> wrote:
>> >
>> >> If you are referring to this training material (
>> >>
>> >>
>> https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataArtisans/flinkTraining/exercises/dataStreamJava/rideCleansing/RideCleansing.java
>> >> ),
>> >> some of the examples are actually already using the ParameterTool.
>> >>
>> >> The problem are probably websites / blogposts etc. that show how to use
>> the
>> >> Flink examples. But I think its fine to break these. All example jars
>> >> contain the version number. If the way we pass arguments to the examples
>> >> changes between 0.9 and 0.10, that should be fine.
>> >>
>> >> I think using the ParameterTool for the examples will improve the
>> >> readability of the examples a lot. Right now, all examples have a
>> >> (copy-pasted) parseParameters() method, which is doing very simplistic
>> >> parameter parsing.
>> >>
>> >> The PT tool also allows to show the input parameters in the web
>> interface.
>> >>
>> >> So I'm voting for doing a breaking change and using parameters such as
>> >> "--input hdfs:/// --output hdfs:/// --iterations 15".
>> >>
>> >> On Fri, Sep 4, 2015 at 1:05 PM, Behrouz Derakhshan <
>> >> behrouz.derakhs...@gmail.com> wrote:
>> >>
>> >> > Will do.
>> >> >
>> >> > Thanks,
>> >> > Behrouz
>> >> >
>> >> > On Fri, Sep 4, 2015 at 11:29 AM, Maximilian Michels 
>> >> > wrote:
>> >> >
>> >> > > Hi Behrouz,
>> >> > >
>> >> > > I would create a new sub-task under the original issue that
>> introduce
>> >> > > the ParameterTool: https://issues.apache.org/jira/browse/FLINK-1525
>> >> > >
>> >> > > Cheers,
>> >> > > Max
>> >> > >
>> >> > > On Fri, Sep 4, 2015 at 11:17 AM, Behrouz Derakhshan
>> >> > >  wrote:
>> >> > > > Hi Max,
>> >> > > >
>> >> > > > What you said makes sense, for "ParameterTool doesn't seem to
>> support
>> >> > > > positional arguments :) but we could fix that." should we create a
>> >> > > separate
>> >> > > > ticket or should it also be part of FLINK-2021 ?
>> >> > > >
>> >> > > > BR,
>> >> > > > Behrouz
>> >> > > >
>> >> > > >
>> >> > > > On Fri, Sep 4, 2015 at 10:55 AM, Maximilian Michels <
>> m...@apache.org>
>> >> > > wrote:
>> >> > > >
>> >> > > >> Hi Behrouz,
>> >> > > >>
>> >> > > >> Thanks for starting the discussion. If I understand your question
>> >> > > >> correctly, you are asking if it breaks the training or other
>> >> external
>> >> > > >> material if we convert the Flink examples to make use of the
>> >> > > >> ParameterTool?
>> >> > > >>
>> >> > > >> We could make the changes such that the examples will accept the
>> >> same
>> >> > > >> parameters but use the ParameterTool internally to verify the
>> >> > > >> parameters and print usage information. I think most examples
>> simply
>> >> > > >> use positional arguments and we could keep it that way. The only
>> >> > > >> problem is that the ParameterTool doesn't seem to support
>> positional
>> >> > > >> arguments :) but we could fix that.
>> >> > > >>
>> >> > > >> Cheers,
>> >> > > >> Max
>> >> > > >>
>> >> > > >> On Thu, Sep 3, 2015 at 5:50 PM, Behrouz Derakhshan
>> >> > > >>  wrote:
>> >> > > >> > Hi,
>> >> > > >> >
>> >> > > >> > I had at look at this ticket FLINK-2021
>> >> > > >> > , the

[jira] [Created] (FLINK-2653) Enable object reuse in MergeIterator

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-2653:
-

 Summary: Enable object reuse in MergeIterator
 Key: FLINK-2653
 URL: https://issues.apache.org/jira/browse/FLINK-2653
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Runtime
Affects Versions: master
Reporter: Greg Hogan


MergeIterator currently discards given reusable objects and simply returns a 
new object from the JVM heap. This inefficiency has a noticeable impact on 
garbage collection and runtime overhead (~5% overall performance by my measure).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool

[Behrouz Derakhshan]

Well the problem is that "YARNSessionFIFOITCase" is testing Yarn by
running *word
count* example using multiple jar files.

So what happens is that, the test is calling this :

Runner runner = startWithArgs(new String[]{"run", "-m", "yarn-cluster",
"-yj", flinkUberjar.getAbsolutePath(),
"-yn", "1",
"-yjm", "768",
"-yD", "yarn.heap-cutoff-ratio=0.5", // test if the cutoff is
passed correctly
"-ytm", "1024",
"-ys", "2", // test requesting slots from YARN.
"--yarndetached", job, tmpInFile.getAbsoluteFile().toString() ,
tmpOutFolder.getAbsoluteFile().toString()},
  "The Job has been submitted with JobID",
RunTypes.CLI_FRONTEND);

For several jar files in different packages, that means if I want this test
to pass, all the word count examples should use the same argument formats.
All in all it was a bit confusing and took me awhile to figure out while
the tests were failing, I ran into the same problem specified here:
https://issues.apache.org/jira/browse/FLINK-1601 , and current logs does
not specify what the underlying issue is, it just says "Runner thread died
before the test was finished. Return value = 1"  .

I think it is a good idea to improve flink-yarn-tests package by adding
more meaning full logs.

On Thu, Sep 10, 2015 at 3:28 PM, Maximilian Michels  wrote:

> I think the primary concern was flink-examples but if you're on it,
> you can also modify the other examples.
>
> On Thu, Sep 10, 2015 at 12:43 PM, Behrouz Derakhshan
>  wrote:
> > Hi,
> >
> > So my understanding was that the changes are only meant for
> flink-examples
> > package. But each package has its own set of examples.
> > And all of them has to be changed.
> > Is that OK?
> >
> > @Ufuk: I agree, I create a ticket for adding Javadocs.
> >
> > BR,
> > Behrouz
> >
> >
> > On Wed, Sep 9, 2015 at 3:53 PM, Maximilian Michels 
> wrote:
> >
> >> It would be nice to support both non-positional and positional
> >> arguments. Like in
> >>
> >> > posarg1 posarg2 --nonpos1 nonpos1value --nonpos2 nonpos2value
> >>
> >> The arguments should also be named but should be expected at a fixed
> >> position counting from the left ignoring non-positional arguments.
> >>
> >> For the time being, it would also be ok with me if we ported all
> >> examples to non-positional arguments.
> >>
> >> On Fri, Sep 4, 2015 at 2:46 PM, Behrouz Derakhshan
> >>  wrote:
> >> > Yes, I was referring mostly to blog posts and other websites and was
> >> > wondering if breaking them is an issue or not.
> >> > I have already created a subtask to add support for positional
> arguments
> >> (
> >> > FLINK-2621 ), so
> the
> >> > examples would be backward compatible.
> >> > The problem with that is, we have to detect from the arguments to the
> >> > program, if they are positional or key/value and parse them
> accordingly.
> >> > But if everyone is OK with completely switching to ParameterTool and
> >> > breaking the support for the old way of executing the examples, then
> my
> >> job
> >> > would be also a lot easier.
> >> >
> >> >
> >> >
> >> > On Fri, Sep 4, 2015 at 2:34 PM, Robert Metzger 
> >> wrote:
> >> >
> >> >> If you are referring to this training material (
> >> >>
> >> >>
> >>
> https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataArtisans/flinkTraining/exercises/dataStreamJava/rideCleansing/RideCleansing.java
> >> >> ),
> >> >> some of the examples are actually already using the ParameterTool.
> >> >>
> >> >> The problem are probably websites / blogposts etc. that show how to
> use
> >> the
> >> >> Flink examples. But I think its fine to break these. All example jars
> >> >> contain the version number. If the way we pass arguments to the
> examples
> >> >> changes between 0.9 and 0.10, that should be fine.
> >> >>
> >> >> I think using the ParameterTool for the examples will improve the
> >> >> readability of the examples a lot. Right now, all examples have a
> >> >> (copy-pasted) parseParameters() method, which is doing very
> simplistic
> >> >> parameter parsing.
> >> >>
> >> >> The PT tool also allows to show the input parameters in the web
> >> interface.
> >> >>
> >> >> So I'm voting for doing a breaking change and using parameters such
> as
> >> >> "--input hdfs:/// --output hdfs:/// --iterations 15".
> >> >>
> >> >> On Fri, Sep 4, 2015 at 1:05 PM, Behrouz Derakhshan <
> >> >> behrouz.derakhs...@gmail.com> wrote:
> >> >>
> >> >> > Will do.
> >> >> >
> >> >> > Thanks,
> >> >> > Behrouz
> >> >> >
> >> >> > On Fri, Sep 4, 2015 at 11:29 AM, Maximilian Michels <
> m...@apache.org>
> >> >> > wrote:
> >> >> >
> >> >> > > Hi Behrouz,
> >> >> > >
> >> >> > > I would create a new sub-task under the original issue that
> >> introduce
> >> >> > > the ParameterTool:
> https://issues.apache.org/jira/browse/FLINK-1525
> >> >> > >
> >> >> > > Cheers,
> >> >> > > Max
> >> >> > >
> >> >> > > On Fri, Sep 4, 2015 at 11:17

[jira] [Created] (FLINK-2654) Add JavaDoc to ParameterTool class

[Behrouz Derakhshan (JIRA)]

Behrouz Derakhshan created FLINK-2654:
-

 Summary: Add JavaDoc to ParameterTool class
 Key: FLINK-2654
 URL: https://issues.apache.org/jira/browse/FLINK-2654
 Project: Flink
  Issue Type: Improvement
Reporter: Behrouz Derakhshan
Priority: Minor


ParameterTool class is missing JavaDocs 
The tool is already being used, and the plan is to use it in all of the example 
codes. We should add JavaDocs before start using it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Flink ML source code

[Hanan Meyer]

Hi

I would like to implement an Anova algorithm based on Flink's ML library.
Unfortunately I can't find the Flink 0.10 source code .
Is there a any way to get the code ? In case that I'll manage to implement
my idea I'll share it with the community of course ...

Thanks ,

Hanan Meyer

[jira] [Created] (FLINK-2655) Minimize intermediate merging of spilled buffers

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-2655:
-

 Summary: Minimize intermediate merging of spilled buffers
 Key: FLINK-2655
 URL: https://issues.apache.org/jira/browse/FLINK-2655
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Runtime
Affects Versions: master
Reporter: Greg Hogan


If the number of spilled buffers exceeds taskmanager.runtime.max-fan then the 
number of files must reduced with an intermediate merge by reading, merging, 
and spilling into a single, larger file.

The current implementation performs an intermediate merge on all files. An 
optimal implementation minimizes the amount of merged data by performing 
partial merges first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flink ML source code

[Robert Metzger]

Hi,

you can find the Flink ML source code here:
https://github.com/apache/flink/tree/master/flink-staging/flink-ml

On Thu, Sep 10, 2015 at 4:12 PM, Hanan Meyer  wrote:

> Hi
>
> I would like to implement an Anova algorithm based on Flink's ML library.
> Unfortunately I can't find the Flink 0.10 source code .
> Is there a any way to get the code ? In case that I'll manage to implement
> my idea I'll share it with the community of course ...
>
> Thanks ,
>
> Hanan Meyer
>

[jira] [Created] (FLINK-2656) FlinkKafkaConsumer is failing with OutOfRangeException

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-2656:
-

 Summary: FlinkKafkaConsumer is failing with OutOfRangeException
 Key: FLINK-2656
 URL: https://issues.apache.org/jira/browse/FLINK-2656
 Project: Flink
  Issue Type: Bug
  Components: Kafka Connector
Affects Versions: 0.10, 0.9.1
Reporter: Robert Metzger
Priority: Critical


FlinkKafkaConsumer is failing with an OutOfRangeException. There is actually a 
configuration parameter for the high level kafka consumer how to handle these 
situations (the high level c) doesn't fail on that exception.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Build failure with maven-junction-plugin

[Matthias J. Sax]

Thanks for the feedback. I doubt that it is a permission problem,
because everything is executed with the same user...

Unfortunately, after deleting the link manually, I cannot reproduce the
error any more...

Btw: the error occurred with "mvn -DskipTests clean install" as well as
"mvn clean" followed by "mvn -DskipTests install".

If the error occurs again, I will try maven debug mode to get more
insight...

For anybody else hitting this error: "rm build-target" on Flink top
level folder resolved the problem for me.


-Matthias

On 09/10/2015 03:27 PM, Maximilian Michels wrote:
> The junction plugin could not create the link "build-target" to the
> build directory in flink-dist. Maybe this is a permission problem. You
> could have turned on the Maven debug mode to see the underlying
> exception.
> 
> On Thu, Sep 10, 2015 at 1:37 PM, Matthias J. Sax  wrote:
>> I could resolve this by manually deleting the link
>> "/home/mjsax/workspace_flink_asf/flink-asf/build-target"
>>
>> But I still don't understand why this is necessary...
>>
>> On 09/10/2015 12:14 PM, Matthias J. Sax wrote:
>>> Hi,
>>>
>>> I am picking up on this old thread, because I have the same error now. I
>>> just created a new branch from a freshly rebased master branch.
>>>
>>> mvn -DskipTests clean install
>>>
 Failed to execute goal com.pyx4j:maven-junction-plugin:1.0.3:link 
 (default) on project flink-dist: Can't create junction source 
 [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/target/flink-0.10-SNAPSHOT-bin/flink-0.10-SNAPSHOT]
  -> [/home/mjsax/workspace_flink_asf/flink-asf/flink-dist/../build-target] 
 -> [Help 1]
>>>
>>> This email thread did not show a fix/reason for this issue.
>>>
>>>
>>> -Matthias
>>>
>>> On 06/03/2015 05:24 PM, Theodore Vasiloudis wrote:
 mvn clean package -Dmaven.javadoc.skip=true -DskipTests

 Called from the flink root.

 On Wed, Jun 3, 2015 at 5:21 PM, Robert Metzger  wrote:

> The WARNING is okay.
>
> How did you call maven, and from which directory (flink root or
> flink-dist?)
>
> On Wed, Jun 3, 2015 at 5:10 PM, Theodore Vasiloudis <
> theodoros.vasilou...@gmail.com> wrote:
>
>> Hello,
>>
>> I'm having some problems with building the project. The error I get is
> for
>> the flink-dist module is:
>>
>> [ERROR] Failed to execute goal com.pyx4j:maven-junction-plugin:1.0.3:link
>>> (default) on project flink-dist: Can't create junction source
>>>
>>
> [/long/dir/here//flink-dist/target/flink-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT]
>>> -> [/long/dir/here/build-target/flink-dist/../build-target] -> [Help 1]
>>>
>>
>> Before that I get this warning in maven-assembly-plugin:2.4:single (bin)
> @
>> flink-dist ---:
>>
>> [WARNING] Assembly file:
>>> long/dir/here/flink-dist/target/flink-0.9-SNAPSHOT-bin is not a regular
>>> file (it may be a directory). It cannot be attached to the project
> build
>>> for installation or deployment.
>>>
>>
>> Has anyone else encountered this? Any ideas on how it can be fixed?
>>
>>
>>
>>
>> --
>> View this message in context:
>>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Build-failure-with-maven-junction-plugin-tp6056.html
>> Sent from the Apache Flink Mailing List archive. mailing list archive at
>> Nabble.com.
>>
>

>>>
>>



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (FLINK-2657) Fix WebClient and CLI interaction

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-2657:
---

 Summary: Fix WebClient and CLI interaction
 Key: FLINK-2657
 URL: https://issues.apache.org/jira/browse/FLINK-2657
 Project: Flink
  Issue Type: Bug
  Components: Web Client
Affects Versions: 0.10
Reporter: Stephan Ewen


The way that the web client and CLI interact is not a good design. The web 
client puts a static variable into the CLI to alter its behavior towards its 
needs.

This is a typical case of no proper separation of concerns.

The shared behavior should go into utility classes, or a common abstract 
superclass, and the CLI frontend and the web client should not be tightly 
intermingled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Releasing 0.10.0-milestone1

[Robert Metzger]

Since the vote has not been started yet, we might consider including this
critical fix for the Kafka consumer into 0.10-milestone1:
https://github.com/apache/flink/pull/1117

On Thu, Sep 10, 2015 at 1:54 PM, Matthias J. Sax  wrote:

> I justed opened a PR for FLINK-2632: "Web Client does not respect the
> class loader of submitted jobs"
>
> https://github.com/apache/flink/pull/1114
>
> -Matthias
>
>
> On 09/10/2015 11:49 AM, Fabian Hueske wrote:
> > All these bugs reported by users and its just four of them of which 2 are
> > already fixed.
> > It should be possible to fix them in a day.
> >
> > 2015-09-10 11:19 GMT+02:00 Robert Metzger :
> >
> >> Thank you very much for taking care of the release management!
> >>
> >> I'm not sure whether it makes sense to block a "preview" release on so
> many
> >> bug fixes. It feels like 80% of our commits are bugfixes or stability
> >> improvements anyways, so we would wait infinitely to release the
> milestone
> >> if we wait for all of them.
> >>
> >> For example for the class loader issue in the web client issue, Max (who
> >> initially brought the issue to our attention) also said: "Would be nice
> to
> >> fix it but IMHO it is not critical for the milestone release."
> >>
> >>
> >> I would vote for merging Till's
> https://github.com/apache/flink/pull/1101
> >> and then start with the RC0 vote.
> >> If there are more bug fixes available and the RC0 vote fails (which is
> >> likely ;) ), we can include them as well.
> >>
> >>
> >>
> >> On Thu, Sep 10, 2015 at 11:04 AM, Fabian Hueske 
> wrote:
> >>
> >>> I can take the role of the release manager and push the 0.10 milestone
> >>> release forward.
> >>>
> >>> I added a list of pending fixes for 0.10 to the wiki:
> >>> https://cwiki.apache.org/confluence/display/FLINK/0.10+Release
> >>> Please extend the list (or respond to this thread) if you have more
> fixes
> >>> that should go into the release.
> >>>
> >>> From the currently open issues only FLINK-2632 (Web Client does not
> >> respect
> >>> the class loader of submitted jobs) is unassigned.
> >>> Anybody around to pick this one up?
> >>>
> >>> Cheers, Fabian
> >>>
> >>> 2015-09-09 12:00 GMT+02:00 Till Rohrmann :
> >>>
>  I forgot to mention that there is also a bug with the `StreamFold`
> >>> operator
>  which we might consider fixing for the milestone release. I've opened
> a
> >>> PR
>  for it.
> 
>  https://issues.apache.org/jira/browse/FLINK-2631
>  https://github.com/apache/flink/pull/1101
> 
> 
>  On Wed, Sep 9, 2015 at 10:58 AM, Gyula Fóra 
> wrote:
> 
> > This sounds good +1 from me as well :)
> > Till Rohrmann  ezt írta (időpont: 2015.
> >>> szept.
> > 9.,
> > Sze, 10:40):
> >
> >> +1 for a milestone release with the TypeInformation issues fixed.
> >> I'm
> >> working on it.
> >>
> >> On Tue, Sep 8, 2015 at 9:32 PM, Stephan Ewen 
> >>> wrote:
> >>
> >>> Great!
> >>>
> >>> I'd like to push one more commit later today.
> >>> A fix for https://issues.apache.org/jira/browse/FLINK-2632 would
>  also
> > be
> >>> highly appreciated by some users.
> >>>
> >>> Anyone volunteering as release manager (for creating release
>  candidates
> >> and
> >>> uploading them)?
> >>>
> >>>
> >>> On Tue, Sep 8, 2015 at 6:11 PM, Kostas Tzoumas <
> >>> ktzou...@apache.org>
> >>> wrote:
> >>>
>  +1 for a milestone release
> 
>  On Tue, Sep 8, 2015 at 5:43 PM, Robert Metzger <
>  rmetz...@apache.org>
>  wrote:
> 
> > +1 for a "milestone1" release. We have a lot of good features
> >>> in
> >> master
> > that people can benefit from.
> >
> > On Tue, Sep 8, 2015 at 5:10 PM, Maximilian Michels <
>  m...@apache.org
> >>
>  wrote:
> >
> >> +1 for releasing a milestone release soon to encourage
> >> people
>  to
> >> try
> >> out the new features.
> >>
> >> There is this bug:
> >> https://issues.apache.org/jira/browse/FLINK-2632
> >> which affects the Web Client's error and results display
> >> for
> > jobs.
> >> Would be nice to fix it but IMHO it is not critical for the
> >> milestone
> >> release.
> >>
> >> On Tue, Sep 8, 2015 at 1:00 PM, Ufuk Celebi <
> >> u...@apache.org>
> >> wrote:
> >>>
>  On 08 Sep 2015, at 12:01, Stephan Ewen <
> >> se...@apache.org>
> >> wrote:
> 
>  Hi all!
> 
>  Some day back we talked about releasing an
> >>> 0.10.0-milestone1
>  release.
> >> The
>  master has advanced quite a bit (especially due to
> >>> high-availability
> >> code).
> 
>  I cherry picked the important additions to the
> > release-0.10.0-milestone1
>  branch (fixes and Kafka consumer/produce

Blog Posts

[Matthias J. Sax]

Hi,

I had a look into "Writing a Blog Post"
(https://cwiki.apache.org/confluence/display/FLINK/Writing+a+Blog+Post)

In the Wiki the "categories" tag indicates "blog". I was looking into
the existing posts and all of them have "news" instead of "blog" as
"categories".

Thus, I am wondering what the difference between "blog" and "news" is?
Or might the Wiki entry be wrong? It's just to clarify. Thx.

-Matthias



signature.asc
Description: OpenPGP digital signature

Re: Blog Posts

[Robert Metzger]

Hi,
I guess nobody is really following the rules ;) (and or using an existing
post as a template)

I think the tag determines the first part after apache.org/ in the url (
http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html).

Maybe we can update the wiki article to reflect the de facto standard ;)


On Thu, Sep 10, 2015 at 7:21 PM, Matthias J. Sax  wrote:

> Hi,
>
> I had a look into "Writing a Blog Post"
> (https://cwiki.apache.org/confluence/display/FLINK/Writing+a+Blog+Post)
>
> In the Wiki the "categories" tag indicates "blog". I was looking into
> the existing posts and all of them have "news" instead of "blog" as
> "categories".
>
> Thus, I am wondering what the difference between "blog" and "news" is?
> Or might the Wiki entry be wrong? It's just to clarify. Thx.
>
> -Matthias
>
>