[jira] [Created] (FLINK-2579) StreamExecutionEnvironment & ExecutionEnvironment do not share interface but have a lot in common

[Arnaud Linz (JIRA)]

Arnaud Linz created FLINK-2579:
--

 Summary: StreamExecutionEnvironment & ExecutionEnvironment do not 
share interface but have a lot in common
 Key: FLINK-2579
 URL: https://issues.apache.org/jira/browse/FLINK-2579
 Project: Flink
  Issue Type: Improvement
  Components: Core
Reporter: Arnaud Linz
Priority: Minor


Both classes 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment and 
org.apache.flink.api.java.ExecutionEnvironment have a lot in common (same 
methods for kryo registration, fromCollection, etc) but are not related by a 
java contract.

That leads to annoying differences, for instance : 
StreamExecutionEnvironment.setParallelism() returns 'this' where as 
ExecutionEnvironment.setParallelism() has not return value.

They have specificities, but maybe they should both implement a common 
Interface to make sure that the common signatures are coherent?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2580) HadoopDataOutputStream does not expose enough methods of org.apache.hadoop.fs.FSDataOutputStream

[Arnaud Linz (JIRA)]

Arnaud Linz created FLINK-2580:
--

 Summary: HadoopDataOutputStream does not expose enough methods of 
org.apache.hadoop.fs.FSDataOutputStream
 Key: FLINK-2580
 URL: https://issues.apache.org/jira/browse/FLINK-2580
 Project: Flink
  Issue Type: Improvement
  Components: Hadoop Compatibility
Reporter: Arnaud Linz
Priority: Minor


I’ve noticed that when you use org.apache.flink.core.fs.FileSystem to write 
into a hdfs file, calling 
org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(), it returns a  
HadoopDataOutputStream that wraps a org.apache.hadoop.fs.FSDataOutputStream 
(under its org.apache.hadoop.hdfs.client .HdfsDataOutputStream wrappper).
 
However, FSDataOutputStream exposes many methods like flush,   getPos etc, but 
HadoopDataOutputStream only wraps write & close.
 
For instance, flush() calls the default, empty implementation of OutputStream 
instead of the hadoop one, and that’s confusing. Moreover, because of the 
restrictive OutputStream interface, hsync() and hflush() are not exposed to 
Flink.

I see two options:

- complete the class to wrap all methods of OutputStream and add a 
getWrappedStream() to access other stuff like hsync().

- get rid of the Hadoop wrapping and directly use Hadoop file system objects.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Bug fix release 0.9.1

[Ufuk Celebi]

Hey all,

the artifacts are uploading at the moment. I will start a vote thread as
soon as everything is up. :-)

– Ufuk

On Wed, Aug 26, 2015 at 10:26 AM, Ufuk Celebi  wrote:

> I went over all issues and pinged a couple of people about open issues.
> Thanks for all the help! All JIRA issues tagged with fixVersion 0.9.1
> should now reflect the last state of the discussion.
>
> There is a related discussion about whether we want to release master as
> 0.9.1 (mod some changes). Please provide your feedback in that thread.
>
> That discussion is currently *not* blocking anything all all 0.9.1 fixes
> need to go into master as well at the moment.
>
> On Fri, Aug 21, 2015 at 4:10 PM, Robert Metzger 
> wrote:
>
>> I would like to release Flink 0.9.1 soon. There were some annoying bugs
>> discovered recently, that should be addressed in a bug fix release.
>>
>> I have the following bugs in my mind:
>>
>> https://issues.apache.org/jira/browse/FLINK-2555 (Hadoop Input/Output
>> Formats are unable to access secured HDFS clusters)
>> https://issues.apache.org/jira/browse/FLINK-2386 (Implement Kafka
>> connector
>> using the new Kafka Consumer API)
>> https://issues.apache.org/jira/browse/FLINK-2543 (State handling does not
>> support deserializing classes through the UserCodeClassloader)
>>
>> I'm working on providing a fix for FLINK-2543 asap (I have this particular
>> bug already fixed, but my test is is triggering another one)
>>
>>
>>
>>
>> On Thu, Aug 6, 2015 at 4:11 PM, Hermann Gábor 
>> wrote:
>>
>> > Hi,
>> >
>> > I did a (thin) fix for
>> > FLINK-2286 Window ParallelMerge sometimes swallows elements of the last
>> > window
>> >
>> > and also added a PR .
>> >
>> > Regards,
>> > Gábor
>> >
>> > On Thu, Aug 6, 2015 at 10:07 AM Maximilian Michels 
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > The following commits have been added to the release-0.9 branch since
>> the
>> > > 0.9.0 release:
>> > >
>> > > c7e8684 [FLINK-2229] Add equals() and hashCode() to
>> ObjectArrayTypeInfo
>> > > 451eb82 [FLINK-2280] GenericTypeComparator.compare() respects
>> ascending
>> > > flag
>> > > acd4317 [FLINK-2353] Respect JobConfigurable interface in Hadoop
>> mapred
>> > > wrappers
>> > > 055997e [docs] fix loading of style sheet with protocol relative base
>> URL
>> > > 7c2a704 [FLINK-2293] [runtime] Fix estimation for the number of hash
>> > > buckets on recursive builds
>> > > 0789460 [FLINK-2285] [streaming] Removed duplicate call in close from
>> > > GroupedActiveDiscretizer
>> > > d946599 [FLINK-2298] Add option to pass a custom name for Flink on
>> YARN
>> > > 9835625 [docs] fix broken links in FAQ
>> > > 69f858e [FLINK-2257] [streaming] Properly forward rich window function
>> > > calls to wrapped functions
>> > > d7cfa55 [docs] correct baseurl for 0.9 documentation
>> > > b5702c2 [docs] correct yarn command-line example
>> > > 9f0da5b [FLINK-2262][utils] rename method for default integer value in
>> > > ParameterTool
>> > >
>> > > I'd propose to add all of them to the 0.9.1 release. They contain
>> > important
>> > > fixes and should thus be included in the minor release.
>> > >
>> > > As for the JIRA issues, it would be great to fix and include all of
>> them.
>> > > Except for
>> > >
>> > > "FLINK-2236 RowSerializer and CaseClassComparator are not in sync
>> > regarding
>> > > Null-Values"
>> > >
>> > > because we reverted null-value support for in the Table API for the
>> 0.9.0
>> > > release. If the support on the master is fixed, we could include it
>> again
>> > > together with the fix for the RowSerializer and CaseClassComperator.
>> > >
>> > > Cheers,
>> > > Max
>> > >
>> > > On Fri, Jul 31, 2015 at 12:40 PM, Fabian Hueske 
>> > wrote:
>> > >
>> > > > Thanks Ufuk for starting this discussion.
>> > > >
>> > > > We should also go through the commit logs of the master branch and
>> see
>> > if
>> > > > we forgot to cherry-pick some fixes over to the release-0.9 branch.
>> > > > I can do that and compile a list of potential fixes.
>> > > >
>> > > > Cheers,
>> > > > Fabian
>> > > >
>> > > > 2015-07-31 11:34 GMT+02:00 Ufuk Celebi :
>> > > >
>> > > > > Hey all!
>> > > > >
>> > > > > I want to start a discussion about the next 0.9 release. Since
>> 0.9.0
>> > > > there
>> > > > > have been 19 commits addressing 16 issues. Of these 16 issues, two
>> > were
>> > > > > critical issues regarding the runtime, which require a new release
>> > > > urgently.
>> > > > >
>> > > > > What's do you think about this?
>> > > > >
>> > > > > ---
>> > > > >
>> > > > > If we agree about doing a new release, I can act as release
>> manager.
>> > > > >
>> > > > > JIRA currently reports 5 open issues with "fix version" set to
>> 0.9.1:
>> > > > >
>> > > > > FLINK-2450 IndexOutOfBoundsException in KryoSerializer
>> > > > > FLINK-2442 PojoType fields not supported by field position keys
>> > > > > FLINK-2356 Resource leak in checkpoint coordinator
>> > > > > FLINK-2286 Window ParallelMerge sometimes swall

[jira] [Created] (FLINK-2581) Add warning to the YARN client when using an older Hadoop version

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-2581:
-

 Summary: Add warning to the YARN client when using an older Hadoop 
version
 Key: FLINK-2581
 URL: https://issues.apache.org/jira/browse/FLINK-2581
 Project: Flink
  Issue Type: Improvement
  Components: YARN Client
Affects Versions: 0.10
Reporter: Robert Metzger
Assignee: Robert Metzger


Some users were facing issues when using Flink on YARN because they were using 
outdated Hadoop releases included into Flink.

In this example, the user was using Flink with hadoop 2.2.0 included, against a 
Hadoop 2.6.0 cluster: 
http://stackoverflow.com/questions/32085990/error-with-kerberos-authentication-when-executing-flink-example-code-on-yarn-clu/32131571

With this change, I'm going to ResourceManager's REST API to get the Hadoop 
version of the remote YARN cluster: 
http://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
This Hadoop class allows us to determine the local Hadoop version we are using: 
https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/util/VersionInfo.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[VOTE] Release Apache Flink 0.9.1 (RC0)

[Ufuk Celebi]

Dear community,

Please vote on releasing the following candidate as Apache Flink version
0.9.1. This is a maintenance release for Flink's latest stable version. The
candidate fixes 37 issues [1] and adds 46 commits.

-
The commit to be voted on:
9b83e092a8cf8ae35486a93a8e0a0d9434d5abb7

Branch:
release-0.9.1-rc0 (
https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-0.9.1-rc0
)

The release artifacts to be voted on can be found at:
http://people.apache.org/~uce/flink-0.9.1-rc0/

Release artifacts are signed with the key with fingerprint 9D403309:
http://www.apache.org/dist/flink/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapacheflink-1043
-

Please vote on releasing this package as Apache Flink 0.9.1.

The vote is open for the next 72 hours and passes if a majority of at least
three +1 PMC votes are cast.

The vote ends on Sunday (August 30, 2015).

[ ] +1 Release this package as Apache Flink 0.9.1
[ ] -1 Do not release this package, because...

– Ufuk

[1]
https://issues.apache.org/jira/browse/FLINK-2572?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.9.1

Testing the 0.9.1 Candidate

[Ufuk Celebi]

Hey all,

we've just started the vote for a new bugfix release with version 0.9.1.

I've created the following document to synchronize testing efforts. Feel
free to assign yourself to certain tests in the document. The list of tests
is taken from the Wiki.

https://docs.google.com/document/d/1GwDYSS8JqbOzoYjHisdwAeWjgOIv20ebuGo7qoRjHe8/edit?usp=sharing

The testing should not be as labour intensive as for the last 0.9.0
release. We need to make sure that the new commits in RC0 didn't break
anything. It's also good to look out for fixes we might have missed for the
RC.

– Ufuk

[jira] [Created] (FLINK-2582) Document how to build Flink with other Scala versions

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-2582:
---

 Summary: Document how to build Flink with other Scala versions
 Key: FLINK-2582
 URL: https://issues.apache.org/jira/browse/FLINK-2582
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 0.10
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 0.10


On can build Flink for different Scala versions.

We should describe in the documentation how to do that, ideally next to 
building for different Hadoop versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2583) Add Stream Sink For Rolling HDFS Files

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-2583:
---

 Summary: Add Stream Sink For Rolling HDFS Files
 Key: FLINK-2583
 URL: https://issues.apache.org/jira/browse/FLINK-2583
 Project: Flink
  Issue Type: New Feature
  Components: Streaming
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
 Fix For: 0.10


In addition to having configurable file-rolling behavior the Sink should also 
integrate with checkpointing to make it possible to have exactly-once semantics 
throughout the topology.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-2584) ASM dependency is not shaded away

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-2584:
--

 Summary: ASM dependency is not shaded away
 Key: FLINK-2584
 URL: https://issues.apache.org/jira/browse/FLINK-2584
 Project: Flink
  Issue Type: Bug
  Components: Core
Affects Versions: 0.9, master
Reporter: Ufuk Celebi
Assignee: Robert Metzger


ASM is not correctly shaded away. If you build the quick start against the 
snapshot version, you will see the following dependencies. Robert is fixing 
this.

{code}
[INFO] +- org.apache.flink:flink-java:jar:0.9.1:compile
[INFO] |  +- org.apache.flink:flink-core:jar:0.9.1:compile
[INFO] |  |  \- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] |  +- org.apache.flink:flink-shaded-include-yarn:jar:0.9.1:compile
[INFO] |  +- org.apache.avro:avro:jar:1.7.6:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] |  |  +- org.xerial.snappy:snappy-java:jar:1.0.5:compile
[INFO] |  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] |  | \- org.tukaani:xz:jar:1.0:compile
[INFO] |  +- com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[INFO] |  |  +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] |  |  \- org.objenesis:objenesis:jar:2.1:compile
[INFO] |  +- com.twitter:chill_2.10:jar:0.5.2:compile
[INFO] |  |  +- org.scala-lang:scala-library:jar:2.10.4:compile
[INFO] |  |  \- com.twitter:chill-java:jar:0.5.2:compile
[INFO] |  +- com.twitter:chill-avro_2.10:jar:0.5.2:compile
[INFO] |  |  +- com.twitter:chill-bijection_2.10:jar:0.5.2:compile
[INFO] |  |  |  \- com.twitter:bijection-core_2.10:jar:0.7.2:compile
[INFO] |  |  \- com.twitter:bijection-avro_2.10:jar:0.7.2:compile
[INFO] |  +- de.javakaffee:kryo-serializers:jar:0.36:compile
[INFO] |  |  +- com.esotericsoftware:kryo:jar:3.0.3:compile
[INFO] |  |  |  +- com.esotericsoftware:reflectasm:jar:1.10.1:compile
[INFO] |  |  |  |  \- org.ow2.asm:asm:jar:5.0.3:compile
[INFO] |  |  |  \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  |  \- com.google.protobuf:protobuf-java:jar:2.6.1:compile
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[RESULT] [VOTE] Release Apache Flink 0.9.1 (RC0)

[Ufuk Celebi]

This vote is cancelled in favor of RC1, due to FLINK-2584.


In the current snapshot and release-0.9 branch, shading of ASM is not
working correctly. The corresponding issue is here:
https://issues.apache.org/jira/browse/FLINK-2584. Robert is working on a
quick fix for 0.9.1.

The good thing is that most release tests will not be affected by this and
can be forwarded to the upcoming RC1. I will prepare a new RC shortly (will
take some time though due to uploading etc.).

– Ufuk

On Thu, Aug 27, 2015 at 1:32 PM, Ufuk Celebi  wrote:

> Dear community,
>
> Please vote on releasing the following candidate as Apache Flink version
> 0.9.1. This is a maintenance release for Flink's latest stable version. The
> candidate fixes 37 issues [1] and adds 46 commits.
>
> -
> The commit to be voted on:
> 9b83e092a8cf8ae35486a93a8e0a0d9434d5abb7
>
> Branch:
> release-0.9.1-rc0 (
>
> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-0.9.1-rc0
> )
>
> The release artifacts to be voted on can be found at:
> http://people.apache.org/~uce/flink-0.9.1-rc0/
>
> Release artifacts are signed with the key with fingerprint 9D403309:
> http://www.apache.org/dist/flink/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapacheflink-1043
> -
>
> Please vote on releasing this package as Apache Flink 0.9.1.
>
> The vote is open for the next 72 hours and passes if a majority of at least
> three +1 PMC votes are cast.
>
> The vote ends on Sunday (August 30, 2015).
>
> [ ] +1 Release this package as Apache Flink 0.9.1
> [ ] -1 Do not release this package, because...
>
> – Ufuk
>
> [1]
> https://issues.apache.org/jira/browse/FLINK-2572?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.9.1
>

Re: [VOTE] Release Apache Flink 0.9.1 (RC0)

[Stephan Ewen]

+/- 0

One minor issue:
  - The binary distribution LICENSE file mentions a wrong version of
javakaffee (mentions 27, but we reference 36). If we redo the release
candidate, this should be fixed.


Performed the following checks:
 - Checked LICENSE / NOTICE files
 - Checked the README file
 - Built against Hadoop 2.6.0
 - Built against Scala 2.11
 - Executed all tests
 - Builds in IntelliJ
 - Manual tests all work



On Thu, Aug 27, 2015 at 1:32 PM, Ufuk Celebi  wrote:

> Dear community,
>
> Please vote on releasing the following candidate as Apache Flink version
> 0.9.1. This is a maintenance release for Flink's latest stable version. The
> candidate fixes 37 issues [1] and adds 46 commits.
>
> -
> The commit to be voted on:
> 9b83e092a8cf8ae35486a93a8e0a0d9434d5abb7
>
> Branch:
> release-0.9.1-rc0 (
>
> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-0.9.1-rc0
> )
>
> The release artifacts to be voted on can be found at:
> http://people.apache.org/~uce/flink-0.9.1-rc0/
>
> Release artifacts are signed with the key with fingerprint 9D403309:
> http://www.apache.org/dist/flink/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapacheflink-1043
> -
>
> Please vote on releasing this package as Apache Flink 0.9.1.
>
> The vote is open for the next 72 hours and passes if a majority of at least
> three +1 PMC votes are cast.
>
> The vote ends on Sunday (August 30, 2015).
>
> [ ] +1 Release this package as Apache Flink 0.9.1
> [ ] -1 Do not release this package, because...
>
> – Ufuk
>
> [1]
>
> https://issues.apache.org/jira/browse/FLINK-2572?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.9.1
>

Re: Testing the 0.9.1 Candidate

[Ufuk Celebi]

The vote has been cancelled. Most tests can be forwarded. I will update the
document as soon as the new RC is out.

On Thu, Aug 27, 2015 at 1:38 PM, Ufuk Celebi  wrote:

> Hey all,
>
> we've just started the vote for a new bugfix release with version 0.9.1.
>
> I've created the following document to synchronize testing efforts. Feel
> free to assign yourself to certain tests in the document. The list of tests
> is taken from the Wiki.
>
>
> https://docs.google.com/document/d/1GwDYSS8JqbOzoYjHisdwAeWjgOIv20ebuGo7qoRjHe8/edit?usp=sharing
>
> The testing should not be as labour intensive as for the last 0.9.0
> release. We need to make sure that the new commits in RC0 didn't break
> anything. It's also good to look out for fixes we might have missed for the
> RC.
>
> – Ufuk
>
>

[jira] [Created] (FLINK-2585) KafkaSource not working

[Boyang Jerry Peng (JIRA)]

Boyang Jerry Peng created FLINK-2585:


 Summary: KafkaSource not working
 Key: FLINK-2585
 URL: https://issues.apache.org/jira/browse/FLINK-2585
 Project: Flink
  Issue Type: Bug
Reporter: Boyang Jerry Peng


I tried running the KafkaConsumerExample with that is subscribing to a command 
line producer of kafka but the KafkaConsumerExample topology was not receiving 
any data from Kafka.  Then I wrote my own topology that uses Kafka as a source 
but it didn't work as well.  The topologies would run but receive not data.  
Can someone help me with this problem?  The code I am running:





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Apache Flink:ProgramInvocationException on Yarn

[Hanan Meyer]

Hello All

When using Eclipse IDE to submit Flink to Yarn single node cluster I'm
getting :
"org.apache.flink.client.program.ProgramInvocationException: Failed to
resolve JobManager"

Using Flink 0.9.0

The Jar copy a file from one location in Hdfs to another and works fine
while executed locally on the single node Yarn cluster -
bin/flink run -c Test ./examples/MyJar.jar
hdfs://localhost:9000/flink/in.txt hdfs://localhost:9000/flink/out.txt

The code skeleton:

ExecutionEnvironment envRemote =
ExecutionEnvironment.createRemoteEnvironment
(FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT);
DataSet data =
envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
data.writeAsText("hdfs://localhost:9000/flink/out.txt");
envRemote.execute();


Please advise,

Hanan Meyer

Re: Apache Flink:ProgramInvocationException on Yarn

[Robert Metzger]

Hi,

Which values did you use for FLINK_SERVER_URL and FLINK_PORT?
Every time you deploy Flink on YARN, the host and port change, because the
JobManager is started on a different YARN container.


On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer  wrote:

> Hello All
>
> When using Eclipse IDE to submit Flink to Yarn single node cluster I'm
> getting :
> "org.apache.flink.client.program.ProgramInvocationException: Failed to
> resolve JobManager"
>
> Using Flink 0.9.0
>
> The Jar copy a file from one location in Hdfs to another and works fine
> while executed locally on the single node Yarn cluster -
> bin/flink run -c Test ./examples/MyJar.jar
> hdfs://localhost:9000/flink/in.txt hdfs://localhost:9000/flink/out.txt
>
> The code skeleton:
>
> ExecutionEnvironment envRemote =
> ExecutionEnvironment.createRemoteEnvironment
> (FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT);
> DataSet data =
> envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> envRemote.execute();
>
>
> Please advise,
>
> Hanan Meyer
>

Re: Apache Flink:ProgramInvocationException on Yarn

[Stephan Ewen]

If you start the job via the "bin/flink" script, then simply use
"ExecutionEnvironment.getExecutionEnvironment()" rather then creating a
remote environment manually.

That way, hosts and ports are configured automatically.

On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger  wrote:

> Hi,
>
> Which values did you use for FLINK_SERVER_URL and FLINK_PORT?
> Every time you deploy Flink on YARN, the host and port change, because the
> JobManager is started on a different YARN container.
>
>
> On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer  wrote:
>
> > Hello All
> >
> > When using Eclipse IDE to submit Flink to Yarn single node cluster I'm
> > getting :
> > "org.apache.flink.client.program.ProgramInvocationException: Failed to
> > resolve JobManager"
> >
> > Using Flink 0.9.0
> >
> > The Jar copy a file from one location in Hdfs to another and works fine
> > while executed locally on the single node Yarn cluster -
> > bin/flink run -c Test ./examples/MyJar.jar
> > hdfs://localhost:9000/flink/in.txt hdfs://localhost:9000/flink/out.txt
> >
> > The code skeleton:
> >
> > ExecutionEnvironment envRemote =
> > ExecutionEnvironment.createRemoteEnvironment
> > (FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT);
> > DataSet data =
> > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> > envRemote.execute();
> >
> >
> > Please advise,
> >
> > Hanan Meyer
> >
>

Re: [RESULT] [VOTE] Release Apache Flink 0.9.1 (RC0)

[Robert Metzger]

Stephan pushed a fix for the ASM problem.
I noticed that guava is also not shaded properly.

As part of https://issues.apache.org/jira/browse/FLINK-2584, I'm going to
add a script to our travis build which will automatically check the
contents of the fat jar for improperly shaded classes.

On Thu, Aug 27, 2015 at 4:21 PM, Ufuk Celebi  wrote:

> This vote is cancelled in favor of RC1, due to FLINK-2584.
>
>
> In the current snapshot and release-0.9 branch, shading of ASM is not
> working correctly. The corresponding issue is here:
> https://issues.apache.org/jira/browse/FLINK-2584. Robert is working on a
> quick fix for 0.9.1.
>
> The good thing is that most release tests will not be affected by this and
> can be forwarded to the upcoming RC1. I will prepare a new RC shortly (will
> take some time though due to uploading etc.).
>
> – Ufuk
>
> On Thu, Aug 27, 2015 at 1:32 PM, Ufuk Celebi  wrote:
>
> > Dear community,
> >
> > Please vote on releasing the following candidate as Apache Flink version
> > 0.9.1. This is a maintenance release for Flink's latest stable version.
> The
> > candidate fixes 37 issues [1] and adds 46 commits.
> >
> > -
> > The commit to be voted on:
> > 9b83e092a8cf8ae35486a93a8e0a0d9434d5abb7
> >
> > Branch:
> > release-0.9.1-rc0 (
> >
> >
> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git;a=shortlog;h=refs/heads/release-0.9.1-rc0
> > )
> >
> > The release artifacts to be voted on can be found at:
> > http://people.apache.org/~uce/flink-0.9.1-rc0/
> >
> > Release artifacts are signed with the key with fingerprint 9D403309:
> > http://www.apache.org/dist/flink/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapacheflink-1043
> > -
> >
> > Please vote on releasing this package as Apache Flink 0.9.1.
> >
> > The vote is open for the next 72 hours and passes if a majority of at
> least
> > three +1 PMC votes are cast.
> >
> > The vote ends on Sunday (August 30, 2015).
> >
> > [ ] +1 Release this package as Apache Flink 0.9.1
> > [ ] -1 Do not release this package, because...
> >
> > – Ufuk
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/FLINK-2572?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.9.1
> >
>

Re: Apache Flink:ProgramInvocationException on Yarn

[Alexey Sapozhnikov]

Hello all.

Some clarification: locally everything works great.
However once we run our Flink on remote linux machine and try to run the
client program from our machine, using create remote environment- Flink
JobManager is raising this exception

On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen  wrote:

> If you start the job via the "bin/flink" script, then simply use
> "ExecutionEnvironment.getExecutionEnvironment()" rather then creating a
> remote environment manually.
>
> That way, hosts and ports are configured automatically.
>
> On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger 
> wrote:
>
>> Hi,
>>
>> Which values did you use for FLINK_SERVER_URL and FLINK_PORT?
>> Every time you deploy Flink on YARN, the host and port change, because the
>> JobManager is started on a different YARN container.
>>
>>
>> On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer  wrote:
>>
>> > Hello All
>> >
>> > When using Eclipse IDE to submit Flink to Yarn single node cluster I'm
>> > getting :
>> > "org.apache.flink.client.program.ProgramInvocationException: Failed to
>> > resolve JobManager"
>> >
>> > Using Flink 0.9.0
>> >
>> > The Jar copy a file from one location in Hdfs to another and works fine
>> > while executed locally on the single node Yarn cluster -
>> > bin/flink run -c Test ./examples/MyJar.jar
>> > hdfs://localhost:9000/flink/in.txt hdfs://localhost:9000/flink/out.txt
>> >
>> > The code skeleton:
>> >
>> > ExecutionEnvironment envRemote =
>> > ExecutionEnvironment.createRemoteEnvironment
>> > (FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT);
>> > DataSet data =
>> > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
>> > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
>> > envRemote.execute();
>> >
>> >
>> > Please advise,
>> >
>> > Hanan Meyer
>> >
>>
>
>

[jira] [Created] (FLINK-2586) Unstable Storm Compatibility Tests

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-2586:
---

 Summary: Unstable Storm Compatibility Tests
 Key: FLINK-2586
 URL: https://issues.apache.org/jira/browse/FLINK-2586
 Project: Flink
  Issue Type: Bug
  Components: Storm Compatibility
Affects Versions: 0.10
Reporter: Stephan Ewen
Priority: Critical
 Fix For: 0.10


The Storm Compatibility tests frequently fail.

The reason is that they kill the topologies after a certain time interval. That 
may fail on CI infrastructure when certain steps are delayed beyond usual. 
Trying to guarantee progress by time is inherently problematic:
  - Waiting too short makes tests unstable
  - Waiting too long makes tests slow

The right way to go is letting the program decide when to terminate, for 
example by throwing a special {{SuccessException}}.

Have a look at the Kafka connector tests, they do this a lot and hence run 
exactly as short or as long as they need to.

Here is an example of a failed run: 
https://s3.amazonaws.com/archive.travis-ci.org/jobs/77499577/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Flink:ProgramInvocationException on Yarn

[Stephan Ewen]

Please subscribe to the mailing list. All your mails are held back and need
to be manually approved.

On Thu, Aug 27, 2015 at 6:49 PM, Alexey Sapozhnikov 
wrote:

> Hello all.
>
> Some clarification: locally everything works great.
> However once we run our Flink on remote linux machine and try to run the
> client program from our machine, using create remote environment- Flink
> JobManager is raising this exception
>
> On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen  wrote:
>
>> If you start the job via the "bin/flink" script, then simply use
>> "ExecutionEnvironment.getExecutionEnvironment()" rather then creating a
>> remote environment manually.
>>
>> That way, hosts and ports are configured automatically.
>>
>> On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger 
>> wrote:
>>
>>> Hi,
>>>
>>> Which values did you use for FLINK_SERVER_URL and FLINK_PORT?
>>> Every time you deploy Flink on YARN, the host and port change, because
>>> the
>>> JobManager is started on a different YARN container.
>>>
>>>
>>> On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer  wrote:
>>>
>>> > Hello All
>>> >
>>> > When using Eclipse IDE to submit Flink to Yarn single node cluster I'm
>>> > getting :
>>> > "org.apache.flink.client.program.ProgramInvocationException: Failed to
>>> > resolve JobManager"
>>> >
>>> > Using Flink 0.9.0
>>> >
>>> > The Jar copy a file from one location in Hdfs to another and works fine
>>> > while executed locally on the single node Yarn cluster -
>>> > bin/flink run -c Test ./examples/MyJar.jar
>>> > hdfs://localhost:9000/flink/in.txt hdfs://localhost:9000/flink/out.txt
>>> >
>>> > The code skeleton:
>>> >
>>> > ExecutionEnvironment envRemote =
>>> > ExecutionEnvironment.createRemoteEnvironment
>>> > (FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT);
>>> > DataSet data =
>>> > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
>>> > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
>>> > envRemote.execute();
>>> >
>>> >
>>> > Please advise,
>>> >
>>> > Hanan Meyer
>>> >
>>>
>>
>>
>

Re: Apache Flink:ProgramInvocationException on Yarn

[Alexey Sapozhnikov]

Hello all.

Some clarification: locally everything works great.
However once we run our Flink on remote linux machine and try to run the
client program from our machine, using create remote environment- Flink
JobManager is raising this exception

On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen  wrote:

> If you start the job via the "bin/flink" script, then simply use
> "ExecutionEnvironment.getExecutionEnvironment()" rather then creating a
> remote environment manually.
>
> That way, hosts and ports are configured automatically.
>
> On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger 
> wrote:
>
>> Hi,
>>
>> Which values did you use for FLINK_SERVER_URL and FLINK_PORT?
>> Every time you deploy Flink on YARN, the host and port change, because the
>> JobManager is started on a different YARN container.
>>
>>
>> On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer  wrote:
>>
>> > Hello All
>> >
>> > When using Eclipse IDE to submit Flink to Yarn single node cluster I'm
>> > getting :
>> > "org.apache.flink.client.program.ProgramInvocationException: Failed to
>> > resolve JobManager"
>> >
>> > Using Flink 0.9.0
>> >
>> > The Jar copy a file from one location in Hdfs to another and works fine
>> > while executed locally on the single node Yarn cluster -
>> > bin/flink run -c Test ./examples/MyJar.jar
>> > hdfs://localhost:9000/flink/in.txt hdfs://localhost:9000/flink/out.txt
>> >
>> > The code skeleton:
>> >
>> > ExecutionEnvironment envRemote =
>> > ExecutionEnvironment.createRemoteEnvironment
>> > (FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT);
>> > DataSet data =
>> > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
>> > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
>> > envRemote.execute();
>> >
>> >
>> > Please advise,
>> >
>> > Hanan Meyer
>> >
>>
>
>

[jira] [Created] (FLINK-2587) FlinkKafkaConsumer may fail with a NPE

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-2587:
-

 Summary: FlinkKafkaConsumer may fail with a NPE
 Key: FLINK-2587
 URL: https://issues.apache.org/jira/browse/FLINK-2587
 Project: Flink
  Issue Type: Bug
  Components: Kafka Connector
Affects Versions: 0.10
Reporter: Robert Metzger


This travis run (https://api.travis-ci.org/jobs/77533759/log.txt?deansi=true) 
failed with the following exception: 
{code}
org.apache.flink.client.program.ProgramInvocationException: The program 
execution failed: Job execution failed.
at org.apache.flink.client.program.Client.run(Client.java:438)
at 
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:122)
at 
org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:96)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.tryExecute(KafkaTestBase.java:267)
at 
org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBigRecordTestTopology(KafkaConsumerTestBase.java:821)
at 
org.apache.flink.streaming.connectors.kafka.KafkaITCase.testBigRecordJob(KafkaITCase.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution 
failed.
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:362)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at 
org.apache.flink.runtime.testingUtils.TestingJobManager$$anonfun$handleTestingMessage$1.applyOrElse(TestingJobManager.scala:260)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
at 
org.apache.flink.runtime.LeaderSessionMessages$$anonfun$receive$1.applyOrElse(LeaderSessionMessages.scala:40)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at 
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceiv

Re: Apache Flink:ProgramInvocationException on Yarn

[Robert Metzger]

I guess you are getting an entire exception after the "org.apache.flink
.client.program.ProgramInvocationException: Failed to
resolve JobManager".
Can you post it here to help us understanding the issue?

On Thu, Aug 27, 2015 at 6:55 PM, Alexey Sapozhnikov 
wrote:

> Hello all.
>
> Some clarification: locally everything works great.
> However once we run our Flink on remote linux machine and try to run the
> client program from our machine, using create remote environment- Flink
> JobManager is raising this exception
>
> On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen  wrote:
>
> > If you start the job via the "bin/flink" script, then simply use
> > "ExecutionEnvironment.getExecutionEnvironment()" rather then creating a
> > remote environment manually.
> >
> > That way, hosts and ports are configured automatically.
> >
> > On Thu, Aug 27, 2015 at 6:39 PM, Robert Metzger 
> > wrote:
> >
> >> Hi,
> >>
> >> Which values did you use for FLINK_SERVER_URL and FLINK_PORT?
> >> Every time you deploy Flink on YARN, the host and port change, because
> the
> >> JobManager is started on a different YARN container.
> >>
> >>
> >> On Thu, Aug 27, 2015 at 6:32 PM, Hanan Meyer 
> wrote:
> >>
> >> > Hello All
> >> >
> >> > When using Eclipse IDE to submit Flink to Yarn single node cluster I'm
> >> > getting :
> >> > "org.apache.flink.client.program.ProgramInvocationException: Failed to
> >> > resolve JobManager"
> >> >
> >> > Using Flink 0.9.0
> >> >
> >> > The Jar copy a file from one location in Hdfs to another and works
> fine
> >> > while executed locally on the single node Yarn cluster -
> >> > bin/flink run -c Test ./examples/MyJar.jar
> >> > hdfs://localhost:9000/flink/in.txt hdfs://localhost:9000/flink/out.txt
> >> >
> >> > The code skeleton:
> >> >
> >> > ExecutionEnvironment envRemote =
> >> > ExecutionEnvironment.createRemoteEnvironment
> >> > (FLINK_SERVER_URL,FLINK PORT,JAR_PATH_ON_CLIENT);
> >> > DataSet data =
> >> > envRemote.readTextFile("hdfs://localhost:9000/flink/in.txt");
> >> > data.writeAsText("hdfs://localhost:9000/flink/out.txt");
> >> > envRemote.execute();
> >> >
> >> >
> >> > Please advise,
> >> >
> >> > Hanan Meyer
> >> >
> >>
> >
> >
>

[jira] [Created] (FLINK-2588) Implement KillOptions

[Matthias J. Sax (JIRA)]

Matthias J. Sax created FLINK-2588:
--

 Summary: Implement KillOptions
 Key: FLINK-2588
 URL: https://issues.apache.org/jira/browse/FLINK-2588
 Project: Flink
  Issue Type: Improvement
  Components: Storm Compatibility
Reporter: Matthias J. Sax
Priority: Minor


In Storm, topologies run forever and need to be terminate via 
`LocalCluster.killTopology()` or `Client.killTopology()`. For both calls, it is 
possible to specify a `KillOptions` parameter. Currently, this parameter is 
ignored.

`KillOptions` in Storm offer only a single option:; `.set_wait_secs(int)`. This 
delays the killing by the specified number of seconds. (Need to double check if 
the call blocks for the specified number of seconds or not).

Furthermore, in Flink `killTopology()` is implemented to send the "stop" 
signal. As the "stop" signal is no hard termination, it would be nice to be 
able to send the "cancel" signal, too.
  # For this, `FlinkKillOptions` should be introduced (extending 
`KillOptions`). With `FlinkKillOptions` it should be possible to set the signal 
type to "cancel".
 # Additionally, an option should be available to make this call blocking (for 
"stop" and "cancel"); ie, the call returns not before the job is cleaned up.
 # Last but not least, a "stop-wait-cancel" feature can be implements: send a 
"stop" signal and wait for a specific time. If job does not finish within this 
time, we assume that the stop signal was ignored and thus, a "cancel" is sent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Flink:ProgramInvocationException on Yarn

[Hanan Meyer]

Hi

1. I have restarted Flink service via stop/start-loval.sh - it have been
restarted successfully ,no errors in log folder
2. default flink port is -6123

Getting this via Eclips IDE:

Thanks


org.apache.flink.client.program.ProgramInvocationException: Failed to
resolve JobManager
at org.apache.flink.client.program.Client.run(Client.java:379)
at org.apache.flink.client.program.Client.run(Client.java:356)
at org.apache.flink.client.program.Client.run(Client.java:349)
at
org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
at
org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
at
org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
at
org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
at Test.main(Test.java:39)
Caused by: java.io.IOException: JobManager at
akka.tcp://flink@FLINK_SERVER_URL:6123/user/jobmanager not reachable.
Please make sure that the JobManager is running and its port is reachable.
at
org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
at
org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
at
org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
at
org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
at org.apache.flink.client.program.Client.run(Client.java:376)
... 7 more
Caused by: akka.actor.ActorNotFound: Actor not found for:
ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL:6123/),
Path(/user/jobmanager)]
at
akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
at
akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$
Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
at
akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
at
akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
at
akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
at
akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
at akka.actor.ActorCell.terminate(ActorCell.scala:369)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)


On Thu, Aug 27, 2015 at 10:47 PM, Robert Metzger 
wrote:

> I guess you are getting an entire exception after the "org.apache.flink
> .client.program.ProgramInvocationException: Failed to
> resolve JobManager".
> Can you post it here to help us understanding the issue?
>
> On Thu, Aug 27, 2015 at 6:55 PM, Alexey Sapozhnikov 
> wrote:
>
> > Hello all.
> >
> > Some clarification: locally everything works great.
> > However once we run our Flink on remote linux machine and try to run the
> > client program from our machine, using create remote environment- Flink
> > JobManager is raising this exception
> >
> > On Thu, Aug 27, 2015 at 7:41 PM, Stephan Ewen  wrote:
> >
> > > If you start the job via the "bi

Re: Apache Flink:ProgramInvocationException on Yarn

[Hanan Meyer]

Hi
 I'm currently using flink 0.9.0 which by maven support Hadoop 1 .
By using flink-clients-0.7.0-hadoop2-incubating.jar with executePlan(Plan
p) method  instead, I'm getting the same exception

Hanan

On Fri, Aug 28, 2015 at 8:35 AM, Hanan Meyer  wrote:

>
> Hi
>
> 1. I have restarted Flink service via stop/start-loval.sh - it have been
> restarted successfully ,no errors in log folder
> 2. default flink port is -6123
>
> Getting this via Eclips IDE:
>
> Thanks
>
>
> org.apache.flink.client.program.ProgramInvocationException: Failed to
> resolve JobManager
> at org.apache.flink.client.program.Client.run(Client.java:379)
> at org.apache.flink.client.program.Client.run(Client.java:356)
> at org.apache.flink.client.program.Client.run(Client.java:349)
> at
> org.apache.flink.client.RemoteExecutor.executePlanWithJars(RemoteExecutor.java:89)
> at
> org.apache.flink.client.RemoteExecutor.executePlan(RemoteExecutor.java:82)
> at
> org.apache.flink.api.java.RemoteEnvironment.execute(RemoteEnvironment.java:71)
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
> at Test.main(Test.java:39)
> Caused by: java.io.IOException: JobManager at
> akka.tcp://flink@FLINK_SERVER_URL:6123/user/jobmanager not reachable.
> Please make sure that the JobManager is running and its port is reachable.
> at
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1197)
> at
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1221)
> at
> org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1239)
> at
> org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReference(JobManager.scala)
> at org.apache.flink.client.program.Client.run(Client.java:376)
> ... 7 more
> Caused by: akka.actor.ActorNotFound: Actor not found for:
> ActorSelection[Anchor(akka.tcp://flink@FLINK_SERVER_URL:6123/),
> Path(/user/jobmanager)]
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> at
> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> at akka.dispatch.BatchingExecutor$
> Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> at
> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> at
> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> at
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> at
> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
> On Thu, Aug 27, 2015 at 10:47 PM, Robert Metzger 
> wrote:
>
>> I guess you are getting an entire exception after the "org.apache.flink
>> .client.program.ProgramInvocationException: Failed to
>> resolve JobManager".
>> Can you post it here to help us understanding the