Re: Flink on EMR Question

2016-01-06 Thread Stephan Ewen
Would it cause problems if I remove it from the "flink-runtime" pom? Seems strange to have a dependency there that we do not even use... On Wed, Jan 6, 2016 at 12:07 PM, Ufuk Celebi wrote: > @Stephan: It was added to the dependency management section in order to > enforce a higher version for S

Re: Flink on EMR Question

2016-01-06 Thread Ufuk Celebi
@Stephan: It was added to the dependency management section in order to enforce a higher version for S3 client, because it was causing problems earlier. > On 06 Jan 2016, at 11:14, Chiwan Park wrote: > > Great! Thanks for addressing! > >> On Jan 6, 2016, at 5:51 PM, Stephan Ewen wrote: >> >>

Re: Flink on EMR Question

2016-01-06 Thread Chiwan Park
Great! Thanks for addressing! > On Jan 6, 2016, at 5:51 PM, Stephan Ewen wrote: > > At a first look, I think that "flink-runtime" does not need Apache Httpclient > at all. I'll try to simply remove that dependency... > > On Wed, Jan 6, 2016 at 7:14 AM, Chiwan Park wrote: > Hi, > > Thanks for

Re: Flink on EMR Question

2016-01-06 Thread Stephan Ewen
At a first look, I think that "flink-runtime" does not need Apache Httpclient at all. I'll try to simply remove that dependency... On Wed, Jan 6, 2016 at 7:14 AM, Chiwan Park wrote: > Hi, > > Thanks for answering me! > > It is happy to hear the problem will be addressed. :) > > About question 2,

Re: Flink on EMR Question

2016-01-05 Thread Chiwan Park
Hi, Thanks for answering me! It is happy to hear the problem will be addressed. :) About question 2, flink-runtime uses Apache Httpclient 4.2.6 and S3 file system api implemented by Amazon uses 4.3.x. There are some API changes, so NoSuchMethodError exception occurs. > On Jan 5, 2016, at 11:5

Re: Flink on EMR Question

2016-01-05 Thread Stephan Ewen
Hi! Concerning (1) We have seen that a few times. The JVMs / Threads do sometimes not properly exit in a graceful way, and YARN is not always able to kill the process (YARN bug). I am currently working on a refactoring of the YARN resource manager (to allow to easy addition of other frameworks) an

Re: Flink on EMR Question

2016-01-05 Thread Maximilian Bode
Hi everyone, Regarding Q1, I believe I have witnessed a comparable phenomenon in a (3-node, non-EMR) YARN cluster. After shutting down the yarn session via `stop`, one container seems to linger around. `yarn application -list` is empty, whereas `bin/yarn-session.sh -q` lists the left-over conta