Re: Flink on EMR

2017-09-28 Thread Stefan Richter
Hi, for issue 1, you could delete the slf4j jar from Flink’s lib folder, but I wonder if this producing any problems even with the warning? For issue 2, my question is where you found that only 5GB have been allocated? Did you consider that Flink only allocates a fraction of the memory for hea

Re: Flink on EMR

2017-09-26 Thread Navneeth Krishnan
Hi All, Any suggestions? Thanks. On Mon, Sep 25, 2017 at 10:14 PM, Navneeth Krishnan < reachnavnee...@gmail.com> wrote: > Hello All, > > I'm trying to deploy flink on AWS EMR and I'm very new to EMR. I'm running > into multiple issues and need some help. > > *Issue1:* > > How did others resolve

Re: Flink on EMR

2017-09-25 Thread Navneeth Krishnan
Hi, I’m using the default flink package that comes with EMR. I’m facing the issue while running my pipeline. Thanks. On Mon, Sep 25, 2017 at 11:09 PM Jörn Franke wrote: > Amazon EMR has already a Flink package. You just need to check the > checkbox. I would not install it on your own. > I think

Re: Flink on EMR

2017-09-25 Thread Jörn Franke
Amazon EMR has already a Flink package. You just need to check the checkbox. I would not install it on your own. I think you can find it in the advanced options. > On 26. Sep 2017, at 07:14, Navneeth Krishnan wrote: > > Hello All, > > I'm trying to deploy flink on AWS EMR and I'm very new to

Re: Flink on EMR Question

2016-01-06 Thread Stephan Ewen
Would it cause problems if I remove it from the "flink-runtime" pom? Seems strange to have a dependency there that we do not even use... On Wed, Jan 6, 2016 at 12:07 PM, Ufuk Celebi wrote: > @Stephan: It was added to the dependency management section in order to > enforce a higher version for S

Re: Flink on EMR Question

2016-01-06 Thread Ufuk Celebi
@Stephan: It was added to the dependency management section in order to enforce a higher version for S3 client, because it was causing problems earlier. > On 06 Jan 2016, at 11:14, Chiwan Park wrote: > > Great! Thanks for addressing! > >> On Jan 6, 2016, at 5:51 PM, Stephan Ewen wrote: >> >>

Re: Flink on EMR Question

2016-01-06 Thread Chiwan Park
Great! Thanks for addressing! > On Jan 6, 2016, at 5:51 PM, Stephan Ewen wrote: > > At a first look, I think that "flink-runtime" does not need Apache Httpclient > at all. I'll try to simply remove that dependency... > > On Wed, Jan 6, 2016 at 7:14 AM, Chiwan Park wrote: > Hi, > > Thanks for

Re: Flink on EMR Question

2016-01-06 Thread Stephan Ewen
At a first look, I think that "flink-runtime" does not need Apache Httpclient at all. I'll try to simply remove that dependency... On Wed, Jan 6, 2016 at 7:14 AM, Chiwan Park wrote: > Hi, > > Thanks for answering me! > > It is happy to hear the problem will be addressed. :) > > About question 2,

Re: Flink on EMR Question

2016-01-05 Thread Chiwan Park
Hi, Thanks for answering me! It is happy to hear the problem will be addressed. :) About question 2, flink-runtime uses Apache Httpclient 4.2.6 and S3 file system api implemented by Amazon uses 4.3.x. There are some API changes, so NoSuchMethodError exception occurs. > On Jan 5, 2016, at 11:5

Re: Flink on EMR Question

2016-01-05 Thread Stephan Ewen
Hi! Concerning (1) We have seen that a few times. The JVMs / Threads do sometimes not properly exit in a graceful way, and YARN is not always able to kill the process (YARN bug). I am currently working on a refactoring of the YARN resource manager (to allow to easy addition of other frameworks) an

Re: Flink on EMR Question

2016-01-05 Thread Maximilian Bode
Hi everyone, Regarding Q1, I believe I have witnessed a comparable phenomenon in a (3-node, non-EMR) YARN cluster. After shutting down the yarn session via `stop`, one container seems to linger around. `yarn application -list` is empty, whereas `bin/yarn-session.sh -q` lists the left-over conta