Review Request 38840: SAMZA-780: fixes an invalid cast of org.apache.hadoop.security.token.Token

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38840/ --- Review request for samza. Repository: samza Description --- [SAMZA-780]

Review Request 38832: SAMZA-779: Remove the dependency on JUnit from SamzaTaskManager

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38832/ --- Review request for samza, Navina Ramesh and Yi Pan (Data Infrastructure). Repos

Graceful shutdown of Samza Task

Hi Yi and all I was wondering if there is a “clean” way of shutting down a samza task. I’m intersted in doing some cleanup and state handling for a graceful shutdown. I am aware of the closable task implementation, but, that is not called using a yarn application -kill If there is a better wa

Re: process killing

What did the SamzaAppMaster log say? It seems to me that the Yarn RM lost the state about the AppMaster instances. We have seen this in a very rare cases in LinkedIn (about once or twice a year) that Yarn leaks out some orphaned processes but really have no chance to reliably reproduce it. If you s

RE: container is running beyond virtual memory limits

Just to give an as complete view of my situation I am compiling what I have done and what my problem is, so maybe you have the most complete information. What I have done is the following in two virtual machines, with 4 cores and 4gb ram each. Install Debian 7.8. Plain with no graphical interfac

Review Request 38807: SAMZA-338 Support metrics report via STDOUT

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/38807/ --- Review request for samza. Repository: samza Description --- Added suppor

RE: process killing

This is an excerpt from the yarn-root-resourcemanager-kfk-samza01.out file. Tell me if you need another file. Thanks, jordi 0001 State change from NEW to SUBMITTED 2015-09-28 11:15:58,081 INFO [ResourceManager Event Processor] capacity.LeafQueue (LeafQueue.java:activateApplications(626)) -

RE: container is running beyond virtual memory limits

I just changed the task options file to add the following line: task.opts=-Xmx128M And I found no change on the behaivour. I see that the job is being launched with the default -Xmx768M value: root 8296 8294 1 11:16 ?00:00:05 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -server -

Re: limits configuration

Hi, Jordi, Yes, that is right. I was just trying to confirm the basics. :) -Yi On Mon, Sep 28, 2015 at 1:51 AM, Jordi Blasi Uribarri wrote: > I copy the same file for both nodes on every change. I understand that it > is the correct way of doing it. > > -Mensaje original- > De: Yi Pan

Re: container is running beyond virtual memory limits

Hi, Jordi, Please find the config variable task.opts in this table: http://samza.apache.org/learn/documentation/0.9/jobs/configuration-table.html This allows you to add additional JVM opts when launching the containers. -Yi On Mon, Sep 28, 2015 at 1:48 AM, Jordi Blasi Uribarri wrote: > The th

RE: limits configuration

I copy the same file for both nodes on every change. I understand that it is the correct way of doing it. -Mensaje original- De: Yi Pan [mailto:nickpa...@gmail.com] Enviado el: lunes, 28 de septiembre de 2015 10:50 Para: dev@samza.apache.org Asunto: Re: limits configuration Just out of

Re: limits configuration

Just out of curiosity, did you just change the yarn-site.xml for RM or for NMs as well? On Fri, Sep 25, 2015 at 3:23 AM, Jordi Blasi Uribarri wrote: > Hi, > > I think that this is more a Yarn question than a Samza one, but I am > really stuck on this. I am having problems understanding some (I b

RE: container is running beyond virtual memory limits

The three tasks have a similar options file, like this one. task.class=flow.OperationJob job.name=flow.OperationJob job.factory.class=org.apache.samza.job.yarn.YarnJobFactory yarn.package.path=http://IP/javaapp.tar.gz systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory sy

Re: JRuby port of Hello Samza

Hi, Elias, That sounds an awesome addition to the Samza eco-system! Do you have a plan to open-source your JRuby version of Hello Samza job? It would be awesome! Thanks a lot! -Yi On Wed, Sep 23, 2015 at 4:13 PM, Elias Levy wrote: > I am exploring Samza for some of our use cases. As we have

Re: container is running beyond virtual memory limits

Hi, Jordi, Can you post your task.opts settings as well? The Xms and Xmx JVM opts will play a role here as well. The Xmx size should be set to less than yarn.container.memory.mb. -Yi On Tue, Sep 22, 2015 at 4:32 AM, Jordi Blasi Uribarri wrote: > I am seeing that I can not get even a single job

Re: process killing

Hm... interesting. What did you see in the application master's logs? I saw that the remaining processes running are SamzaAppMasters. On Tue, Sep 22, 2015 at 1:05 AM, Jordi Blasi Uribarri wrote: > Hi, > > I have two machines running yarn and samza. They are samza 0.9.1 and > hadoop 2.6.0. > > I

Re: YARN scheduler configuration

Hi, Tommy, Sorry to notice this late. I will reach out to our SRE team to see whether Yarn provides any feature like that. Meanwhile, I think that it is worth to check w/ Yarn's mailing list for this question. -Yi On Mon, Sep 14, 2015 at 1:16 PM, Tommy Becker wrote: > We are currently running

Re: Task scheduling

Hi, Alvaro, Sorry I saw this email late. I think that there is a fundamental question to be asked first: why are tasks in the same job have different priority and execution orders? Samza's design principal is that the tasks within a single job should be independent and identical processes, which d