Re: Tracing the Samza+YARN startup process

No problem -- I'm happy that we finally figured this out and could share our results. ECS could actually be a good choice for Node Managers; it's easy in ECS to scale node counts up and down and to cycle out unhealthy servers. Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM,

Re: Tracing the Samza+YARN startup process

Great and detailed report! Really appreciate it! -Yi On Tue, Jun 18, 2019 at 2:37 PM Malcolm McFarland wrote: > Just want to follow up on this, for anybody that might be trying to do > something similar. > > There are two things that were getting in the way of us using YARN+Samza on > ECS: 1) Y

Re: Tracing the Samza+YARN startup process

Just want to follow up on this, for anybody that might be trying to do something similar. There are two things that were getting in the way of us using YARN+Samza on ECS: 1) YARN needs to be able to resolve its hostname to something that's publicly available; and 2) Samza needs to be able to open

Re: Tracing the Samza+YARN startup process

Apart from /etc/hosts and /bin/hostname the only other relevant place might be to modify values in /etc/resolv.conf, to point to, e.g., a dnsmasq instance. On Fri, May 31, 2019 at 2:43 PM Malcolm McFarland wrote: > Hey Rayman, > > The ops group and I went through the configuration today and obse

Re: Tracing the Samza+YARN startup process

Hey Rayman, The ops group and I went through the configuration today and observed the YARN containers as they were coming up. We seem to have found the root of the problem, and I'm putting this out there for anybody else that's trying to do something similar on AWS ECS: The ECS container instance

Re: Tracing the Samza+YARN startup process

Yes I think your hunch is right. Each container queries the AM over HTTP to obtain the jobModel that it is supposed to run. The AM runs a HTTP server usually on a dynamically allocated free port on the machine it's running on. So its possible that a firewall rule blocks the container when it tries

Re: Tracing the Samza+YARN startup process

Thanks for the image, appreciate you taking the effort to do that! I'm still hitting this wall. The AM will launch the container, the container will go from "accepted" to "running", but there will be no output from the container (I'm piping all of the Samza, org.apache, org.kafka, and our own appli

Re: Tracing the Samza+YARN startup process

I uploaded the image here: https://www.dropbox.com/s/rv57v165ysp12c5/samza%20flow.png?dl=0 Are you still running into this issue? Is there anything in the container's log that shows any exceptions/errors. On Wed, May 22, 2019 at 10:15 PM Malcolm McFarland wrote: > Hey rayman, > > What it looks

Re: Tracing the Samza+YARN startup process

Hey rayman, What it looks like is that the AM has started, the container has started, but, ie, here will be the last messages I see in the Samza logs: 2019-05-23T05:10:45.048ZINFOMaking a request for ANY_HOST 2019-05-23T05:10:45.057ZINFOStarting the container allocator thr

Re: Tracing the Samza+YARN startup process

Hi Malcolm, This figure (attached) gives an overview of the flow. Is this something you were looking for? Also, by "don't fully start up" do you mean that applications are missing some containers (but the ApplicationMaster is running)? Or the application is missing entirely. -- thanks rayman [im