Thanks for the image, appreciate you taking the effort to do that! I'm still hitting this wall. The AM will launch the container, the container will go from "accepted" to "running", but there will be no output from the container (I'm piping all of the Samza, org.apache, org.kafka, and our own application's logging output to a Kafka topic). During these periods, the container will hang out at ~100MB/8GB memory usage and stall. There's no error output when this happens; it just kind of stops. My suspicion is that our Ops group has a firewall rule up that's interfering with this,or maybe just isn't white-listing a port correctly, and if I could identify where the application is stalling, it'd probably help to narrow down the possibilities.
Cheers, Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this message is prohibited. The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If you have received this message in error, please notify the sender immediately and delete the original message. On Thu, May 30, 2019 at 1:39 PM rayman preet <rayman7...@gmail.com> wrote: > I uploaded the image here: > https://www.dropbox.com/s/rv57v165ysp12c5/samza%20flow.png?dl=0 > > Are you still running into this issue? > Is there anything in the container's log that shows any exceptions/errors. > > On Wed, May 22, 2019 at 10:15 PM Malcolm McFarland <mmcfarl...@cavulus.com > > > wrote: > > > Hey rayman, > > > > What it looks like is that the AM has started, the container has started, > > but, ie, here will be the last messages I see in the Samza logs: > > > > 2019-05-23T05:10:45.048Z INFO Making a request for ANY_HOST > > 2019-05-23T05:10:45.057Z INFO Starting the container allocator > > thread > > 2019-05-23T05:10:47.098Z INFO Received new token for : > > <valid_host>:8032 > > 2019-05-23T05:10:47.102Z INFO Container allocated from RM on > > <same_valid_host> > > 2019-05-23T05:10:47.105Z INFO Container allocated from RM on > > <same_valid_host> > > > > At this point, it seems to stall, and no more output is produced. > > > > Also, I couldn't see you diagram (it's possible my company's email > filters > > attachments); can I see that on the web anywhere? > > > > Cheers, > > Malcolm > > > > On Wed, May 22, 2019 at 4:30 PM rayman preet <rayman7...@gmail.com> > wrote: > > > > > Hi Malcolm, > > > > > > This figure (attached) gives an overview of the flow. Is > > > this something you were looking for? > > > > > > Also, by "don't fully start up" do you mean that > > > applications are missing some containers (but the ApplicationMaster is > > > running)? > > > Or the application is missing entirely. > > > > > > -- > > > thanks > > > rayman > > > [image: Samza Job Launch Sequence.png] > > > > > > On Tue, May 21, 2019 at 3:58 PM Malcolm McFarland < > > mmcfarl...@cavulus.com> > > > wrote: > > > > > >> Hey Folks, > > >> > > >> I'm still trying to pin down why these applications are sometimes not > > >> starting. Everything looks fine in the YARN web UI and in the > > >> immediately available logs, but the applications don't always fully > > >> start up. Does anybody have a rundown about how to trace the Samza > > >> startup process on a YARN cluster, from Accepted status, to > > >> localization, to the application master startup, to the actual > > >> application's startup? > > >> > > >> Cheers, > > >> Malcolm > > >> > > >> -- > > >> Malcolm McFarland > > >> Cavulus > > >> > > >> > > >> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any > > >> unauthorized or improper disclosure, copying, distribution, or use of > > >> the contents of this message is prohibited. The information contained > > >> in this message is intended only for the personal and confidential use > > >> of the recipient(s) named above. If you have received this message in > > >> error, please notify the sender immediately and delete the original > > >> message. > > >> > > > > > > > > > -- > > > thanks > > > rayman > > > > > > > > > -- > > Malcolm McFarland > > Cavulus > > 1-800-760-6915 > > mmcfarl...@cavulus.com > > > > > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any > > unauthorized or improper disclosure, copying, distribution, or use of the > > contents of this message is prohibited. The information contained in this > > message is intended only for the personal and confidential use of the > > recipient(s) named above. If you have received this message in error, > > please notify the sender immediately and delete the original message. > > > > > -- > thanks > rayman >
