Anyone any thoughts on my below query? Can you please provide a bit more details or point me to the code how samza > is mapping yarn container id to durable sequential integers?
On Sat, Apr 24, 2021 at 10:45 AM Debraj Manna <subharaj.ma...@gmail.com> wrote: > Thanks Jagadish for replying. Yes I am treating it as a opaque durable I'd > to execute some code on a container specific container. > > Just for my understanding can you please provide a bit more details or > point me to the code > how samza is mapping yarn container id to durable sequential integers? > > On Sat, 24 Apr 2021, 01:44 Jagadish Venkatraman, <jagadish1...@gmail.com> > wrote: > >> Hi Debraj, >> >> Clarifying a bit on Yi’s response, since it was referring to the physical >> Yarn container id.. >> >> If there are N Yarn containers, samza.container.ids are generated >> sequentially from 0 to N-1. This ID is meant to be durable - ie., if a >> particular container fails, the Samza AM will restart it with the same ID. >> >> Having said that, you should just treat it as an opaque key that uniquely >> identifies a container within a Samza job. >> >> Could you share some details on how you intend to use this? >> >> On Friday, April 23, 2021, Debraj Manna <subharaj.ma...@gmail.com> wrote: >> >> > Thanks, Yi for replying. >> > >> > I also checked that class. But Container#getId().toString() >> > returna s string like container_e02_1619095810959_0006_10_000004. But I >> am >> > seeing samza.container.id is an integer like 0, 1 that is getting set >> as >> > system var. Can you let me know how Container#getId().toString() is >> getting >> > mapped to an integer? >> > >> > For example, below is the output of ps -ef for samza yarn container and >> I >> > am seeing *-Dsamza.container.id <http://Dsamza.container.id>=2* & >> > *-Dsamza.container.name >> > <http://Dsamza.container.name>=samza-container-2* >> > >> > yarn 7706 7704 7 Apr22 ? 02:08:23 >> > /usr/lib/jvm/zulu-11-amd64/bin/java -Xmx8820M >> > -XX:-OmitStackTraceInFastThrow -XX:NewRatio=8 -Xss256K >> > -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError >> > -XX:HeapDumpPath=/var/lib/heap-dumps/samzajobs >> > -XX:NativeMemoryTracking=summary -Dio.netty.allocator.type=unpooled >> > -Dio.grpc.netty.shaded.io.netty.allocator.type=unpooled -server >> > *-Dsamza.container.id >> > <http://Dsamza.container.id>=2 -Dsamza.container.name >> > <http://Dsamza.container.name>=samza-container-2* >> > -DisThreadContextMapInheritable=true >> > -Dlog4j.configuration=file:/var/lib/hadoop-yarn/cache/ >> > yarn/nm-local-dir/usercache/ubuntu/appcache/application_ >> > 1619095810959_0006/container_e02_1619095810959_0006_10_ >> > 000004/__package/lib/log4j.xml >> > >> -Dsamza.log.dir=/var/log/hadoop-yarn/containers/application_1619095810959_ >> > 0006/container_e02_1619095810959_0006_10_000004 >> > -Djava.io.tmpdir=/var/lib/hadoop-yarn/cache/yarn/nm- >> > local-dir/usercache/ubuntu/appcache/application_ >> > 1619095810959_0006/container_e02_1619095810959_0006_10_ >> > 000004/__package/tmp >> > -cp >> > /etc/hadoop/conf::/var/lib/hadoop-yarn/cache/yarn/nm- >> > local-dir/usercache/ubuntu/appcache/application_ >> > 1619095810959_0006/container_e02_1619095810959_0006_10_ >> > 000004/__package/lib/activation-1.1.jar:/var/lib/ >> > hadoop-yarn/cache/yarn/nm-local-dir/usercache/ubuntu/appcache/ >> > >> > >> > On Fri, Apr 23, 2021 at 11:09 AM Yi Pan <nickpa...@gmail.com> wrote: >> > >> > > Hi, Debraj, >> > > >> > > In YARN environment, Samza uses YARN generated containerIds as >> > > environmental variables to set each container process's >> > samza.container.id >> > > . >> > > i.e. when containers are requested by Samza AM process in YARN, YARN >> RM >> > > will reply with a set of allocated container objects, which is of >> class >> > > org.apache.hadoop.yarn.api.records.Container. That's the resource >> class >> > to >> > > uniquely identify a container in YARN and >> Container#getId().toString() is >> > > the container ID string we set to samza.container.id. >> > > >> > > Best, >> > > >> > > -Yi >> > > >> > > On Wed, Apr 21, 2021 at 11:28 PM Debraj Manna < >> subharaj.ma...@gmail.com> >> > > wrote: >> > > >> > > > The same has been asked in stackoverflow >> > > > < >> > > > >> > > https://stackoverflow.com/questions/67207850/how-does- >> > samza-generate-the-container-id-when-the-application-is-deployed-in-yar >> > > > > >> > > > also. Anyone any thoughts on this? >> > > > >> > > > >> > > > >> > > https://stackoverflow.com/questions/67207850/how-does- >> > samza-generate-the-container-id-when-the-application-is-deployed-in-yar >> > > > >> > > > On Wed, Apr 21, 2021 at 6:08 PM Debraj Manna < >> subharaj.ma...@gmail.com >> > > >> > > > wrote: >> > > > >> > > > > Hi >> > > > > >> > > > > Can someone let me know how is "samza.container.id" generated >> when a >> > > > > samza app is running in yarn? >> > > > > >> > > > > Thanks, >> > > > > >> > > > > >> > > > >> > > >> > >> >> >> -- >> Jagadish >> >
