On another, similar theme, has anybody tried running Samza on Hadoop 2.8.5? I'm experimenting with it right now, and can't get it to recognize the CPU core configuration. I'm curious if anybody knows about an API change between 2.7.x and 2.8.x in how applications are requested.
What would the effect be on a container that was only allowed one CPU core? Would it be ok to trade that off for more containers? Cheers, Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this message is prohibited. The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If you have received this message in error, please notify the sender immediately and delete the original message. On Sun, Feb 23, 2020 at 11:56 AM Malcolm McFarland <mmcfarl...@cavulus.com> wrote: > Hey folks, > > Does anybody have recommendations for resource allocation configs when > running Samza on YARN? Ie, for a box that has 32GB of memory and 4 CPUs -- > and let's say we're running a Samza task with 1000 partitions -- any > suggestions on what to set for: > > *YARN* > yarn.nodemanager.resource.memory-mb > yarn.nodemanager.resource.cpu-vcores > yarn.nodemanager.resource.percentage-physical-cpu-limit > > *SAMZA* > cluster-manager.container.memory.mb > cluster-manager.container.cpu.cores > yarn.am.container.memory.mb > task.opts > yarn.am.opts > job.container.count > job.container.thread.pool.size > > Also, do you recommend scaling up in box YARN node processing capability, > or out in YARN node count? > > Thanks, > Malcolm McFarland > Cavulus > > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any > unauthorized or improper disclosure, copying, distribution, or use of the > contents of this message is prohibited. The information contained in this > message is intended only for the personal and confidential use of the > recipient(s) named above. If you have received this message in error, > please notify the sender immediately and delete the original message. >
