Re: Resource allocation on YARN

2020-02-27 Thread Abhishek S
on contained in this > message is intended only for the personal and confidential use of the > recipient(s) named above. If you have received this message in error, > please notify the sender immediately and delete the original message. > > > On Tue, Feb 25, 2020 at 2:09 PM Abhishek S w

Re: Resource allocation on YARN

2020-02-25 Thread Abhishek S
Hey Malcolm, **YARN** *yarn.nodemanager.resource.memory-mb (Amount of physical memory, in MB, that can be allocated for containers)* The value for this depends on if there are any other side-car applications on the machine that the node-manager runs on. eg. on your 32GB machine - if other apps on

Re: Problem : upgrade 1.2 to 1.3 LocalApplicationRunner + default system in SamzaApplication.describe results in null MetadataStoreFactory

2020-01-15 Thread Abhishek S
Thunder, I think setting "job.coordinator.system" or "job.default.system" to be the name of your default system (in configs) solves the problem without code changes. Then again, I that its reasonable to expect the default system (set with "withDefaultSystem(systemDescriptor)") to get picked up wh

Re: Problem: upgrade 1.2 to 1.3 caused loss of clean shutdown on SIGTERM

2020-01-10 Thread Abhishek S
ou said you already took > care of shutdown sequence in cluster mode, yes? I'm open to helping on this > one, just let me know. > > thanks, > Thunder > > > > On Tue, Jan 7, 2020 at 7:12 PM Abhishek S wrote: > > > The rational behind moving the shutdown han

Re: Problem: upgrade 1.2 to 1.3 caused loss of clean shutdown on SIGTERM

2020-01-07 Thread Abhishek S
The rational behind moving the shutdown handler out of SamzaContainer was to let standalone jobs (which are usually part of other online applications) maintain their own shutdown hooks. This prevents Samza hooks from running in parallel or causing a deadlock with the shutdown hooks of parent applic