Re: Running w/ multiple CPUs/container on YARN

Hey Prateek, The upgrade to Hadoop 2.7.6 went fine; everything seems to be working, and access to S3 via an access key/secret pair is working as well. However, my requested tasks are still only getting allocated 1 core, despite requesting more than that. Once again, I have a 3-node cluster that sh

[GitHub] [samza] mynameborat commented on issue #905: SAMZA-2055: Async high level api

mynameborat commented on issue #905: SAMZA-2055: Async high level api URL: https://github.com/apache/samza/pull/905#issuecomment-479058100 `testSinkEndToEndWithKeyWithNullRecords` seems to be flaky. I have a 60% pass rate locally. ![image](https://user-images.githubusercontent.com/4694

[GitHub] [samza] prateekm commented on a change in pull request #981: SAMZA-2152 : Bugfix: Making KafkaSytemAdmin's metadataConsumer accesses thread-safe, enabling StreamRegexMonitors only when requir

prateekm commented on a change in pull request #981: SAMZA-2152 : Bugfix: Making KafkaSytemAdmin's metadataConsumer accesses thread-safe, enabling StreamRegexMonitors only when required URL: https://github.com/apache/samza/pull/981#discussion_r271392441 ## File path: samza-kafka/s

[GitHub] [samza] prateekm commented on a change in pull request #981: SAMZA-2152 : Bugfix: Making KafkaSytemAdmin's metadataConsumer accesses thread-safe, enabling StreamRegexMonitors only when requir

prateekm commented on a change in pull request #981: SAMZA-2152 : Bugfix: Making KafkaSytemAdmin's metadataConsumer accesses thread-safe, enabling StreamRegexMonitors only when required URL: https://github.com/apache/samza/pull/981#discussion_r271393999 ## File path: samza-kafka/s

Re: Running w/ multiple CPUs/container on YARN

Is it still the same message from the AM? The one that says: "Got AM register response. The YARN RM supports container requests with max-mem: 14336, max-cpu: 1" On Tue, Apr 2, 2019 at 12:09 AM Malcolm McFarland wrote: > Hey Prateek, > > The upgrade to Hadoop 2.7.6 went fine; everything seems to

Re: Running w/ multiple CPUs/container on YARN

And just to double check, you also changed the yarn.resourcemanager.scheduler.class to CapacityScheduler? On Tue, Apr 2, 2019 at 9:49 AM Prateek Maheshwari wrote: > Is it still the same message from the AM? The one that says: "Got AM > register response. The YARN RM supports container requests w

[GitHub] [samza] rmatharu commented on issue #981: SAMZA-2152 : Bugfix: Making KafkaSytemAdmin's metadataConsumer accesses thread-safe, enabling StreamRegexMonitors only when required

rmatharu commented on issue #981: SAMZA-2152 : Bugfix: Making KafkaSytemAdmin's metadataConsumer accesses thread-safe, enabling StreamRegexMonitors only when required URL: https://github.com/apache/samza/pull/981#issuecomment-479104652 addressed your comments. adminClient accesses seem

Re: Running w/ multiple CPUs/container on YARN

Hi Prateek, I'm not getting an error now, just an unyielding vcore allotment of 1. I just verified that we're setting yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler and yarn.scheduler.capacity.resource-calculator=org.apache

Re: Running w/ multiple CPUs/container on YARN

Interestingly, I just tried setting yarn.scheduler.minimum-allocation-vcores=2 and restarting everything. On startup, the RM now displays a Minimum Allocation of , but my application container still shows "Resource:4096 Memory, 1 VCores". The statistics page for the "default" queue shows "Used Reso

Re: Running w/ multiple CPUs/container on YARN

Found the issue, and thank goodness it was a configuration issue on my end: I was setting the yarn.scheduler.maximum-allocation-vcores too low and artificially constraining the cluster. This is (as the name implies) the maximum allocation for the entire cluster; I had interpreted the description fr