Re: Resource usage of a spark application

On Thu, May 21, 2015 at 5:22 AM Peter Prettenhofer < peter.prettenho...@gmail.com> wrote: > Thanks Akhil, Ryan! > > @Akhil: YARN can only tell me how much vcores my app has been granted but > not actual cpu usage, right? Pulling mem/cpu usage from the OS means i need > to map JVM executor processe

Re: Resource usage of a spark application

Yes Peter that's correct, you need to identify the processes and with that you can pull the actual usage metrics. Thanks Best Regards On Thu, May 21, 2015 at 2:52 PM, Peter Prettenhofer < peter.prettenho...@gmail.com> wrote: > Thanks Akhil, Ryan! > > @Akhil: YARN can only tell me how much vcores

Re: Resource usage of a spark application

Thanks Akhil, Ryan! @Akhil: YARN can only tell me how much vcores my app has been granted but not actual cpu usage, right? Pulling mem/cpu usage from the OS means i need to map JVM executor processes to the context they belong to, right? @Ryan: what a great blog post -- this is super relevant for

Re: Resource usage of a spark application

Hi Peter, a few months ago I was using MetricsSystem to export to Graphite and then view in Grafana; relevant scripts and some instructions are here if you want to take a look. On Sun, May 17, 2015 at 8:48 AM Peter Prettenhofer < peter.prett

Re: Resource usage of a spark application

You can either pull the high level information from your resource manager, or if you want more control/specific information you can write a script and pull the resource usage information from the OS. Something like this