Re: JMXSink for YARN deployment

Hello guys I got a solution. I set -Dcom.sun.management.jmxremote.port=0 to let system assign a unused port. Kyle 2016-01-12 16:54 GMT+08:00 Kyle Lin : > Hello there > > > I run both driver and master on the same node, so I got "Port already in > use" exception. > > Is there any solution to se

Re: JMXSink for YARN deployment

Hello there I run both driver and master on the same node, so I got "Port already in use" exception. Is there any solution to set different port for each component? Kyle 2015-12-05 5:54 GMT+08:00 spearson23 : > Run "spark-submit --help" to see all available options. > > To get JMX to work yo

Re: JMXSink for YARN deployment

Run "spark-submit --help" to see all available options. To get JMX to work you need to: spark-submit --driver-java-options "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=JMX_PORT" --con

Re: JMXSink for YARN deployment

We use a metrics.property file on YARN by submitting applications like this: spark-submit --conf spark.metrics.conf=metrics.properties --class CLASS_NAME --master yarn-cluster --files /PATH/TO/metrics.properties /PATH/TO/CODE.JAR /PATH/TO/CONFIG.FILE APP_NAME -- View this message in context:

Re: JMXSink for YARN deployment

Hi, Jerry said "I'm guessing", so maybe the thing to try is to check if his guess is correct. What about running sudo lsof | grep metrics.properties ? I imagine you should be able to see it if the file was found and read. If Jerry is right, then I think you will NOT see it. Next, how about try

Re: JMXSink for YARN deployment

perties, but the prerequisite is that each container >>> should find this file in their local FS because this file is loaded locally. >>> >>> >>> >>> Besides I think this might be a kind of workaround, a better solution is >>> to fix this by some oth

Re: JMXSink for YARN deployment

s by some other solutions. Thanks Jerry *From:*Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com <mailto:vladimir.tretya...@sematext.com>] *Sent:* Thursday, September 11, 2014 10:08 PM *Cc:* user@spark.apache.org <mailto:user@sp

Re: JMXSink for YARN deployment

>> >> Jerry >> >> >> >> *From:* Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com] >> *Sent:* Thursday, September 11, 2014 10:08 PM >> *Cc:* user@spark.apache.org >> *Subject:* Re: JMXSink for YARN deployment >> >> >

Re: JMXSink for YARN deployment

mailto:user@spark.apache.org> *Subject:* Re: JMXSink for YARN deployment Hi Shao, thx for explanation, any ideas how to fix it? Where should I put metrics.properties file? On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai mailto:saisai.s...@intel.com>> wrote: Hi， I’

Re: JMXSink for YARN deployment

akov [mailto:vladimir.tretya...@sematext.com] > *Sent:* Thursday, September 11, 2014 10:08 PM > *Cc:* user@spark.apache.org > *Subject:* Re: JMXSink for YARN deployment > > > > Hi Shao, thx for explanation, any ideas how to fix it? Where should I put > metrics.properties f

RE: JMXSink for YARN deployment

to fix this by some other solutions. Thanks Jerry From: Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com] Sent: Thursday, September 11, 2014 10:08 PM Cc: user@spark.apache.org Subject: Re: JMXSink for YARN deployment Hi Shao, thx for explanation, any ideas how to fix it? Where should I

Re: JMXSink for YARN deployment

Hi Shao, thx for explanation, any ideas how to fix it? Where should I put metrics.properties file? On Thu, Sep 11, 2014 at 4:18 PM, Shao, Saisai wrote: > Hi， > > > > I’m guessing the problem is that driver or executor cannot get the > metrics.properties configuration file in the yarn container,

RE: JMXSink for YARN deployment

Hi， I’m guessing the problem is that driver or executor cannot get the metrics.properties configuration file in the yarn container, so metrics system cannot load the right sinks. Thanks Jerry From: Vladimir Tretyakov [mailto:vladimir.tretya...@sematext.com] Sent: Thursday, September 11, 2014 7