Hi Saisai,
As I mentioned,
I am getting very less information using the /metrics/json URI and
*also the /metrics/master/json and /metrics/applications/json URIs do not
seem to be working*.
Please verify the following at your end.
The metrics dumped using /metrics/json is as folllows:
{
"version": "3.0.0",
"gauges":
{
"DAGScheduler.job.activeJobs":
{
"value": 2
},
"DAGScheduler.job.allJobs":
{
"value": 8
},
"DAGScheduler.stage.failedStages":
{
"value": 0
},
"DAGScheduler.stage.runningStages":
{
"value": 2
},
"DAGScheduler.stage.waitingStages":
{
"value": 0
},
"app-20151116150148-0001.driver.BlockManager.disk.diskSpaceUsed_MB":
{
"value": 0
},
"app-20151116150148-0001.driver.BlockManager.memory.maxMem_MB":
{
"value": 1590
},
"app-20151116150148-0001.driver.BlockManager.memory.memUsed_MB":
{
"value": 0
},
"app-20151116150148-0001.driver.BlockManager.memory.remainingMem_MB":
{
"value": 1590
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_processingDelay":
{
"value": 142
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_processingEndTime":
{
"value": 1447666380153
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_processingStartTime":
{
"value": 1447666380011
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_schedulingDelay":
{
"value": 0
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_submissionTime":
{
"value": 1447666380011
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_totalDelay":
{
"value": 142
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_processingEndTime":
{
"value": 1447666380153
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_processingStartTime":
{
"value": 1447666380011
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_records":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_submissionTime":
{
"value": 1447666380011
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.receivers":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.retainedCompletedBatches":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.runningBatches":
{
"value": 0
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.totalCompletedBatches":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.totalProcessedRecords":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.totalReceivedRecords":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.unprocessedBatches":
{
"value": 0
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.waitingBatches":
{
"value": 0
}
},
"counters":
{
},
"histograms":
{
},
"meters":
{
},
"timers":
{
"DAGScheduler.messageProcessingTime":
{
"count": 161,
"max": 213.21269999999998,
"mean": 2.848791141751413,
"min": 0.044309999999999995,
"p50": 0.268713,
"p75": 0.37620899999999996,
"p95": 13.734098,
"p98": 26.988401,
"p99": 34.595124,
"p999": 213.21269999999998,
"stddev": 16.63943615966855,
"m15_rate": 27.1366618014181,
"m1_rate": 4.237092045329674,
"m5_rate": 20.796710761839634,
"mean_rate": 1.267708945789378,
"duration_units": "milliseconds",
"rate_units": "calls/second"
}
}
}
Thanks and Regards,
Rakesh
On Mon, Nov 16, 2015 at 3:18 PM, Saisai Shao <[email protected]> wrote:
> it should worked. I tested in my local environment with "curl
> http://localhost:4040/metrics/json/";, there's metrics dumped. For cluster
> metrics, you have to change your base url to point to cluster manager.
>
> Thanks
> Jerry
>
> On Mon, Nov 16, 2015 at 5:42 PM, ihavethepotential <
> [email protected]> wrote:
>
>> Hi all,
>>
>> I am trying to get the metrics using the MetricsServlet sink(that I guess
>> is
>> enabled by default) as mentioned in the Spark documentation:
>>
>> "5. MetricsServlet is added by default as a sink in master, worker and
>> client
>> # driver, you can send http request "/metrics/json" to get a snapshot of
>> all
>> the
>> # registered metrics in json format. For master, requests
>> "/metrics/master/json" and
>> # "/metrics/applications/json" can be sent seperately to get metrics
>> snapshot of
>> # instance master and applications. MetricsServlet may not be configured
>> by
>> self."
>>
>> I am getting very less information using the /metrics/json URI and also
>> the
>> /metrics/master/json and /metrics/applications/json URIs do not seem to be
>> working.
>>
>> My intent is to get the metrics for
>> 1. Cluster summary
>> 2. Apps running and completed
>> 3. Executors summary of the apps
>> 4. Job summary of the apps
>> 5. DAGs metrics for the app jobs
>>
>> Help for the same is greatly appreciated.
>>
>> Thanks and Regards,
>> Rakesh
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-enable-MetricsServlet-sink-in-Spark-1-5-0-tp25392.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>>
>
