I have used Java Melody for this purpose on past Java based servers, but I haven't tried to embed it in Jetty.
-----Original Message----- From: Petersen, Robert (Contr) [mailto:robert.peters...@ftr.com] Sent: Monday, November 06, 2017 4:50 PM To: solr-user@lucene.apache.org Subject: Re: Anyone have any comments on current solr monitoring favorites? Hi Walter, OK now that sounds really interesting. I actually just turned on logging in Jetty and yes did see all the intra-cluster traffic there. I'm pushing our ELK team to pick out the get search requests across the cluster and aggregate them for me. We'll see how that looks but that would just be for user query analysis and not for real time analysis. Still looking for something to monitor real time since apparently my company has all it's new relic licenses tied up with other level one processes and doesn't want to buy any more of them at this time... đ lol And yes when I looked directly at the Graphite data backing Grafana at my last position it was just scary! Thanks Robi PS early adapter for influxDB in general or just for this use case? ________________________________ From: Walter Underwood <wun...@wunderwood.org> Sent: Monday, November 6, 2017 1:44:01 PM To: solr-user@lucene.apache.org Subject: Re: Anyone have any comments on current solr monitoring favorites? We use New Relic across the site, but it doesnât split out traffic to different endpoints. It also cannot distinguish between search traffic to the cluster and intra-cluster traffic. With four shards, the total traffic is 4X bigger than the incoming traffic. We have a bunch of business metrics (orders) and other stuff that is currently in Graphite. Weâll almost certainly move all that to InfluxDB and Grafana. The Solr metrics were overloading the Graphite database, so weâre the first service that is trying InfluxDB. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Nov 6, 2017, at 1:31 PM, Petersen, Robert (Contr) > <robert.peters...@ftr.com> wrote: > > Hi Walter, > > > Yes, now I see it. I'm wondering about using Grafana and New Relic at > the same time since New Relic has a dashboard and also costs money for > corporate use. I guess after a reread you are using Grafana to > visualize the influxDB data and New Relic just for JVM right? Did > this give you more control over the solr metrics you are monitoring? > (PS I've never heard of influxDB) > > > Thanks > > Robi > > ________________________________ > From: Walter Underwood <wun...@wunderwood.org> > Sent: Monday, November 6, 2017 11:26:07 AM > To: solr-user@lucene.apache.org > Subject: Re: Anyone have any comments on current solr monitoring favorites? > > Look back down the string to my post. We use Grafana. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > >> On Nov 6, 2017, at 11:23 AM, Petersen, Robert (Contr) >> <robert.peters...@ftr.com> wrote: >> >> Interesting! Finally a Grafana user... Thanks Daniel, I will follow your >> links. That looks promising. >> >> >> Is anyone using Grafana over Graphite? >> >> >> Thanks >> >> Robi >> >> ________________________________ >> From: Daniel Ortega <danielortegauf...@gmail.com> >> Sent: Monday, November 6, 2017 11:19:10 AM >> To: solr-user@lucene.apache.org >> Subject: Re: Anyone have any comments on current solr monitoring favorites? >> >> Hi Robert, >> >> We use the following stack: >> >> - Prometheus to scrape metrics (https://prometheus.io/) >> - Prometheus node exporter to export "machine metrics" (Disk, network >> usage, etc.) (https://github.com/prometheus/node_exporter) >> - Prometheus JMX exporter to export "Solr metrics" (Cache usage, QPS, >> Response times...) (https://github.com/prometheus/jmx_exporter) >> - Grafana to visualize all the data scrapped by Prometheus ( >> https://grafana.com/) >> >> Best regards >> Daniel Ortega >> >> 2017-11-06 20:13 GMT+01:00 Petersen, Robert (Contr) < >> robert.peters...@ftr.com>: >> >>> PS I knew sematext would be required to chime in here! đ >>> >>> >>> Is there a non-expiring dev version I could experiment with? I think >>> I did sign up for a trial years ago from a different company... I >>> was actually wondering about hooking it up to my personal AWS based solr >>> cloud instance. >>> >>> >>> Thanks >>> >>> Robi >>> >>> ________________________________ >>> From: Emir ArnautoviÄ <emir.arnauto...@sematext.com> >>> Sent: Thursday, November 2, 2017 2:05:10 PM >>> To: solr-user@lucene.apache.org >>> Subject: Re: Anyone have any comments on current solr monitoring favorites? >>> >>> Hi Robi, >>> Did you try Sematextâs SPM? It provides host, JVM and Solr metrics >>> and more. We use it for monitoring our Solr instances and for consulting. >>> >>> Disclaimer - see signature :) >>> >>> Emir >>> -- >>> Monitoring - Log Management - Alerting - Anomaly Detection Solr & >>> Elasticsearch Consulting Support Training - http://sematext.com/ >>> >>> >>> >>>> On 2 Nov 2017, at 19:35, Walter Underwood <wun...@wunderwood.org> wrote: >>>> >>>> We use New Relic for JVM, CPU, and disk monitoring. >>>> >>>> I tried the built-in metrics support in 6.4, but it just didnât do >>>> what >>> we want. We want rates and percentiles for each request handler. >>> That gives us 95th percentile for textbooks suggest or for homework >>> search results page, etc. The Solr metrics didnât do that. The Jetty >>> metrics didnât do that. >>>> >>>> We built a dedicated servlet filter that goes in front of the Solr >>> webapp and reports metrics. It has some special hacks to handle some >>> weird behavior in SolrJ. A request to the â/srpâ handler is sent as >>> â/select?qt=/srpâ, so we normalize that. >>>> >>>> The metrics start with the cluster name, the hostname, and the >>> collection. The rest is generated like this: >>>> >>>> URL: GET /solr/textbooks/select?q=foo&qt=/auto >>>> Metric: textbooks.GET./auto >>>> >>>> URL: GET /solr/textbooks/select?q=foo >>>> Metric: textbooks.GET./select >>>> >>>> URL: GET /solr/questions/auto >>>> Metric: questions.GET./auto >>>> >>>> So a full metric for the cluster âsolr-cloudâ and the host âsearch01" >>> would look like âsolr-cloud.search01.solr.textbooks.GET./auto.m1_rateâ. >>>> >>>> We send all that to InfluxDB. Weâve configured a template so that >>>> each >>> part of the metric name is mapped to a field, so we can write >>> efficient queries in InfluxQL. >>>> >>>> Metrics are graphed in Grafana. We have dashboards that mix >>>> Cloudwatch >>> (for the load balancer) and InfluxDB. >>>> >>>> Iâm still working out the kinks in some of the more complicated >>>> queries, >>> but the data is all there. I also want to expand the servlet filter >>> to report HTTP response codes. >>>> >>>> wunder >>>> Walter Underwood >>>> wun...@wunderwood.org >>>> http://observer.wunderwood.org/ (my blog) >>>> >>>> >>>>> On Nov 2, 2017, at 9:30 AM, Petersen, Robert (Contr) < >>> robert.peters...@ftr.com> wrote: >>>>> >>>>> OK I'm probably going to open a can of worms here... lol >>>>> >>>>> >>>>> In the old old days I used PSI probe to monitor solr running on >>>>> tomcat >>> which worked ok on a machine by machine basis. >>>>> >>>>> >>>>> Later I had a grafana dashboard on top of graphite monitoring >>>>> which was >>> really nice looking but kind of complicated to set up. >>>>> >>>>> >>>>> Even later I successfully just dropped in a newrelic java agent >>>>> which >>> had solr monitors and a dashboard right out of the box, but it costs >>> money for the full tamale. >>>>> >>>>> >>>>> For basic JVM health and Solr QPS and time percentiles, does >>>>> anyone >>> have any favorites or other alternative suggestions? >>>>> >>>>> >>>>> Thanks in advance! >>>>> >>>>> Robi >>>>> >>>>> ________________________________ >>>>> >>>>> This communication is confidential. Frontier only sends and >>>>> receives >>> email on the basis of the terms set out at >>> http://www.frontier.com/email_ disclaimer. >>>> >>> >>> >> >> ________________________________ >> >> This communication is confidential. Frontier only sends and receives email >> on the basis of the terms set out at >> http://www.frontier.com/email_disclaimer. >
