Hello, Joe Continue this thread, I got following monitoring tools on my DEV, 1. graphite + statsD 2. kafka-web-console 3. JMX + jconsole 4. kafkaOffsetMonitor 5. Kafka Manager (yahoo just open source it)
They all work fine locally on my dev, but I am thinking to get it work out of box cause I just do not want to install them on my production server, how possible can I install all of them in a VM, remotely connecting the production? It seems I can't find config to allow me to do this, or no such out of box feature? thanks AL On Fri, Jan 9, 2015 at 6:32 PM, Sa Li <sal...@gmail.com> wrote: > Thank you very much, Joe, I will try all of them, keep here posted. > On Jan 9, 2015 5:55 PM, "Joe Stein" <joe.st...@stealth.ly> wrote: > >> Hi, https://github.com/stealthly/metrics-kafka is a project to be used as >> an example of how to use Kafka as a central point to send all of your >> metrics for your entire infrastructure. The consumers integrate so as to >> abstract the load and coupling of services so systems can just send their >> stats to Kafka and then you can do whatever you want with them from there >> (often multiple things). We also build a Yammer Metrics Reporter (which is >> what Kafka uses to send its Metrics) for Kafka itself so brokers can send >> their stats into a Kafka topic and used downstream (typically another >> cluster). The issue you reported was caused by changes by github and I >> just pushed fixes for them so things are working again. >> >> If you are not looking for that type of solution and want to just see and >> chart broker metrics then I would suggest taking a look at >> https://github.com/airbnb/kafka-statsd-metrics2 and you point it to >> https://github.com/kamon-io/docker-grafana-graphite. I find this a very >> quick out the box way to see what is going on with a broker when no stats >> reporter is already in place. If you want a Kafka metrics reporter for >> just >> graphite check out https://github.com/damienclaveau/kafka-graphite for >> just >> ganglie https://github.com/criteo/kafka-ganglia for just Riemann >> https://github.com/TheLadders/KafkaRiemannMetricsReporter and/or you also >> can use a service like SPM >> https://apps.sematext.com/spm-reports/mainPage.do?selectedApplication=4293 >> or DataDog https://www.datadoghq.com/ >> >> Hope this help, thanks! >> >> /******************************************* >> Joe Stein >> Founder, Principal Consultant >> Big Data Open Source Security LLC >> http://www.stealth.ly >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >> ********************************************/ >> >> >> On Fri, Jan 9, 2015 at 7:51 PM, Sa Li <sal...@gmail.com> wrote: >> >> > Hello, all >> > >> > I like to use the tool metrics-kafka which seems to be attractive to >> report >> > kafka metric and use graphite to graph metrics, however I am having >> trouble >> > to make it work. >> > >> > In https://github.com/stealthly/metrics-kafka, it says: >> > >> > In the main metrics-kafka folder >> > >> > 1) sudo ./bootstrap.sh 2) ./gradlew test 3) sudo ./shutdown.sh >> > When I run ./bootstrap, see this is what I got >> > root@DO-mq-dev:/home/stuser/jmx/metrics-kafka# ././bootstrap.sh >> > /dev/stdin: line 1: syntax error near unexpected token `newline' >> > /dev/stdin: line 1: `<!DOCTYPE html>' >> > /dev/stdin: line 1: syntax error near unexpected token `newline' >> > /dev/stdin: line 1: `<!DOCTYPE html>' >> > e348a98a5afb8b89b94fce51b125e8a2045d9834268ec64c3e38cb7b165ef642 >> > 2015/01/09 16:49:21 Error response from daemon: Could not find entity >> for >> > broker1 >> > >> > And this is how I vagrant up: >> > root@DO-mq-dev:/home/stuser/jmx/metrics-kafka# vagrant up >> > /usr/share/vagrant/plugins/provisioners/docker/plugin.rb:13:in >> > `require_relative': >> > /usr/share/vagrant/plugins/provisioners/docker/config.rb:23: syntax >> error, >> > unexpected tPOW (SyntaxError) >> > def run(name, **options) >> > ^ >> > /usr/share/vagrant/plugins/provisioners/docker/config.rb:43: syntax >> error, >> > unexpected keyword_end, expecting $end >> > from >> /usr/share/vagrant/plugins/provisioners/docker/plugin.rb:13:in >> > `block in <class:Plugin>' >> > from /usr/lib/ruby/vendor_ruby/vagrant/registry.rb:27:in `call' >> > from /usr/lib/ruby/vendor_ruby/vagrant/registry.rb:27:in `get' >> > from >> > /usr/share/vagrant/plugins/kernel_v2/config/vm_provisioner.rb:34:in >> > `initialize' >> > from /usr/share/vagrant/plugins/kernel_v2/config/vm.rb:223:in >> `new' >> > from /usr/share/vagrant/plugins/kernel_v2/config/vm.rb:223:in >> > `provision' >> > from /home/stuser/jmx/metrics-kafka/Vagrantfile:29:in `block (2 >> > levels) in <top (required)>' >> > from /usr/lib/ruby/vendor_ruby/vagrant/config/v2/loader.rb:37:in >> > `call' >> > from /usr/lib/ruby/vendor_ruby/vagrant/config/v2/loader.rb:37:in >> > `load' >> > from /usr/lib/ruby/vendor_ruby/vagrant/config/loader.rb:104:in >> > `block (2 levels) in load' >> > from /usr/lib/ruby/vendor_ruby/vagrant/config/loader.rb:98:in >> > `each' >> > from /usr/lib/ruby/vendor_ruby/vagrant/config/loader.rb:98:in >> > `block in load' >> > from /usr/lib/ruby/vendor_ruby/vagrant/config/loader.rb:95:in >> > `each' >> > from /usr/lib/ruby/vendor_ruby/vagrant/config/loader.rb:95:in >> > `load' >> > from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:335:in >> > `machine' >> > from >> /usr/lib/ruby/vendor_ruby/vagrant/plugin/v2/command.rb:142:in >> > `block in with_target_vms' >> > from >> /usr/lib/ruby/vendor_ruby/vagrant/plugin/v2/command.rb:175:in >> > `call' >> > from >> /usr/lib/ruby/vendor_ruby/vagrant/plugin/v2/command.rb:175:in >> > `block in with_target_vms' >> > from >> /usr/lib/ruby/vendor_ruby/vagrant/plugin/v2/command.rb:174:in >> > `map' >> > from >> /usr/lib/ruby/vendor_ruby/vagrant/plugin/v2/command.rb:174:in >> > `with_target_vms' >> > from /usr/share/vagrant/plugins/commands/up/command.rb:56:in >> `block >> > in execute' >> > from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:210:in >> `block >> > (2 levels) in batch' >> > from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:208:in >> `tap' >> > from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:208:in >> `block >> > in batch' >> > from <internal:prelude>:10:in `synchronize' >> > from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:207:in >> > `batch' >> > from /usr/share/vagrant/plugins/commands/up/command.rb:55:in >> > `execute' >> > from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:38:in `execute' >> > from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:484:in >> `cli' >> > from /usr/bin/vagrant:127:in `<main>' >> > >> > Any idea to make it work? >> > >> > thanks >> > >> > >> > -- >> > >> > Alec Li >> > >> > -- Alec Li
