Try SemaText : a fully managed monitoring SaaS : http://sematext.com/ ...
On Mon, Dec 22, 2014 at 7:46 PM, Rajasekar Elango
wrote:
> Hi Sa Li,
>
> You can also try jmxtrans + graphite (for charting). jmxtrans has graphite
> output adapter out of the box.
>
> Regards,
> Raja.
>
> On Mon, Dec 22,
Hi Sa Li,
You can also try jmxtrans + graphite (for charting). jmxtrans has graphite
output adapter out of the box.
Regards,
Raja.
On Mon, Dec 22, 2014 at 10:39 PM, YuanJia Li wrote:
> Hi Sa Li,
> You can try to use jmxtrans+opentsdb to monitor kafka. Jmxtrans is
> collecting data with JMX and
Hi Sa Li,
You can try to use jmxtrans+opentsdb to monitor kafka. Jmxtrans is collecting
data with JMX and sending to opentsdb. Opentsdb is graphing and alerting.
YuanJia Li
From: Sa Li
Date: 2014-12-23 08:41
To: users
Subject: kafka monitoring system
Hi, all
I am thinking to make a relia
Hi Sa Li,
Have a look at SPM for monitoring Kafka:
http://sematext.com/spm
http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/
https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+FAQ#SPMFAQ-KafkaMonitoring
Otis
--
Monitoring * Alerting * Anomaly Detection * C
Did you check the GC logs in the server? We ran into this and the default
setting of 1G max heap on the broker process was nowhere near enough. We
currently have set to 4G.
-T
-Original Message-
From: Birla, Lokesh [lokesh.bi...@verizon.com]
Received: Monday, 22 Dec 2014, 5:27PM
To: user
Hello,
I am running 3 brokers, one zookeeper and producer all on separate machine. I
am also sending very low load around 6K msg/sec. Each msg is around 150 bytes
only.
I ran the load for only 5 minutes and during this time, I see leadership
chained very often.
I created 3 partitions.
Here le
Hi, all
I am thinking to make a reliable monitoring system for our kafka production
cluster. I read such from documents:
"Kafka uses Yammer Metrics for metrics reporting in both the server and the
client. This can be configured to report stats using pluggable stats
reporters to hook up to your mo
I have three nodes: 100, 101, and 102
When I restart all of them, seems now everything is ok, but I would like to
paste the error messages I got from server.log from each node, see if you
can help to understand what is the problem.
on node 100
[2014-12-23 00:04:39,401] ERROR [KafkaApi-100] Error
Hello, Neha
This is the error from server.log
[2014-12-22 23:53:25,663] WARN [KafkaApi-100] Fetch request with
correlation id 1227732 from client ReplicaFetcherThread-0-100 on partition
[perf_producer_p8_test,1] failed due to Leader not local for partition
[perf_producer_p8_test,1] on broker 100
Can you share a reproducible test case?
On Tue, Dec 9, 2014 at 7:11 AM, Mohit Kathuria
wrote:
> Neha,
>
> The same issue reoccured with just 2 consumer processes. The exception was
> related to conflict in writing the ephemeral node. Below was the exception.
> Topic name is
> "lst_plugin_com.sp
I restart the kafka server, it is the same thing, sometime nothing listed
on ISR, leader, I checked the state-change log
[2014-12-22 23:46:38,164] TRACE Broker 100 cached leader info
(LeaderAndIsrInfo:(Leader:101,ISR:101,102,100,LeaderEpoch:0,ControllerEpoch:4),ReplicationFactor:3),AllReplicas:101
There is possibly some error in your broker logs. Can you check if you see
any and send it around?
On Mon, Dec 22, 2014 at 2:46 PM, Sa Li wrote:
> Hi, All
>
> I created a topic with 3 replications and 6 partitions, but when I check
> this topic, seems there is no leader and isr were set for this
Terry,
The zookeeper client used by the high level Kafka consumer has a separate
thread that does the heartbeat in the background. So even if it takes long
to process the message, it should not make the consumer's session to time
out or make the consumer rebalance. You may be running into long GC
Hi, All
I created a topic with 3 replications and 6 partitions, but when I check
this topic, seems there is no leader and isr were set for this topic, see
bin/kafka-topics.sh --create --zookeeper 10.100.98.100:2181
--replication-factor 3 --partitions 6 --topic perf_producer_p6_test
SLF4J: Class p
Hi, All
I've run bin/kafka-producer-perf-test.sh on our kafka-production cluster, I
found the number of partitions really have huge impacts on the producer
performance, see:
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.s
We definitely will. Wanted to wait until we got more confident the API was
solid, ran in production for a little, and added a bit more documentation.
Cheers,
Thunder
-Original Message-
From: Joe Stein [joe.st...@stealth.ly]
Received: Monday, 22 Dec 2014, 10:05AM
To: users@kafka.apache.o
Thunder, can you add that to
https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net
didn't know it existed but cool that it uses Rx
On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges
wrote:
> Hi there,
>
> We looked at both of these a while back and ended up writing our own (
> h
Hi there,
We looked at both of these a while back and ended up writing our own
(https://github.com/ntent-ad/kafka4net).
The first one in the list was completely synchronous, and had no concept of
batching. We initially attempted to use the second one (kafka-net) but had some
issues with detec
If that kafka-producer-per-test.sh doesn't work for you and you choose the
DIY approach, https://github.com/sematext/ActionGenerator may be of help.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Dec 22,
What kind of network do you have? gigabit? if so 90 MB/s would make
sense
Also since you have one partition what's your raw transfer speed to the
disk? 90 MB/s makes sense here as well...
If I were looking for rapid replica catch up I'd have at least 2x Gbit and
partitioned topics spread out o
Increasing replica.fetch.max.bytes will help, but will only get diminishing
returns at some point.
Increasing num.replica.fetchers will also help. You need to make sure that
the leaders are balanced in the cluster. See
http://kafka.apache.org/documentation.html#basic_ops_leader_balancing
Thanks,
I double posted by accident, sorry. Have another thread discussing this.
Thanks!
On Dec 22, 2014 11:21 AM, "Jun Rao" wrote:
> Yes, that's a potential issue. Perhaps we just need to have a lower default
> value for metadata.fetch.timeout.ms ?
>
> Thanks,
>
> Jun
>
> On Wed, Dec 17, 2014 at 11:10 P
Perhaps you can try getting some help from the Storm mailing list.
Thanks,
Jun
On Thu, Dec 18, 2014 at 7:22 AM, Nilesh Chhapru <
nilesh.chha...@ugamsolutions.com> wrote:
> Hi All,
>
> Please give some inputs as this is pending since long and need to meet the
> deadlines
>
> Regards,
> Nilesh Ch
Yes, that's a potential issue. Perhaps we just need to have a lower default
value for metadata.fetch.timeout.ms ?
Thanks,
Jun
On Wed, Dec 17, 2014 at 11:10 PM, Paul Pearcy
wrote:
> Heya,
> Playing around with the 0.8.2-beta producer client. One of my test cases
> is to ensure producers can d
You may want to take a look at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-WhydoIseeerror
"Shouldnotsetlogendoffsetonpartition"inthebrokerlog?
Thanks,
Jun
On Wed, Dec 17, 2014 at 9:50 PM, Terry Cumaranatunge
wrote:
> Hello,
>
> I'm using 0.8.2-beta with 3 brokers and 3 zookeeper
have to conduce load and stress test by publishing at least 10K
messages/sec to my Kafka brokers. Here is the setup:
Number of brokers : 2
Topic : 1
Partition :2. Each broker leader of one.
I would like to know if i can use "kafka-producer-per-test.sh" in first
place. also, if any one knows the
Another option is a HTTP wrapper around the actual producer and doing a
HTTP POST from C# to a REST server e.g.
https://github.com/stealthly/dropwizard-kafka-http which I know folks have
done successfully.
/***
Joe Stein
Founder, Principal Consultant
Big
What kind of load testing do you plan to do?
Thanks,
Jun
On Wed, Dec 17, 2014 at 3:42 AM, nitin sharma
wrote:
> Hi Team,
>
> I have to make a decision on whether i should go with Kafka producer test
> utility or build my own java tool for my load testing .
>
> Kindly let me know if anyone know
Hmm, in 0.7, each producer is supposed to have only one ZK connection. Do
you see ZK session expiration in the producer log?
Also, 0.7 is pretty old and will no longer be patched. So, you probably
want to try the latest 0.8.1.1.
Thanks,
Jun
On Tue, Dec 16, 2014 at 11:53 PM, Zhao GuoHao(媒体产品技术中心
We are using kafka version 0.8.1.1 and trying to produce from a C# app.
I realize that there is no official C# library release and want to get your
experience with the existing solutions that are currently available.
I am looking for a solution that is a) stable enough for production environment
Hey Pramod,
few things:
a. You can keep 2 brokers but you can increase the ProducerSend thread on
your producer side to push more messages. best way try to create more
threads that execute the loop where "send" is called.
b. try to avoid/reduce putting any logic computation in the while loop ..
Yo
btw -- curious to know how well your kafka broker handles the load...
please do inform us your result.
Regards,
Nitin Kumar Sharma.
On Mon, Dec 22, 2014 at 9:52 AM, nitin sharma
wrote:
> Hey Pramod,
>
> few things:
> a. You can keep 2 brokers but you can increase the ProducerSend thread on
> y
32 matches
Mail list logo
