Good evening. I have read through section of monitoring. I tried to map
each section to corresponding JMX attribute. I will appreciate if you
answer a few questions bellow.
Thanks so much in advance,
Vadim
What this JMX
"kafka.controller":type="KafkaController",name="ActiveControllerCount" fo
thanks Jay.
after setting up a kafka instance and running some examples. I think I
understand it better now. as the paper pointed out, Kafka made a
design choice that made the problem simpler: essentially the "at least once
" delivery actually means "at least once given sufficiently long time and
Neha:
thanks for your response. I figured it out. it seems that the new
hadoop-1.x default port is changed from 8020 to 9000, so I had to use port
9000 in my default fs setting
I have to put a line
defaultFS=hdfs://localhost:9000/
something like that into my camus.properties.
camus_...@google
I'm sorry I'm not too familiar with the camus code base. Have you tried
pinging camus_...@googlegroups.com ?
Thanks,
Neha
On Tue, Sep 3, 2013 at 1:40 PM, Yang wrote:
> I tried to run camus with the properties file in the examples dir:
>
> java -cp .. com.linkedin.camus.etl.kafka.CamusJob -
Interesting. What errors/exceptions do you see in the producer logs?
Thanks,
Neha
On Tue, Sep 3, 2013 at 3:28 PM, Rajasekar Elango wrote:
> We found a issue that if broker host is un resolvable, the number of file
> handle keep increasing for every message we produce and eventually it uses
> up
16 GB is a very large heap. GC tuning becomes trickier as the size of the
heap increases. Are you sure you need that much memory to operate the
mirror maker? For us, the following GC settings have worked well -
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Java
Thanks,
Ne
We found a issue that if broker host is un resolvable, the number of file
handle keep increasing for every message we produce and eventually it uses
up all available files handles in operating system. If broker itself is not
running and broker host name is resolvable, open file handles count stays
I tried to run camus with the properties file in the examples dir:
java -cp .. com.linkedin.camus.etl.kafka.CamusJob -P
myproperties.properties
then it says that my output dir does not exist:
~/tools/camus/camus-etl-kafka$ java -cp
target/camus-etl-kafka-0.1.0-SNAPSHOT.jar
com.linkedin.camus
Tomcat uses commons-logging for logging. You might be able to write an adapter
towards Kafka, in a similar way as the log4j-kafka appender. I think this would
be cleaner than writing something Tomcat-specific that intercepts your requests
and logs them through Kafka.
/Max
--
Maxime Petazzoni
Sr
Hello
I have a cluster with:
* three nodes: rb01, rb02 and rb03
* three topics: rb_event, rb_monitor, rb_flow
* three partitions per topic
* replicas 2
The current status is:
Topic Partition Leader Replicas In-Sync replicas
Thanks Neha,
I did not take a thread dump before restarting, will get it when it happens
again. We are using 16 Gigs of jvm heap. Do you have a recommendation on
jvm GC options.?
Thanks,
Raja.
On Tue, Sep 3, 2013 at 12:26 PM, Neha Narkhede wrote:
> 2013-09-01 05:59:27,792 [main-EventThread] IN
in many setups we have production web server logs rotated on local disks,
and then collected using some sort of scp processes.
I guess the ideal way to use kafka is to write a module for tomcat and
catches the request , send through the kafka api. but is there a "quick and
dirty" producer included
Vadim,
If configured properly, Kafka should garbage collect objects fairly
regularly since request/response objects are typically short lived. It
seems that the only thing that would cause memory usage increase
proportional to the number of topics is metrics. One issue with garbage
collection in K
2013-09-01 05:59:27,792 [main-EventThread] INFO
(org.I0Itec.zkclient.ZkClient) - zookeeper state changed (Disconnected)
2013-09-01 05:59:27,692 [main-SendThread(
mandm-zookeeper-asg.data.sfdc.net:2181)] INFO
(org.apache.zookeeper.
ClientCnxn) - Client session timed out, have not
heard from serv
This is a common JVM tuning scenario. You should adjust the values based on
empirical data. See the heap size section of
http://docs.oracle.com/cd/E21764_01/web./e13814/jvm_tuning.htm
On Aug 30, 2013 10:40 PM, "Vadim Keylis" wrote:
> I followed linkedin setup example in the docs and located 3
Since you mentioned you are having trouble with Zookeeper, thought I'd
point you to our recommendations on Zookeeper operations -
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Zookeeper
.
These are fairly general and well known facts about zookeeper operations.
If you are
What you saw is not normal. In sync replicas should always be a subset of
assigned replicas. Did that anomaly happen before or after reassigning
partitions? Is that reproducible?
Thanks,
Jun
On Tue, Sep 3, 2013 at 12:16 AM, Pablo Nebrera <
pablonebr...@eneotecnologia.com> wrote:
> Hello
>
> I
We found that mirrormaker stopped consuming and producing over the week end
(09/01). Just seeing "Client session timed out" messages in mirrormaker
log. I restarted to it today 09/03 to resume processing. Here is the logs
line in reverse order.
2013-09-03 14:20:40,918
[mirrormakerProd_ops-mmrs1-1
Most memory used by Kafka in serving requests and replicating the data is
transient. So, that typically won't cause OOME. How large is the
produce/fetch purgatory size (there is a jmx bean for each)?
Thanks,
Jun
On Mon, Sep 2, 2013 at 10:54 PM, Vadim Keylis wrote:
> Jun. I will try to do heap
Mark,
I had the same question! Camus is super awesome, but doesn't have out of the
box support for just writing Strings into HDFS. I submitted this pull request
to support that:
https://github.com/linkedin/camus/pull/28
You can clone this directly from the wikimedia branch of Camus:
https:/
20 matches
Mail list logo
