One thing I noticed about your code. I thought in kafka 0.8.0 topics are
not created automatically on first message. I do not see anywhere in your
code which creates the topics.
I am creating the topic as part of my tests.
Before I was not setting that property, and getting some of the messages.
Do you need to use that configuration to get the tests (as currently
checked in) to pass ?I did not find i needed that particular knob
(although it is a good one to know about).
and Sorry about your suffering... I can sympathize !
- cb
On Wed, Nov 6, 2013 at 7:38 PM, Edward Capriolo wrote
After about 5 days of relentless head pounding, and twittling about
everything under the sun, I figured it out.
If you read to the bottom of this page:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
You find:
consumerProps.put("auto.offset.reset", "smallest");
Now I ca
We don't have this global setting right now - so the guideline is to
capacity plan to set your log retention settings and set alerts on
disk space accordingly.
In order to recover you should be able to change your retention
settings, but the log cleaner is not called on start up so that won't
work
This question seems to come up often - added this to the FAQ.
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Idon%27twantmyconsumer%27soffsetstobecommittedautomatically.CanImanuallymanagemyconsumer%27soffsets%3F
On Tue, Nov 05, 2013 at 04:02:09PM +, Roman Garcia wrote:
> Thanks Neh
So, I've hit this issue with kafka - when disk is full kafka stops
with exception
FATAL [KafkaApi-1] Halting due to unrecoverable I/O error while
handling produce request: (kafka.server.KafkaApis)
kafka.common.KafkaStorageException: I/O exception in append to log 'perf1-2'
I think it will be usef
Each message size has a default limit of around 1 MB then, and multiple
messages are batched together, presumably, up to some other default batch
limit.
Has anyone needed to increase the message limit to greater than 1 MB? If so, to
what limit and were there any issues?
JP
On Nov 5, 2013, a
Ok,
I updated https://issues.apache.org/jira/browse/KAFKA- with this
scenario also.
Jason
On Tue, Nov 5, 2013 at 9:11 PM, Neha Narkhede wrote:
> >. Could a fetch request from a consumer cause a Topic creation request
> (seems
> implausible).
>
> Yes, that seems like a way the broker ca
typeNames is required in settings section for beanName to be included in
the metric name when jmxtrans sends it out. Here's an example:
"outputWriters" : [ {
"@class" :
"com.googlecode.jmxtrans.model.output.GangliaWriter",
"settings" : {
"groupName" : "kafka",
Can do and done!
https://issues.apache.org/jira/browse/KAFKA-1123
On Nov 6, 2013, at 11:04 AM, Jun Rao wrote:
> Yes, that seems like a problem for IPv6. Could you file a jira? In the
> meantime, could you try using the hostname instead of the ip address?
>
> Thanks,
>
> Jun
>
>
> On Wed, N
One file is good for now. I just didn't find any documentation on this, so I
figured I'd ask. Guess I should have just looked at the config directory.
-Original Message-
From: Guozhang Wang [mailto:wangg...@gmail.com]
Sent: Wednesday, November 06, 2013 12:16 PM
To: users@kafka.apache.o
Hi Casey,
Did you want to route all the error log entries to one file and the others
to another file?
Guozhang
On Wed, Nov 6, 2013 at 9:07 AM, Neha Narkhede wrote:
> Yes, configure the kafka/config/log4j.properties that ships with Kafka.
>
> Thanks,
> Neha
>
>
> On Wed, Nov 6, 2013 at 8:48 AM,
Yes, configure the kafka/config/log4j.properties that ships with Kafka.
Thanks,
Neha
On Wed, Nov 6, 2013 at 8:48 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Hello,
>
> How can I have the brokers log errors to a file? Do I just have to
> configure something like log4j or is s
Hello,
How can I have the brokers log errors to a file? Do I just have to configure
something like log4j or is something else used?
Thanks.
Casey
Yes, that seems like a problem for IPv6. Could you file a jira? In the
meantime, could you try using the hostname instead of the ip address?
Thanks,
Jun
On Wed, Nov 6, 2013 at 7:43 AM, Andrew Otto wrote:
> Hm,
>
> Does Kafka support IPv6? I'm trying it now, but I'm just using the
> console-p
We will be using Kafka to transport access logs from frontend cache servers to
HDFS. But! We aren't doing this in production yet, so I wouldn't quite yet
feel comfortable saying we are powered by Kafka. I hope to get this into
production more in a few weeks*, and at that time we'll write a ni
Hm,
Does Kafka support IPv6? I'm trying it now, but I'm just using the
console-producer, which seems to not be able to read in --broker-list with IPv6
addresses. It looks like it is interpreting the colons in the address as the
addy:port separator.
-Andrew
Yes, thank you Andrew this is great stuff.
May I ask how Wikimedia is using Kafka so I can update the powered by
https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
Cool, thanks for sharing this!
-Neha
On Nov 6, 2013 6:16 AM, "Andrew Otto" wrote:
> Hi,
>
> I just got jmxtrans set up with Kafka 0.8 over at Wikimedia. We're using
> puppet, and the https://github.com/wikimedia/puppet-kafka module to
> install and set up Kafka. This module now comes with a
>
Hi,
I just got jmxtrans set up with Kafka 0.8 over at Wikimedia. We're using
puppet, and the https://github.com/wikimedia/puppet-kafka module to install and
set up Kafka. This module now comes with a kafka::server::jmxtrans class which
will automatically set up jmxtrans with some nice Kafka q
20 matches
Mail list logo
