Hello,
We're having the following issue with Kafka and/or Zookeeper:
If a broker (id=1) is running, and you start another broker with id=1, the new
broker will exit saying "A broker is already registered on the path
/brokers/ids/1". However, I noticed when I query zookeeper /brokers/ids/1
disap
o manually re-assign partitions.
Gwen
On Mon, Nov 24, 2014 at 11:15 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Hello,
>
> First, is there a limit to how many Kafka brokers you can have?
>
> Second, if a Kafka broker node fails and I start a new broker on a
Hello,
First, is there a limit to how many Kafka brokers you can have?
Second, if a Kafka broker node fails and I start a new broker on a new node, is
it correct to assume that the cluster will copy data to that node to satisfy
the replication factor specified for a given topic? In other words
e wire protocol
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
>> > ) it is all about how the consumer is designed and developed.
>> >
>> > /***
>> > Joe Stein
>> > Founder, Principal Consultant
>
Hello,
We're looking into using Kafka for a improved version of a system and the
question of how to scale Kafka came up. Specifically, we want to try to make
the system scale as transparently as possible. The concern was that if we go
from N to N*2 consumers that we would have some that are s
Hello,
We're using Kafka 0.8.1.1 and we're trying to run it in a Docker container.
For the most part, this has been fine, however one of the containers has
stopped a couple times and when I look at the log output from Docker (E.g.
Kafka STDOUT), I don't see any errors. At one point it states
Roger,
My understanding of both, beyond what Zookeeper already does, are:
1. Consul can be used to monitor a service and report it's status. This can be
very useful for knowing if a service, such as Zookeeper of Kafka, goes down.
This can be done through a built-in web interface.
2. Confd lev
exception of a few operations) that requires a broker restart can be done
in a rolling manner.
On Wed, Oct 15, 2014 at 7:16 PM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Hello,
>
> We're looking into deploying Kafka and Zookeeper into an environment where
>
Hello,
We're looking into deploying Kafka and Zookeeper into an environment where we
want things to be as easy to stand up and administer. To do this, we're
looking into using Consul, or similar, and Confd to try to make this as
automatic as possible. I was wondering if anyone had an experien
Nevermind...I just found it in the docs and it looks like it has been looked
into.
Casey Sybrandy MSWE
Sr. Software Engineer
CACI/Six3Systems
301-206-6000 (Office)
301-206-6020 (Fax)
11820 West Market Place
Suites N-P
Fulton, MD. 20759
From: Sybrandy, Casey
Sent
on in the broker?
Guozhang
On Wed, Dec 11, 2013 at 8:43 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> First, I saw the partial message looking at raw network traffic via
> Wireshark, not the output of the iterator as the iterator never seems to
> provide me any dat
KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F
> ?
> If that doesn't help, could you file a jira and attach your log?
> Apache
> mailing list doesn't support attachments.
>
> Thanks,
>
> Jun
>
>
> On Wed, Dec 11, 2013 at 6:15 AM, Sybrandy, Casey &l
ead"? Could you upload the
> output and also the log of the consumer here?
>
> Guozhang
>
>
> On Tue, Dec 10, 2013 at 12:26 PM, Sybrandy, Casey <
> casey.sybra...@six3systems.com> wrote:
>
> > Hello,
> >
> > First, I'm using version 0.7.2.
>
Hello,
First, I'm using version 0.7.2.
I'm trying to read some messages from a broker, but looking at wireshark, it
appears that only part of a message is being read by the consumer. After that,
no other data is read and I can verify that there are 10 messages on the
broker. I have the consu
> Thanks,
> Neha
>
>
> On Wed, Nov 6, 2013 at 8:48 AM, Sybrandy, Casey <
> casey.sybra...@six3systems.com> wrote:
>
> > Hello,
> >
> > How can I have the brokers log errors to a file? Do I just have to
> > configure something
Hello,
How can I have the brokers log errors to a file? Do I just have to configure
something like log4j or is something else used?
Thanks.
Casey
Interesting. I didn't know that Kafka had deduplication capabilities. How do
you leverage it? Also, is it supported in Kafka 0.7.x?
-Original Message-
From: Guozhang Wang [mailto:wangg...@gmail.com]
Sent: Tuesday, October 01, 2013 11:33 AM
To: users@kafka.apache.org
Subject: Re: use c
Hello,
What's the difference between the .toIterator() and .iterator() methods for
KafkaStream? I see they return different types and one of my coworkers is
noticing that when he uses .toIterator() will block at times where .iterator()
will not block.
Casey
Another possible solution is to use stunnel to authenticate clients with a
certificate. It's a bit harder to spoof a certificate than an IP address.
-Original Message-
From: Benjamin Black [mailto:b...@b3k.us]
Sent: Thursday, August 29, 2013 1:10 PM
To: users@kafka.apache.org
Subject: R
ug 1, 2013 at 7:30 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Hello,
>
> We're seeing an issue running 0.7.0 where one or more of our consumers
> are pausing after about an hour when we have a lot of threads
> configured. Our setup is as follows:
&g
Hello,
We're seeing an issue running 0.7.0 where one or more of our consumers are
pausing after about an hour when we have a lot of threads configured. Our
setup is as follows:
* 4 brokers configured for 32 threads and 32 partitions on each broker.
* 2 consumers each processi
In the past there was some discussion about having a C client for non-JVM
languages. Is this still planned as well? Being able to work with Kafka from
other languages would be a great thing. Where I work, we interact with Kafka
via Java and Ruby (producer), so having an official C library tha
a.apache.org
Cc: kafka-us...@incubator.apache.org
Subject: Re: Duplicate Messages on the Consumer
Have you actually examined the Kafka files on disk, to make sure those dupes
are really there? Or is this a case of reading the same message more than once?
Philip
On Thu, Jul 18, 2013 at 8:55 AM, Sybrandy, Casey
wrote:
Hello,
We recently started seeing duplicate messages appearing at our consumers.
Thankfully, the database is set up so that we don't store the dupes, but it is
annoying. It's not every message, only about 1% of them. We are running 0.7.0
for the broker with Zookeeper 3.3.4 from Cloudera and
users@kafka.apache.org
Subject: Re: NoBrokersForPartitionException
Could you try 0.7.2?
Thanks,
Jun
On Wed, Jul 10, 2013 at 11:38 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Hello,
>
> Apologies for bringing this back from the dead, but I'm getting the
> same except
Hello,
Apologies for bringing this back from the dead, but I'm getting the same
exception using Kafka 0.7.0. What could be causing this?
Thanks.
Casey
-Original Message-
From: Jun Rao [mailto:jun...@gmail.com]
Sent: Tuesday, March 12, 2013 12:14 AM
To: users@kafka.apache.org
Subject:
IIRC, I think I tried to use stunnel with Kafka once and it worked fine and the
configuration wasn't too bad, at least for a simple configuration.
-Original Message-
From: Dragos Manolescu [mailto:dragos.manole...@servicenow.com]
Sent: Friday, June 07, 2013 4:51 PM
To: users@kafka.apache
That's what I would have assumed. And no, we're not using compression.
Thanks.
From: Jun Rao [mailto:jun...@gmail.com]
Sent: Wednesday, May 08, 2013 11:26 AM
To: users@kafka.apache.org
Cc: Sybrandy, Casey
Subject: Re: Binary Data and Kafka
No. Kafka broker stores the binary data as
All,
Does the Kafka broker Base64 encode the messages? We are sending binary data
to the brokers and I looked at the logs to confirm that they data was being
stored, however all of the data, with a few exceptions, looks to be Base64
encoded. I didn't expect this, so I wanted to ask and confir
Hello,
IIRC, no, it does not. Where I work, one team had the same issue and built
some custom code to handle the encryption and decryption of messages at the
producer and consumer. However, you have to take key management into account
as once a message is written to the broker, you can't decr
What happens if you change the consumer to write to disk or to the console (if
text)? Do you still have the issues? I'm wondering if it's your algorithms
that are taking up so much memory. I would say before you start tweaking
options randomly, start with the simplest case.
Casey
__
If done right, a parody of the Kmart logo might look good as well.
-Original Message-
From: Neha Narkhede [mailto:neha.narkh...@gmail.com]
Sent: Wednesday, December 19, 2012 5:24 PM
To: users@kafka.apache.org
Subject: Re: Logo
I really like the black and white sketches with some stylized
rebalancing is the consumer side GC. If so, you
will see logs saying sth like "expired session" for ZK.
Thanks,
Jun
On Thu, Dec 13, 2012 at 5:54 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Jun,
>
> Funny you should ask...more data was loaded l
eading data issue
Casey,
Not sure what's happening there. Is this reproducible?
Thanks,
Jun
On Wed, Dec 12, 2012 at 9:52 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Hello,
>
> We have a strange issue on one of our systems that we just can't seem
&g
Hello,
We have a strange issue on one of our systems that we just can't seem to figure
out. We loaded in 5000 records yesterday and have not loaded any since then
into our brokers. Examining the logs for the brokers, we confirmed that no
data has been sent to those topics since we did the loa
If he/she uses more than than 1 broker and uses either a zk-based producer or a
broker list based producer with a vip, it should be able to proceed when one of
the brokers is down.
Thanks,
Jun
On Tue, Dec 11, 2012 at 6:58 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
u are using
DefaultEventHandler, there is a trace level logging that tells you which
broker a request is sent to.
Thanks,
Jun
On Mon, Dec 10, 2012 at 8:10 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Is it at least possible to see which broker a message is sent to?
y the partition # in the request itself.
Thanks,
Jun
On Mon, Dec 10, 2012 at 7:26 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Is it possible to log/see which broker, and perhaps partition, a producer
> sent a message to? I'm using the SyncProducer if that matters.
>
A while back I experimented with this and stunnel worked pretty well for this.
Granted, I only did some basic work with it, but it looked promising.
From: Jay Kreps [jay.kr...@gmail.com]
Sent: Friday, December 07, 2012 5:58 PM
To: users@kafka.apache.org
S
Apologies for not responding sooner. My mail client must have been
malfunctioning at the time as I never saw your responses until today.
As for the error, it looks like it's a bug on my part that just didn't click
until I read Jim's responses. I have a config file that I specify the options
i
Re-sending since I didn't get any responses before and want to make sure this
made it to the correct group.
From: Sybrandy, Casey
Sent: Wednesday, November 28, 2012 2:39 PM
To: users@kafka.apache.org
Subject: Zookeeper Configuration Question
All,
We'
All,
We're having an issue with Zookeeper, which has nothing to do with Kafka, but
my consumers don't appear to be attempting to connect with the two nodes that
are up. I specify my zk.connect as such: host1:2181,host2:2181,host3:2181.
Is this correct? Should this work? I didn't see anything
42 matches
Mail list logo
