To write the client code, you can follow the examples in
http://kafka.apache.org/quickstart.html (0.7) or
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example(0.8).
You don't need to know scala to write the clients in java, but you do need
the scala jar at runtime.
In 0.8, if
This means that the broker closed the socket connection for some reason.
The broker log around the same time should show the reason. Could you dig
that out?
Thanks,
Jun
On Tue, Apr 23, 2013 at 3:35 PM, Karl Kirch wrote:
> I occasionally am getting some batch send errors from the stock async
>
Does this happen on every message that you type in producer console?
Thanks,
Jun
On Tue, Apr 23, 2013 at 4:15 PM, Yin Yin wrote:
> I tried to run the kafka 0.8 version as instructed in Quick Start. The
> kafka server shows the following message when I launch the producer. ERROR
> Error while
It doesn't really matter, but bringing ZK up first is probably better.
Thanks,
Jun
On Tue, Apr 23, 2013 at 12:14 PM, Karl Kirch wrote:
> Thanks,
>
> I'll try to keep an eye on that and see if that fixes the problem.
> Also, how will that affect things if I have a node completely go away
> (i.
Hi,
The API docs are here
-http://people.apache.org/~joestein/kafka-0.7.1-incubating-docs/.
These are not very complete, we will try our best to publish 0.8 APIs
clearly. For Java examples, you can look at examples/ directory. Both
clients exist in the main kafka jar.
Thanks,
Neha
On Tue, Apr 23
Could you run the tool kafka.tools.ConsumerOffsetChecker? It will show you
which consumer instance owns which partition.
Thanks,
Jun
On Tue, Apr 23, 2013 at 12:12 AM, Karl Kirch wrote:
> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10
> partitions per node) and Zookeeper 3.3
./sbt
> project java-examples
> clean
> package
Thanks,
Neha
On Tue, Apr 23, 2013 at 2:07 PM, Yu, Libo wrote:
> Hi,
>
> There is a ReadMe about how to run the two samples.
> If there are instructions about how to compile them,
> that would be very useful. Users can easily develop
> some test ca
I really need the speed of the async producer (unless the sync producer is able
to get up in the 100k/sec range...) so the sync producer is going to be a tough
sell.
I've also double checked my config settings and they're good.
I did notice some slow fsync warnings in the Kafka broker logs tho
Usually when these types of errors are because you're not connecting to the
proper host:port. Double check your configs, make sure everything is
running and listening on the host:port you think they are.
Have you tried using the sync producer to work out your bugs? My guess is
the sync producer wo
Hey Karl, I have a very similar setup (3 kafka 0.7.2 brokers, 3 ZK 3.4.3
nodes) that I'm running right now and am getting the same error on the
producers. Haven't resolved it yet:
ERROR ProducerSendThread--1585663279
kafka.producer.async.ProducerSendThread - Error in handling batch of 200
events
j
Hmmm… that didn't seem to help.
Anyone else see this sort of errors?
Karl
On Apr 23, 2013, at 5:58 PM, Karl Kirch
wrote:
> I'm going to try bumping up the "numRetries" key in my producer config.
> Is this a good option in this case?
> I am using the zookeeper connect option so I'm aware tha
I tried to run the kafka 0.8 version as instructed in Quick Start. The kafka
server shows the following message when I launch the producer. ERROR Error
while fetching metadata for partition [test,0] (kafka.admin.AdminUtils$)
kafka.common.LeaderNotAvailableException: No leader exists for partitio
I'm going to try bumping up the "numRetries" key in my producer config.
Is this a good option in this case?
I am using the zookeeper connect option so I'm aware that I may get stuck
retrying to a failed node, but if it's just a temporary network glitch I'll at
least get a bit more of a chance t
Hi,
I have read many documentation about Kafka but am still confused about
the Java development . If I want to integrate kafka into our system, I need
to write my own producer and consumer in Java. It is not very clear to me
what Java APIs are available to client developers and there is no detaile
I occasionally am getting some batch send errors from the stock async producer.
This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
Is there anyway to check what happens when those batch errors occur?
Or bump up the retry count? (looks like it only did a single retry).
I need the spe
Hi,
There is a ReadMe about how to run the two samples.
If there are instructions about how to compile them,
that would be very useful. Users can easily develop
some test cases from those examples. I am pulling
my hair trying to figure it out:). Thanks.
Regards,
Libo
Thanks. I tried it out and it seems to work properly.
Regards,
Libo
Thanks,
I'll try to keep an eye on that and see if that fixes the problem.
Also, how will that affect things if I have a node completely go away (i.e.
zookeeper and kafka both go offline for a single node).
Should I still bring up zookeeper on that node first or does it matter at that
point?
You should bring up your Zookeeper instances first and then the Kafka
brokers.
On Tue, Apr 23, 2013 at 11:56 AM, Karl Kirch wrote:
> Now to make things even more interesting. I restarted 2 and now it sees
> all 3 nodes.
> I think I've got some sort of weirdness happening with how I'm bringing
>
Now to make things even more interesting. I restarted 2 and now it sees all 3
nodes.
I think I've got some sort of weirdness happening with how I'm bringing
everything online.
I've tried starting up all the zookeeper instances and then each of the
brokers. I've tried starting up all of the ka
I've tried doing what was suggested (creating the topic folders and restarting)
and I'm getting slightly different but still bad results.
Now it'll write to node 1 and 3 but not to 2.
Karl
On Apr 23, 2013, at 1:27 PM, Neha Narkhede
wrote:
> Can you see if you are running to
> https://issues.a
Also keep in mind that anything done at the transport (SSL for example)
layer won't solve your 'at rest' problems.
All messages are written to disk, so unless the broker does some encryption
logic you haven't solved the data visibility issues.
I also think this should be a producer/consumer probl
The other issue with this model is that you're Kafka servers are available on
the internet ... so anyone else can push data into them. At least, if you are
running in a cross-cloud/cross-datacenter environment. We're struggling with
the same design issues right now.
What we've hoped for is that
Can you see if you are running to
https://issues.apache.org/jira/browse/KAFKA-278 ?
Thanks,
Neha
On Tue, Apr 23, 2013 at 11:24 AM, Karl Kirch wrote:
> Zookeeper based.
>
> Karl
>
> On Apr 23, 2013, at 1:15 PM, Neha Narkhede
> wrote:
>
>> What kind of producer do you use ? Zookeeper based or br
Its odd that you found the bundled zookeeper version of Kafka 0.8 to
be 3.3.3. It should be 3.3.4. Also, there is no compatibility with
using the zookeeper 3.4.3 server with the 3.3.4 client.
Thanks,
Neha
On Tue, Apr 23, 2013 at 11:00 AM, Yu, Libo wrote:
> Hi,
>
> I downloaded kafka 0.8 and noti
Zookeeper based.
Karl
On Apr 23, 2013, at 1:15 PM, Neha Narkhede
wrote:
> What kind of producer do you use ? Zookeeper based or broker list ?
>
> Thanks,
> Neha
>
> On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch wrote:
>> For some extra info, this was a restart of the just zookeeper and kafka
Thanks for your answer.
Regards,
Libo
Hi,
I downloaded kafka 0.8 and notice the bundled zookeeper is version 3.3.3.
I am trying to install it on a cluster and so some testing. The cluster already
has zookeeper 3.4.3 running and I am going to use it with kafka 0.8. Is there
any compatibility issue? No previous version kafka is running
What kind of producer do you use ? Zookeeper based or broker list ?
Thanks,
Neha
On Tue, Apr 23, 2013 at 9:02 AM, Karl Kirch wrote:
> For some extra info, this was a restart of the just zookeeper and kafka, our
> clients were all still online.
>
> Karl
>
> On Apr 23, 2013, at 10:50 AM, Karl Kir
Yes,
I think encryption at the message level is a workable solution, as long as
you don't care about exposing the meta data that goes with it (e.g. topic
names, kafka broker/zk server locations, etc.).
Jason
On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers
wrote:
> Hi
>
> We are planning to use
Hi
We are planning to use Kafka, but like others on this list we have a need
to be able to secure communication. The approaches people have suggested on
this list are:
- Encrypt the messages at the producer (e.g
http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+commu
Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions
per node) and Zookeeper 3.3.4 on each node. When we first start up the cluster,
there are no issues and things work just fine, but after a while things start
acting a bit haywire. It looks like we start to miss messages
I cleaned up our wiki a bit.
https://cwiki.apache.org/confluence/display/KAFKA/Index
Thanks,
Jun
For some extra info, this was a restart of the just zookeeper and kafka, our
clients were all still online.
Karl
On Apr 23, 2013, at 10:50 AM, Karl Kirch
wrote:
> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions
> per node) and Zookeeper 3.3.4 on each node. I'm se
Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10 partitions
per node) and Zookeeper 3.3.4 on each node. I'm seeing that only one broker is
being used to actually send messages through. We're seeing this behavior after
a restart and deletion of the kafka data (/tmp/kafka-logs). I
Updated the 0.8 quick start wiki with a link to set up a ZK cluster.
Thanks,
Jun
On Tue, Apr 23, 2013 at 7:42 AM, Chris Curtin wrote:
> Beat me to it ;)
>
> Only caveat is I wouldn't use /tmp for this, since if you're running
> tmpwatch the 'myid' file will get removed unexpectedly since it do
For real time consumers, the overhead from the file system should be small
since the requested data is likely in pagecache and we use zero-copy
transfer.
Thanks,
Jun
On Mon, Apr 22, 2013 at 11:07 PM, Pankaj Misra
wrote:
> Hi All,
>
> I am working on using Kafka for building a highly scalable s
You can run kafka.tools.ConsumerOffsetChecker to check the consumer lag. If
the consumer is lagging, this indicates a problem on the consumer side.
Thanks,
Jun
On Mon, Apr 22, 2013 at 9:13 PM, Andrew Neilson wrote:
> Hmm it is highly unlikely that that is the culprit... There is lots of
> band
Good point.
thanks!
Jason
On Tue, Apr 23, 2013 at 10:42 AM, Chris Curtin wrote:
> Beat me to it ;)
>
> Only caveat is I wouldn't use /tmp for this, since if you're running
> tmpwatch the 'myid' file will get removed unexpectedly since it doesn't
> seem to be changed at it. That was fun to find
Beat me to it ;)
Only caveat is I wouldn't use /tmp for this, since if you're running
tmpwatch the 'myid' file will get removed unexpectedly since it doesn't
seem to be changed at it. That was fun to find :)
We use /var/zookeeper for our storage.
Thanks,
Chris
On Tue, Apr 23, 2013 at 10:30 AM
Thanks Eric - this helps quite a bit.
I will play around with it.
Jason
On Tue, Apr 23, 2013 at 10:21 AM, Eric Sites wrote:
> Jason,
>
> You need to modify the ZooKeeper config and add the following:
>
> dataDir=/tmp/zookeeper
>
> initLimit=50
> syncLimit=2
>
> server.1=kafka001.domain.com:2888
Jason,
You need to modify the ZooKeeper config and add the following:
dataDir=/tmp/zookeeper
initLimit=50
syncLimit=2
server.1=kafka001.domain.com:2888:3888
server.2=kafka002.domain.com:2888:3888
server.3=kafka003.domain.com:2888:3888
# Make sure you open those 2 points on each of the serve
Thanks Chris and Neha.
Chris - I've been through the link you mentioned before. However, that
appears to be using one instance of zookeeper, which makes whichever server
that runs zookeeper as the single point of failure?
Jason
On Tue, Apr 23, 2013 at 8:28 AM, Chris Curtin wrote:
> I following
I following these instructions to get the first 'play' cluster going:
https://cwiki.apache.org/KAFKA/kafka-08-quick-start.html
Instead of running the 3 brokers on the same machine, I ran on on each
machine.
Note that you will need to do a little bit of zookeeper setup to get a
cluster running, I
Anyone going this year? Probably nothing to do with Kafka, but lots of
interesting talks on Solr/Lucene and some "big data" stuff (I think
there's a Storm talk?)
http://www.lucenerevolution.org/
-David
Sounds like you want something like zeromq?
http://zguide.zeromq.org/page:all#Divide-and-Conquer
-David
On 4/23/13 2:07 AM, Pankaj Misra wrote:
Hi All,
I am working on using Kafka for building a highly scalable system. As I
understand and have seen, Kafka broker has a very impressive and sca
It seems there are two underlying things here: storing messages to
stable storage, and making messages available to consumers (i.e.,
storing messages on the broker). One can be achieved simply and reliably
by spooling to local disk, the other requires network and is inherently
less reliable. Bu
47 matches
Mail list logo
