Re: Kafka 0.7 Quickstart Errors

Perfect Jun! It works. Thanks a ton. On Mon, Jul 8, 2013 at 9:00 AM, Jun Rao wrote: > The following is the weird part. 0:0 is not a valid host and port. Could > you take a look at the EC2 FAQ in > https://cwiki.apache.org/confluence/display/KAFKA/FAQ? It's for the > consumers, but may apply to t

0.8 protocol exception

Hi there. I'm building the 0.8 version of a client to nodejs. I never coded for node and most of my code is following what the prozees guys did (I'm talking to them on updating the lib) But, I'm facing some errors when I test a very simple metadata request. I'm getting this exception on kafka: ja

Re: 0.8 protocol exception

Ok, so I've found out the error: The documentation is outdated, the MetadataRequest BNF should be: NumberOfTopics [TopicList] Had to check the scala source code for that. Is there a place with a most to date doc? Regards On Mon, Jul 8, 2013 at 6:42 AM, Vinicius Carvalho < viniciusccarva...@gm

Does the KAFKA support the FTP data source directly?

Hello, We have lots of logs generated by our production cluster in our FTP server, and we want to try to use the KAFKA to load these data. But as a newer, I don't know whether the KAFKA support the FTP data source directly. I have browsed most of the KAFKA wiki & document & presentation slid

Re: 0.8 protocol exception

Thanks for point this out. Could you help update https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocolaccordingly? Jun On Mon, Jul 8, 2013 at 4:12 AM, Vinicius Carvalho < viniciusccarva...@gmail.com> wrote: > Ok, so I've found out the error: The documentation is outd

kafka multi threaded consumer with very low throughput

I have 2 kafka brokers running on two systems with the same configuration CPU - Dual Core RAM - 4 GB I'm trying to benchmark my kafka setup Number of messages - 1 for 2 partitions & 2 threads configuration - consumer consumes in 1.175 seconds for 1 partition & 1 thread configuration - consu

Re: kafka multi threaded consumer with very low throughput

What is the size of each message? --Tom On Mon, Jul 8, 2013 at 9:04 AM, Anurup Raveendran < anurup.raveend...@fluturasolutions.com> wrote: > I have 2 kafka brokers running on two systems with the same configuration > > CPU - Dual Core > RAM - 4 GB > > I'm trying to benchmark my kafka setup > Nu

Re: kafka multi threaded consumer with very low throughput

305 bytes On Mon, Jul 8, 2013 at 8:41 PM, Tom Brown wrote: > What is the size of each message? > > --Tom > > > On Mon, Jul 8, 2013 at 9:04 AM, Anurup Raveendran < > anurup.raveend...@fluturasolutions.com> wrote: > > > I have 2 kafka brokers running on two systems with the same configuration > >

Re: kafka multi threaded consumer with very low throughput

here is the source code of the program package kafka.examples; import kafka.consumer.ConsumerConfig; import kafka.consumer.KafkaStream; import kafka.javaapi.consumer.ConsumerConnector; import kafka.message.Message; import kafka.consumer.ConsumerIterator; import java.util.HashMap; import java.uti

Kafka issue with "Reconnect due to socket error"

I have a 4-broker Kafka system running in Amazon EC2, and we are using Kafka 0.8 beta1. Most of the standard default configurations remain unchanged. Running the kafka tool ConsumerOffsetChecker is causing socket errors to occur. Some of these socket reset errors are also in the kafka server log.

Re: Changing the number of partitions after a topic is created

Hi Jun, I wonder when will the tool be available? We're very interested in changing the number of partitions for a topic after creation too. Thanks! Tim On Thu, Jul 4, 2013 at 9:06 PM, Jun Rao wrote: > Currently, once a topic is created, the number of partitions can't be > changed. We are wo

Zookeeper vs statics list of brokers

Hi , When I tried to use Zookeeper based broker discovery in 0.7 ,it still asking for broker.list property. But in quickstart page only zk.connect property is shown as needed. Could you please tell me what is mandatory for zookeeper based broker discovery? Thanks, Sujitha "This messa

Re: 0.8 protocol exception

I had the exact same problem when I started writing code for the new protocol. This is an oddity with the way the protocol spec uses EBNF to specify arrays. Checkout the section on protocol primitives, especially arrays. https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protoc

Re: 0.8 protocol exception

Thanks Colin, yep you are right I was missing the ShortString way to represent arrays. All is fine now, I'm able to connect and execute MetadataRequest/Responses. Moving to ProducerRequest/Responses now :) On Mon, Jul 8, 2013 at 5:13 PM, Colin Blower wrote: > I had the exact same problem when I

Re: 0.8 protocol exception

Hey Vinicius, Feel free to contact me if you would like help with the protocol stuff. I'm sure you will be able to figure it out from the wiki and scala code, but it would be nice to make the wiki as detailed as possible for the next set of developers. Thanks for working on a new Kafka client. I

Fwd: having problem with 0.8 gzip compression

I am testing with Kafka 0.8 beta and having problem of receiving message in consumer. There is no error so does anyone have any insights. When I commented out the "compression.code" everything works fine. My producer: public class TestKafka08Prod { public static void main(String [] args) {

Re: kafka multi threaded consumer with very low throughput

Try producing at least 100s MB of data and rerun you test. Thanks, Jun On Mon, Jul 8, 2013 at 8:04 AM, Anurup Raveendran < anurup.raveend...@fluturasolutions.com> wrote: > I have 2 kafka brokers running on two systems with the same configuration > > CPU - Dual Core > RAM - 4 GB > > I'm trying

Re: Kafka issue with "Reconnect due to socket error"

The consumer reconnects because the broker closed the socket. Any error/exception on the broker side around the same time? Thanks, Jun On Mon, Jul 8, 2013 at 10:25 AM, Dennis Haller wrote: > I have a 4-broker Kafka system running in Amazon EC2, and we are using > Kafka 0.8 beta1. Most of the

Re: Changing the number of partitions after a topic is created

We are shooting for providing a patch in a couple of weeks. Thanks, Jun On Mon, Jul 8, 2013 at 10:33 AM, Timothy Chen wrote: > Hi Jun, > > I wonder when will the tool be available? We're very interested in changing > the number of partitions for a topic after creation too. > > Thanks! > > Tim

Re: Zookeeper vs statics list of brokers

Is this for the producer or the consumer? Thanks, Jun On Mon, Jul 8, 2013 at 10:43 AM, Nandigam, Sujitha wrote: > > Hi , > > > When I tried to use Zookeeper based broker discovery in 0.7 ,it still > asking for broker.list property. But in quickstart page only zk.connect > property is shown as

Questions regarding broker

Hi, I have two questions regarding the kafka broker setup. 1. Assuming i have a 4-broker and 2-zookeeper (running in quorum mode) setup, if topicA-partition0 has the leader set to broker4, can I change the leader to other broker without killing the current leader? 2. What is the latency of sw