Re: Best way to erase consumer state

One thing might worth checking is the offset.storage value in your consumer config. Is it set to ³kafka² or ³zookeeper² for you consumer? If the offset is set to kafka, then the offset will be stored in the offset topic in Kafka instead of in zookeeper. So deleting zookeeper path won¹t work for tho

Re: How to send serialized object in Kafka Producer

Dear Guozhang, Sorry for the late reply. Please see the sample code which i written to send the bean object. import java.util.Properties; import kafka.producer.KeyedMessage; import kafka.producer.ProducerConfig; public class KafkaProducerBean { final static String TOPIC = "*alarm*"; public sta

Producer connection unsucessfull

Hi, I've installed the Hortonworks Sandbox and try to get into Kafka. Unfortunately, even the simple tutorial does not work :( http://kafka.apache.org/documentation.html#introduction If I try to send a message via bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test I get

Re: Producer connection unsucessfull

I haven't run the sandbox but check if the kafka server is started at all. ps -ef | grep kafka 2014-12-05 14:34 GMT+01:00 Marco : > Hi, > > I've installed the Hortonworks Sandbox and try to get into Kafka. > > Unfortunately, even the simple tutorial does not work :( > http://kafka.apache.org/d

Re: Producer connection unsucessfull

Yes, it's online and version -0.8.1.2.2.0.0-1084. jps lists it also 2014-12-05 14:56 GMT+01:00 svante karlsson : > I haven't run the sandbox but check if the kafka server is started at all. > > ps -ef | grep kafka > > > > 2014-12-05 14:34 GMT+01:00 Marco : > >> Hi, >> >> I've installed the Hortonw

Re: Producer connection unsucessfull

I think the default port for kafka running there is 6667. Can you check server.properties to see whats the port number -Harsha On Fri, Dec 5, 2014, at 06:10 AM, Marco wrote: > Yes, it's online and version -0.8.1.2.2.0.0-1084. jps lists it also > > 2014-12-05 14:56 GMT+01:00 svante karlsson : > >

Re: Producer connection unsucessfull

The port in server.configuration is indeed 6667. bin/kafka-console-producer.sh --broker-list localhost:6667 --topic test -> same error :( I've tried also to change the port, use the hostname instead of localhost I'm running the stuff in VMWare with sharing IP-address from my host...don't k

Which consumer to use with Kafka 0.8.2-beta

We are upgrading to 0.8.2 and have rewritten our producer using the new org.apache.kafka.clients.producer.KafkaProducer which works great. Should we also convert the consumers to use org.apache.kafka.clients.consumer.KafkaConsumer.java or wait for 0.9.3 for the new consumer client. Thanks, Dav

Re: Which consumer to use with Kafka 0.8.2-beta

The new consumer client is slated for 0.9.0 https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

RE: Which consumer to use with Kafka 0.8.2-beta

So although he code is in the 0.8.2 client jar, we should hold off. Thanks. -Original Message- From: Joe Stein [mailto:joe.st...@stealth.ly] Sent: Friday, December 05, 2014 10:17 AM To: users@kafka.apache.org Subject: Re: Which consumer to use with Kafka 0.8.2-beta The new consumer clie

Re: Producer connection unsucessfull

Ok, I've download Kafka by myself and that works. Anyways, thx for help, guys! 2014-12-05 15:55 GMT+01:00 Marco : > The port in server.configuration is indeed 6667. > > bin/kafka-console-producer.sh --broker-list localhost:6667 --topic test > > -> same error :( > > I've tried also to change the p

Re: kafka consumer to write into DB

Thank you very much for the reply, Neha, I have a question about consumer, I consume the data from kafka and write into DB, of course I have to create a hash map in memory, load data into memory and bulk copy to DB instead of insert into DB line by line. Does it mean I need to ack each message whil

Re: Producer connection unsucessfull

on sandbox kafka broker is binding to the ip of the VM. You can either use the hostname (sandbox.hortonworks.com) or ip of the sandbox netstat -anp | grep 6667 tcp0 0 :::10.0.2.15:6667 :::* LISTEN 9485/java tcp0 0 :::

Re: kafka consumer to write into DB

if you are using scala/akka this will handle the batching and acks for you. https://github.com/sclasen/akka-kafka#akkabatchconsumer On Fri, Dec 5, 2014 at 9:21 AM, Sa Li wrote: > Thank you very much for the reply, Neha, I have a question about consumer, > I consume the data from kafka and write

Re: kafka consumer to write into DB

Thanks, Neha, is there a java version batch consumer? thanks On Fri, Dec 5, 2014 at 9:41 AM, Scott Clasen wrote: > if you are using scala/akka this will handle the batching and acks for you. > > https://github.com/sclasen/akka-kafka#akkabatchconsumer > > On Fri, Dec 5, 2014 at 9:21 AM, Sa Li

Kafka replication factor mode

Hi All, I have two nodes kafka cluster and using replication factor 2. If both the node is running, then i am able to push data, but if any nodes goes down, then getting exception. I explored and found that, the default replication factor mode is quorum, means at least (f+1) node must be availa

Is Kafka documentation regarding null key misleading?

Hello, I've come across a (seemingly) strange situation when my Kafka producer gave so uneven distribution across partitions. I found that I used null key to produce messages, guided by the following clause in the documentation: "If the key is null, then a random broker partition is picked." Howev

Re: Is Kafka documentation regarding null key misleading?

Yes, it is *very* misleading in my opinion - I've seen so many people surprised with that behaviour... Technically it's 100% correct of course: "If the key is null, then the Producer will assign the message to a random Partition." - that's what actually happens, because assignment is random. Howeve

Re: Is Kafka documentation regarding null key misleading?

If you look under Producer configs you see the following key ‘topic.metadata.refresh.interval.ms’ with a default of 600 * 1000 (10 minutes). It is not entirely clear but this controls how often a producer will a null key partitioner will switch partitions that it is writing to. In my production

Reading only the latest message

What is the best mechanism to retrieve the latest message from a kafka partition. We intend for our producer, on startup or recovery, to read the upstream sequence number in the last message in the partition and request for the upstream system to start sending from that sequence number++. Curr

Re: Can Mirroring Preserve Every Topic's Partition?

Thank you for your replies Guozhang and Neha, though I have some followup questions. I wrote my own Java Consumer and Producer based off of the Kafka Producer API and High Level Consumer. Let's call them MyConsumer and MyProducer. MyProducer uses a custom Partitioner class called SimplePartitioner

Need to use Kafka with spark

Hi Team, I am able to install Kafka in Ubuntu and also Kafka. I need to read data from facebook and do some filter and send it to Mongo db and create a report which will be used analytics team. I would like to know how can i read data from web and the kafka can send it to spark for transformation

Re: How to send serialized object in Kafka Producer

You need to write your own serializer (e.g. BeanSerializer extends Encoder[T]) for bean object and specify that in: properties.put("serializer.class","BeanSerializer") Guozhang

Re: [DISCUSSION] adding the serializer api back to the new java producer

Jiangjie, The issue with adding the serializer in ProducerRecord is that you need to implement all combinations of serializers for key and value. So, instead of just implementing int and string serializers, you will have to implement all 4 combinations. Adding a new producer constructor like Prod

Broker don't get back when killed and restarted

Hi, Kafka group We try to improve the fault-tolerance of kafka cluster. We setup 4 nodes kafka cluster and 3 nodes zookeeper cluster. ubuntu version: Ubuntu 14.04.1 zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT kafka version: kafka_2.8.0-0.8.0 kafka0.x.x.x broker:9092 borker.i

Re: [DISCUSSION] adding the serializer api back to the new java producer

This thread has diverged multiple times now and it would be worth summarizing them. There seems to be the following points of discussion - 1. Can we keep the serialization semantics outside the Producer interface and have simple bytes in / bytes out for the interface (This is what we have today)

Re: [DISCUSSION] adding the serializer api back to the new java producer

Hey Sriram, Thanks! I think this is a very helpful summary. Let me try to address your point about passing in the serde at send time. I think the first objection is really to the paired key/value serializer interfaces. This leads to kind of a weird combinatorial thing where you would have an avr

Re: kafka consumer to write into DB

Not that I know of. On Fri, Dec 5, 2014 at 9:44 AM, Sa Li wrote: > Thanks, Neha, is there a java version batch consumer? > > thanks > > > > On Fri, Dec 5, 2014 at 9:41 AM, Scott Clasen wrote: > > > if you are using scala/akka this will handle the batching and acks for > you. > > > > https://git

Re: Can Mirroring Preserve Every Topic's Partition?

Going back to your previous requirement of ensuring that the data in the target cluster is in the same order as the source cluster, all you need is to specify a key with every record in your data. The mirror maker and its producer takes care of placing all the data for a particular key in the same

Re: Reading only the latest message

You can use the getOffsetsBefore() API and specify -1L to get the offset of the last committed message (at the time of the request) for that partition. On Fri, Dec 5, 2014 at 12:42 PM, Orelowitz, David wrote: > What is the best mechanism to retrieve the latest message from a kafka > partition. >

Re: Broker don't get back when killed and restarted

Have you tried using the latest stable version of Kafka (0.8.1.1) with controlled shutdown? On Fri, Dec 5, 2014 at 2:39 PM, Haeley Yao wrote: > Hi, Kafka group > > We try to improve the fault-tolerance of kafka cluster. We setup 4 nodes > kafka cluster and 3 nodes zookeeper cluster. > > ubuntu v

Re: [DISCUSSION] adding the serializer api back to the new java producer

Hi Jun, Thanks for pointing out this. Yes, putting serialization/deserialization code into record does lose some flexibility. Some more thinking, I think no matter what we do to bind the producer and serializer/deserializer, we can always to the same thing on Record, i.e. We can also have some con

Re: kafka consumer to write into DB

Hi Sa, I created bulk consumer which consumes, processes and post to ElasticSearch. There are config for the size of message consumption. And you can modify the code about what you want to do about the consumed message. https://github.com/reachkrishnaraj/kafka-elasticsearch-standalone-consumer

want to subscribe