Regarding using of apache kafka

Hi all is it good to Kafka only when very less events daily means 5 millions events per month. I am thinking to use apache Kafka because it integrate nicely with spark streaming in any of the big data service providers like hortonworks,dataBricks Do i need to setup and assign different machine

NULL Record Error. Unable to append a new message to a topic in Camus.

Hello all, I am using Camus to insert messages into Kafka which then I will upload to HDFS. I'm using CDH5.4 with parcels and JSON file format for data I am able to create a new topic and insert messages into it at one go but if I try to insert a new message or append a new message into the topic i

Encoders/Decoders and Serializers/Deserializers

I'm trying to understand the difference between Encoders/Decoders and Serializers/Deserializers and when I should be using each. Specifically the types I'm referring to are: - kafka.serializer.Encoder - kafka.serializer.Decoder - org.apache.kafka.common.serialization.Serializer - org.a

Re: Encoders/Decoders and Serializers/Deserializers

Linkedin/Confluent did a similar thing with their avro schema registry https://github.com/confluentinc/schema-registry/tree/master/avro-serializer/src/main/java/io/confluent/kafka/serializers which might help Not sure of the exact answer the to different between Encoder/Serializer though sorry.

Re: Encoders/Decoders and Serializers/Deserializers

Thanks Dan, My current thinking is that perhaps they do the same job but Encoders are specific to the Scala API whereas Serializers are the Java API equivalent. It seems that the version I'm using (0.8.2.1) has a complete Java Producer API, but not Consumer and that this may have been the source o

Re: New consumer - consumer group init

Hey Stevo, The new consumer doesn't have any threads of its own, so I think construction should be fairly cheap. -Jason On Sun, Jul 19, 2015 at 2:13 PM, Stevo Slavić wrote: > Hello Guozhang, > > It would be enough if consumer group could, besides at construction time, > be set once only after

Re: New producer in production

Hi James, Thanks for the info. For my use case I have a custom partition key assignment to the producers. Regards, Siva. On Fri, Jul 17, 2015 at 1:36 PM, James Cheng wrote: > Be aware that the old producer and new producer have different > partitioning algorithms: > > http://mail-archives.apac

Re: New consumer - consumer group init

Hi Stevo, I am still not very clear on your point yet, I guess I was trying to figure out under which circumstances would users prefer to re-set the group id at an existing consumer rather than creating a new instance. As Jason mentioned, since the new consumer is single threaded it should usually

Consuming from Kafka but don't need to save offsets

Hi, I have a web service that serves up some data that it obtains from a kafka topic. When the process starts up, it wants to load the entire kafka topic into memory, and serve the data up from an in-memory hashtable. The data in the topic has primary keys and is log compacted, and so the total

Re: Consuming from Kafka but don't need to save offsets

James, You can reference confluent IO schema registry implementation. http://docs.confluent.io/1.0/schema-registry/docs/index.html It does similar thing as what you described. A REST front end that serves data from a compacted topic and HA is also provided in the solution. On Tue, 21 Jul 2015 at

kafka-topics.sh - Include topic deletion information?

Would it be possible to enhance the kafka-topics.sh script so that it can show, against the topic it's listing, whether a particular topic is marked for deletion? Right now, to figure out whether a topic has been marked for deletion, one has to use the zookeeper-shell script and list the topics