RE: Partition key not working properly

Hi Svante, Thanks for your reply! As you said, my purpose is let "all messages with the same key goes to the same partition", but the actual case is even I hard code the same partition key(let's say the key is "1") for three messages, the messages are still goes to different partitions. Regar

Re: Partition key not working properly

By default, the partition key is used for hashing then it's placed in a partition that has the appropriate hashed keyspace. If you have three physical partitions and then give the partition key "5" it has nothing to do with physical partition 5 (that does not exist) , similar to physical: partitio

Can Mirroring Preserve Every Topic's Partition?

Howdy friends, I'd like to mirror the topics on several clusters to a central cluster, and I'm looking at using the default Mirrormaker to do so. I've already done some basic testing on the Mirrormaker found here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 and man

Re: dynamically changing log level on a running broker?

You can do this through jconsole on the log4j mbean. Thanks, Jun On Tue, Nov 25, 2014 at 2:56 AM, ben fleis wrote: > Hello hello, > > From what I see at KAFKA-16 < > https://issues.apache.org/jira/browse/KAFKA-16> > and KAFKA-429 , it should > b

Re: Does Kafka Producer service ?

Assuming that you use a single producer in the async mode, the Kafka overhead should be limited to a single thread. Using a cheaper compression codec such as snappy will also help reducing the CPU load. Thanks, Jun On Tue, Nov 25, 2014 at 12:02 PM, Krishna Raj wrote: > Hi Jun, > > Thanks for r

Partition key not working properly

Hi all, I'm struggling with how to use the partition key mechanism properly. My logic is set the partition number as 3, then create three partition keys as "0", "1", "2", then use the partition keys to create three KeyedMessage such as KeyedMessage(topic, "0", message), KeyedMessage(topic,

Re: How many messages does each broker have?

Hi Jiangjie, This is what I have understood. Please correct me if I am wrong I don’t use the partition class at all(KeyedMessage data = new KeyedMessage(topic_name,new_mes). It partitions messages randomly to different partitions. I don’t see it sticking to any broker for 10 mins. I guess it fol

Re: How many messages does each broker have?

Palur, Just adding to what Guozhang said, the answer to your question might depend on which producer you are using. Assuming you are producing messages without keys to the same topic, in new producer(KafkaProducer), the messages will go to brokers in a round robin way, so the messages will end up

Re: logging agent based on fuse and kafka: first release

Great. Thanks for sharing. I added it to our ecosystem wiki. On Tue, Nov 25, 2014 at 9:58 AM, yazgoo wrote: > Hi, > > First I'd like to thank kafka developers for writing kafka. > > This is an announcement for the first release of a f

MetadataResponse error code handling

Hi folks, I was wondering in which cases the PartitionMetadata section of a MetadataResponse [1] can contain useful information? I had been working under the assumption that when the PartitionErrorCode was 0 (NoError) then the rest of the data would be legitimate, and that when the error was non-

Re: How many messages does each broker have?

Thank you Gouzhang. I dont find 0.9 version in the following page https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.1.1 I am looking for kafka jar version 0.9. Can you also tell me how to turn off "flushing messages to disk" in kafka. I never want this to happen Thank you On Tue, Nov 25,

Re: Does Kafka Producer service ?

Hi Jun, Thanks for replying back on this. Appreciated. I do understand that the Kafka Client just needs a protocol compatibility with the Application which is producing the messages. To clarity a bit more: I witnessed a scenario where a large scale website uses the Kafka Library in their Web Ap

Re: Does Kafka Producer service ?

Could you be a bit more specific about the issue? As long as there is protocol compatibility btw the Kafka client and the broker, upgrading the Kafka client library should be easy, right? Thanks, Jun On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj wrote: > Hello Amazing Kafka Creators & User, > >

Re: How many messages does each broker have?

Palur, If the 8 partitions are hosted on each one of the nodes, assuming replication factor 1 then each node will get roughly 10 / 8 messages due to the random partitioner. If you want to know exactly how many messages is on each broker then you can use a simple consumer which allows you to sp

logging agent based on fuse and kafka: first release

Hi, First I'd like to thank kafka developers for writing kafka. This is an announcement for the first release of a file system logging agent based on kafka. It is written for collecting logs from servers running all kind of software, as a generic way to collect logs without needing to know about

Re: [DISCUSSION] adding the serializer api back to the new java producer

Bhavesh, This api change doesn't mean you need to change the format of the encoded data. It simply moves the serialization logic from the application to a pluggable serializer. As long as you preserve the serialization logic, the consumer should still see the same bytes. If you are talking about

Re: benchmark kafka on 10GbE network

Yeah, neither of those are simple to optimize. The CRC is already the optimized java crc we stole from Hadoop. It may be possible to make that faster still but probably not easy. It might be possible to optimize out some of the interrupt calls, though I'm not exactly sure. One thing, though, is th

Re: [DISCUSSION] adding the serializer api back to the new java producer

How will mix bag will work with Consumer side ? Entire site can not be rolled at once so Consumer will have to deals with New and Old Serialize Bytes ? This could be app team responsibility. Are you guys targeting 0.8.2 release, which may break customer who are already using new producer API (be

Re: [DISCUSSION] adding the serializer api back to the new java producer

+1 for this change. what about de-serializer class in 0.8.2? Say i am using new producer with Avro and old consumer combination. then i need to give custom Decoder implementation for Avro right?. On Tue, Nov 25, 2014 at 9:19 PM, Joe Stein wrote: > The serializer is an expected use of the prod

Re: [DISCUSSION] adding the serializer api back to the new java producer

The serializer is an expected use of the producer/consumer now and think we should continue that support in the new client. As far as breaking the API it is why we released the 0.8.2-beta to help get through just these type of blocking issues in a way that the community at large could be involved i

How many messages does each broker have?

Dear Developers, I am using the default partitioning logic(Random Partitioning) to produce messages into brokers. That is I don't use a partitioner.class. My requirement is If I produce 10 messages using the below code for a broker that has 8 partitions across 8 nodes. How many messages will

Re: [DISCUSSION] adding the serializer api back to the new java producer

+1 on this change — APIs are forever. As much as we’d love to see 0.8.2 release ASAP, it is important to get this right. -JW > On Nov 24, 2014, at 5:58 PM, Jun Rao wrote: > > Hi, Everyone, > > I'd like to start a discussion on whether it makes sense to add the > serializer api back to the new

Re: benchmark kafka on 10GbE network

Thanks for the explanation. Here are some stats for I/O thread. *io-ratio 0.155* *io-time-ns-avg 16418.5* *io-wait-ratio 0.59* *io-wait-time-ns-avg 62881* It seems to confirm that IO spent much more time waiting than doing real work. Given the above stats, how could I trace down and pinpoint t

dynamically changing log level on a running broker?

Hello hello, >From what I see at KAFKA-16 and KAFKA-429 , it should be possible to change log levels on a running broker process. The patches appears (to my naive eyes: I speak no Java/MBeans, and ve