You could do what you are asking with a custom encoder/decoder so the
message bytes are made up of "messageType+AvroMessage" for the bytes of the
message. The message can be whatever byte structure you want and not just
the avro binary i.e.
https://github.com/linkedin/camus/blob/master/camus-kafka-
Indeed my message size varies b/w ~500kb to ~5mb per avro.
I am using kafka as a I need a scalable pub-sub messaging architecture with
multiple produces and consumers and guaranty of delivery.
Keeping data on filesystem or hdfs won't give me that.
Also In the link below [1] there is a linkedin's
Just wondering Mukesh - the reason you want this feature is because your value
payload is not small (tens of kb). Don't know if that is the right usage of
kafka. It might be worthwhile to store the avro files in a filesystem (regular,
cluster fs, hdfs or even hbase) and the value in your kafka m
Thanks for your response guys,
Filtering messages in the application works fine for me, I am just thinking
out loud as fetching just the key will be much faster than fetching the
entire Message&Metadata and will avoid unnecessary data transfer b/w kafka
& consumer nodes, hence having an API to exp
I think partitioning is best left for the semantics of the message (i.e.
userId, customerId, etc) and not the type of message. If your consumers
only need specific message types then separate the messages types by
topics. This will make the consumers that don't need those message types
work better
Hi,
One option is to partition the data using key and consume from relevant
partition.
Or your current approach (filtering messages in the application) should be
OK.
Using separate getMetaData/getkey and getMessage may hit the consumer
performance/throughput.
Regards,
Kumar
On Fri, Jan 2, 2015
Any pointers guys?
On 1 Jan 2015 15:26, "Mukesh Jha" wrote:
> Hello Experts,
>
> I'm using a kafka topic to store bunch of messages where the key contains
> metadata and value is the data (avro file in our case).
> There are multiple consumers for each topic and the consumer can decide if
> the m
