You can’t. I have file a wish for something like this: https://issues.apache.org/jira/browse/KAFKA-3726. – Best regards, Radek Gruchalski ra...@gruchalski.com de.linkedin.com/in/radgruchalski
Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On June 1, 2016 at 2:58:09 PM, VG (vlin...@gmail.com) wrote: You are wrong Unmesh. Kafka design forces a partition to be on a single node only. My question is around the scalability of the partition itself. How to overcome the restriction of a single node for a partition ? Any clues anyone... On Wed, Jun 1, 2016 at 5:24 PM, Unmesh Joshi <unmeshjo...@gmail.com> wrote: > I do not see why this is a limitation. Any data storage application you use > will be limited by physical capacity of the nodes. > Distributed applications like Kafka (Distributed message broker), HDFS ( > Distributed file system), Cassandra ( distributed key value dB), by design > allow to store huge amount of data by partitioning it on multiple machine. > 'Multiple' here means 'Cloud Scale'. Tens of thousands of machines spanning > across data centres. > This actually has no limit on data storage capacity then. > > As far as duration for persisting per node log, there is no reason why you > can not store it for ever. > On 1 Jun 2016 9:07 a.m., "VG" <vlin...@gmail.com> wrote: > > > Hi, > > > > There are number of messages floating on the internet suggesting that > Kafka > > cannot persist messages infinitely ? > > Primarily that Kafka partitions are pinned to a node and they can’t > > outgrow the storage capacity of a node.. > > > > Can someone help me understand this limitation and how it can be > overcome ? > > > > Regards, > > Vipul > > >
