I do not see why this is a limitation. Any data storage application you use will be limited by physical capacity of the nodes. Distributed applications like Kafka (Distributed message broker), HDFS ( Distributed file system), Cassandra ( distributed key value dB), by design allow to store huge amount of data by partitioning it on multiple machine. 'Multiple' here means 'Cloud Scale'. Tens of thousands of machines spanning across data centres. This actually has no limit on data storage capacity then.
As far as duration for persisting per node log, there is no reason why you can not store it for ever. On 1 Jun 2016 9:07 a.m., "VG" <vlin...@gmail.com> wrote: > Hi, > > There are number of messages floating on the internet suggesting that Kafka > cannot persist messages infinitely ? > Primarily that Kafka partitions are pinned to a node and they can’t > outgrow the storage capacity of a node.. > > Can someone help me understand this limitation and how it can be overcome ? > > Regards, > Vipul >
