Kafka persists messages to disk (would be SSD if that is what you have). However, if you have a large enough memory and are pulling off messages quickly, then the receivers will likely get the messages directly from memory and the write to disk will happen asynchronously in the background. Because Kafka does not wait for the OS to sync to disk before acknowledging receipt you can get data loss which is why Kafka also has the concept of having backup partitions.
-Dave -----Original Message----- From: Laxmi Narayan NIT DGP [mailto:nit.dgp...@gmail.com] Sent: Tuesday, January 3, 2017 1:26 AM To: users@kafka.apache.org Subject: Re: adding more partition where does kafka runs RAM or SSD ? My question how to delete message frm Kafka topics. Is message expire is only way ? *Regards,* *Laxmi Narayan Patel* *MCA NIT Durgapur (2011-2014)* *Mob:- 9741292048,8345847473 * On Tue, Jan 3, 2017 at 12:46 PM, Kunal Gupta <kunal.gu...@cube26.com> wrote: > For your Question 1 No, Message will be read only once by anyone of > the consumer in a consumer group > > For your Question 2 Yes, Because each consumer group will read a > message exactly once. Kafka is meant for Guaranteed Message Processing > > There are three ways of Guaranteed Message Processing 1) Exactly once > 2) At least Once 3) At most once you can get information well in > google. > > If you set auto commit true then message will not be deleted, it all > deals with consumer offset for reading a topic. At which offset > consumer group has read the topic. > > *Thanks, Kunal* > *+91-9958189589* > *Data Analyst* > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798 > <http://dl.acm.org/citation.cfm?id=2790798> * *Second Paper > Publication : http://dl.acm.org/citation.cfm?id=2947432 > <http://dl.acm.org/citation.cfm?id=2947432>* > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > <http://learnhardwithkunalgupta.blogspot.in> * > > > > > > On Tue, Jan 3, 2017 at 12:41 PM, Laxmi Narayan NIT DGP < > nit.dgp...@gmail.com > > wrote: > > > Hi , > > > > If I have kafka cluster along with replication . > > > > Say cluster of 3 kafka-server and replication factor of message with 2. > > > > Que 1: > > > > I have 1-consumer group but 2 consumer in one group reading from > > 1-Topic, > > > > If I enable auto commit true in consumer , will same message will be > > read twice ? > > > > Que2: > > > > If I have 2 consumer group reading from 1-Topic > > > > If I enable auto commit true in consumer , will same message will be > > read twice ? > > > > > > and If I set auto-commit true will message will be deleted ? > > > > > > > > > > > > > > *Regards,* > > *Laxmi Narayan Patel* > > *MCA NIT Durgapur (2011-2014)* > > *Mob:- 9741292048,8345847473 * > > > > On Tue, Jan 3, 2017 at 12:34 PM, Kunal Gupta > > <kunal.gu...@cube26.com> > > wrote: > > > > > I don't think so because more partition can led to unavailability > > > but > yes > > > can led to higher throughput. But it cause more problems like > increasing > > > end to end latency, requires more open file handler and require > > > more > > memory > > > at client side. > > > > > > *Thanks, Kunal* > > > *+91-9958189589* > > > *Data Analyst* > > > *First Paper Publication : > > > **http://dl.acm.org/citation.cfm?id=2790798 > > > <http://dl.acm.org/citation.cfm?id=2790798> * *Second Paper > > > Publication : http://dl.acm.org/citation.cfm?id=2947432 > > > <http://dl.acm.org/citation.cfm?id=2947432>* > > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > > > <http://learnhardwithkunalgupta.blogspot.in> * > > > > > > > > > > > > > > > > > > On Tue, Jan 3, 2017 at 12:25 PM, Laxmi Narayan NIT DGP < > > > nit.dgp...@gmail.com > > > > wrote: > > > > > > > Hi, > > > > > > > > Does adding more partitions to kafka will help in scaling ? > > > > > > > > > > > > *Regards,* > > > > *Laxmi Narayan Patel* > > > > *MCA NIT Durgapur (2011-2014)* > > > > *Mob:- 9741292048,8345847473 * > > > > > > > > > > This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
