Also here you can see guys have same idea https://forum.confluent.io/t/what-happens-if-i-increase-log-segment-bytes/5845
On Tue, 22 Apr 2025 at 15:41, Mikhail Fesenko <prog...@gmail.com> wrote: > Currently we have big amount of topics and big data, and using high disk > storage, so rotate files too fast in case of high traffic means less sense, > kafka has 50-100k open files index/offset/log so decrease amount of > files/improve cache for them, it can give many improvements for big > kafka topics > > On Fri, 21 Feb 2025 at 16:14, Chia-Ping Tsai <chia7...@gmail.com> wrote: > >> hi Mikhail >> >> > So Increasing the limit to 3–4GB (or more) would better align with >> today’s storage capabilities. >> Out of curiosity, what is the benefit of using the larger segment? >> >> While changing the type from int to long is a minor change, allowing or >> encouraging users to create large segments requires stronger >> justification. >> >> thanks, >> Chia-Ping >> >> >> Mikhail Fesenko <prog...@gmail.com> 於 2025年2月21日 週五 下午10:31寫道： >> >> > Guys, any thoughts on that? Please let me know! >> > >> > On Thu, 13 Feb 2025 at 22:31, Mikhail Fesenko <prog...@gmail.com> >> wrote: >> > >> > > Hi everyone, >> > > >> > > I’d like to propose a change the data type of log.segment.bytes from >> int >> > > to long, allowing segment sizes beyond the current 2GB limit. >> > > >> > > Rationale: >> > > >> > > Currently, the maximum segment size is constrained by the int type, >> > > capping it at 2GB (max int). However, with modern hardware—such as >> > > large-scale deployments on machines with multiple high-capacity disks >> > > (e.g., EPYC servers with 4×15TB drives)—this limitation makes segment >> > > sizes inefficiently small. And too many handles (log, index, >> metadata). >> > > So Increasing the limit to 3–4GB (or more) would better align with >> > today’s >> > > storage capabilities. >> > > >> > > Would love to hear your thoughts ! >> > > >> > > Best, >> > > >> > > Mikhail Fesenko >> > > >> > >> >
