Currently we have big amount of topics and big data, and using high disk storage, so rotate files too fast in case of high traffic means less sense, kafka has 50-100k open files index/offset/log so decrease amount of files/improve cache for them, it can give many improvements for big kafka topics
On Fri, 21 Feb 2025 at 16:14, Chia-Ping Tsai <chia7...@gmail.com> wrote: > hi Mikhail > > > So Increasing the limit to 3–4GB (or more) would better align with > today’s storage capabilities. > Out of curiosity, what is the benefit of using the larger segment? > > While changing the type from int to long is a minor change, allowing or > encouraging users to create large segments requires stronger justification. > > thanks, > Chia-Ping > > > Mikhail Fesenko <prog...@gmail.com> 於 2025年2月21日 週五 下午10:31寫道： > > > Guys, any thoughts on that? Please let me know! > > > > On Thu, 13 Feb 2025 at 22:31, Mikhail Fesenko <prog...@gmail.com> wrote: > > > > > Hi everyone, > > > > > > I’d like to propose a change the data type of log.segment.bytes from > int > > > to long, allowing segment sizes beyond the current 2GB limit. > > > > > > Rationale: > > > > > > Currently, the maximum segment size is constrained by the int type, > > > capping it at 2GB (max int). However, with modern hardware—such as > > > large-scale deployments on machines with multiple high-capacity disks > > > (e.g., EPYC servers with 4×15TB drives)—this limitation makes segment > > > sizes inefficiently small. And too many handles (log, index, metadata). > > > So Increasing the limit to 3–4GB (or more) would better align with > > today’s > > > storage capabilities. > > > > > > Would love to hear your thoughts ! > > > > > > Best, > > > > > > Mikhail Fesenko > > > > > >
