Also here you can see guys have same idea
https://forum.confluent.io/t/what-happens-if-i-increase-log-segment-bytes/5845
On Tue, 22 Apr 2025 at 15:41, Mikhail Fesenko wrote:
> Currently we have big amount of topics and big data, and using high disk
> storage, so rotate files too fast in case of
This is interesting, could be a huge barrier for this change, thanks for
your clarification
On Fri, 21 Feb 2025 at 18:52, Jun Rao wrote:
> Hi, Mikhail,
>
> Currently, the index file uses 4 bytes to represent the file position of a
> record batch within a segment file. This is based on the assump
Currently we have big amount of topics and big data, and using high disk
storage, so rotate files too fast in case of high traffic means less sense,
kafka has 50-100k open files index/offset/log so decrease amount of
files/improve cache for them, it can give many improvements for big
kafka topics
Hi, Mikhail,
Currently, the index file uses 4 bytes to represent the file position of a
record batch within a segment file. This is based on the assumption that
each segment is capped at 2GB. RemoteLogSegmentMetadata is a public
interface and currently uses int to represent segmentSizeInBytes. If
hi Mikhail
> So Increasing the limit to 3–4GB (or more) would better align with
today’s storage capabilities.
Out of curiosity, what is the benefit of using the larger segment?
While changing the type from int to long is a minor change, allowing or
encouraging users to create large segments requi
Guys, any thoughts on that? Please let me know!
On Thu, 13 Feb 2025 at 22:31, Mikhail Fesenko wrote:
> Hi everyone,
>
> I’d like to propose a change the data type of log.segment.bytes from int
> to long, allowing segment sizes beyond the current 2GB limit.
>
> Rationale:
>
> Currently, the maxim
Hi everyone,
I’d like to propose a change the data type of log.segment.bytes from int to
long, allowing segment sizes beyond the current 2GB limit.
Rationale:
Currently, the maximum segment size is constrained by the int type, capping
it at 2GB (max int). However, with modern hardware—such as la
