I suppose it is way log management works in kafka. I am not sure the exact reason for this. Also the index files that are constructed have a mapping of relative offset to the base offset of log file to the real offset. The key value in index file is of the form <Int,Int>.
Thanks, Mayuresh On Wed, May 13, 2015 at 5:57 PM, Lance Laursen <llaur...@rubiconproject.com> wrote: > Hey folks, > > Any update on this? > > On Thu, Apr 30, 2015 at 5:34 PM, Lance Laursen < > llaur...@rubiconproject.com> > wrote: > > > Hey all, > > > > I am attempting to create a topic which uses 8GB log segment sizes, like > > so: > > ./kafka-topics.sh --zookeeper localhost:2181 --create --topic > perftest6p2r > > --partitions 6 --replication-factor 2 --config max.message.bytes=655360 > > --config segment.bytes=8589934592 > > > > And am getting the following error: > > Error while executing topic command For input string: "8589934592" > > java.lang.NumberFormatException: For input string: "8589934592" > > at > > > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > > at java.lang.Integer.parseInt(Integer.java:583) > > ... > > ... > > > > Upon further testing with --alter topic, it would appear that > > segment.bytes will not accept a value higher than 2,147,483,647, which is > > the upper limit for a signed 32bit int. This then restricts log segment > > size to an upper limit of ~2GB. > > > > We run Kafka on hard drive dense machines, each with 10gbit uplinks. We > > can set ulimits higher in order to deal with all the open file handles > > (since Kafka keeps all log segment file handles open), but it would be > > preferable to minimize this number, as well as minimize the amount of log > > segment rollover experienced at high traffic (ie: a rollover every 1-2 > > seconds or so when saturating 10gbe). > > > > Is there a reason (performance or otherwise) that a 32 bit integer is > used > > rather than something larger? > > > > Thanks, > > -Lance > > > > > > > -- > > [image: elogo.png] > > Leading the Automation of Advertising > > LANCE LAURSEN | Systems Architect > > ••• (M) 310.903.0546 > > 12181 BLUFF CREEK DRIVE, 4TH FLOOR, PLAYA VISTA, CA 90094 > > RUBICONPROJECT.COM <http://www.rubiconproject.com/> | @RUBICONPROJECT > <http://twitter.com/rubiconproject> > -- -Regards, Mayuresh R. Gharat (862) 250-7125
