Hi Liam, We are running 2.3.1. I was hoping I wouldn't need to modify anything at the broker level since I do not have control/access to the broker config, just the consumer configuration. Am I out of luck in that case?
On Fri, Mar 20, 2020 at 3:27 PM Liam Clarke <liam.cla...@adscale.co.nz> wrote: > Hi Ryan, > > Firstly, what version Kafka? > > Secondly check the broker's message.max.bytes and the topic's > max.message.bytes, I suspect they're set a lot lower (or not at all) and > will override your fetch.min.bytes. > > Cheers, > > Liam Clarke > > On Sat, 21 Mar. 2020, 11:09 am Ryan Schachte, <coderyanschac...@gmail.com> > wrote: > > > Hey guys. > > I'm trying to maximize the amount of data I'm batching from Kafka. The > > output is me writing the data to a file on server. I'm adding extremely > > high values to my consumer configuration and I'm still getting multiple > > files written with very small file sizes. > > > > As seen below, I wait a long time to retrieve my min bytes. After ~20 > > seconds the poll completes with N records and writes a pretty small file. > > I'm interpreting that as the wait time not being respected nor is the min > > bytes. Why would this be the case? > > Code: > > > > props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, > args.enableAutoCommit); > > props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, args.minFetchBytes); > > props.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, args.maxFetchBytes); > > props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, > > args.maxPartitionFetchBytes); > > props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, args.maxPollRecords); > > props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, args.maxFetchWait); > > > > Consumer configuration: > > > > --max_fetch_bytes 2147483000--min_fetch_bytes > > 2147483000--max_poll_records 2147483000--max_partition_fetch_bytes > > 2147483000--enable_auto_commit false--fetch_max_wait 900000 > > >
