I do see the default for message.max.bytes is set to 1MB though. That would be for each record or each poll?
On Fri, Mar 20, 2020 at 3:36 PM Ryan Schachte <coderyanschac...@gmail.com> wrote: > Hi Liam, > We are running 2.3.1. I was hoping I wouldn't need to modify anything at > the broker level since I do not have control/access to the broker config, > just the consumer configuration. Am I out of luck in that case? > > > > On Fri, Mar 20, 2020 at 3:27 PM Liam Clarke <liam.cla...@adscale.co.nz> > wrote: > >> Hi Ryan, >> >> Firstly, what version Kafka? >> >> Secondly check the broker's message.max.bytes and the topic's >> max.message.bytes, I suspect they're set a lot lower (or not at all) and >> will override your fetch.min.bytes. >> >> Cheers, >> >> Liam Clarke >> >> On Sat, 21 Mar. 2020, 11:09 am Ryan Schachte, <coderyanschac...@gmail.com >> > >> wrote: >> >> > Hey guys. >> > I'm trying to maximize the amount of data I'm batching from Kafka. The >> > output is me writing the data to a file on server. I'm adding extremely >> > high values to my consumer configuration and I'm still getting multiple >> > files written with very small file sizes. >> > >> > As seen below, I wait a long time to retrieve my min bytes. After ~20 >> > seconds the poll completes with N records and writes a pretty small >> file. >> > I'm interpreting that as the wait time not being respected nor is the >> min >> > bytes. Why would this be the case? >> > Code: >> > >> > props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, >> args.enableAutoCommit); >> > props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, args.minFetchBytes); >> > props.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, args.maxFetchBytes); >> > props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, >> > args.maxPartitionFetchBytes); >> > props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, args.maxPollRecords); >> > props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, args.maxFetchWait); >> > >> > Consumer configuration: >> > >> > --max_fetch_bytes 2147483000--min_fetch_bytes >> > 2147483000--max_poll_records 2147483000--max_partition_fetch_bytes >> > 2147483000--enable_auto_commit false--fetch_max_wait 900000 >> > >> >
