Hi Ryan, Firstly, what version Kafka?
Secondly check the broker's message.max.bytes and the topic's max.message.bytes, I suspect they're set a lot lower (or not at all) and will override your fetch.min.bytes. Cheers, Liam Clarke On Sat, 21 Mar. 2020, 11:09 am Ryan Schachte, <coderyanschac...@gmail.com> wrote: > Hey guys. > I'm trying to maximize the amount of data I'm batching from Kafka. The > output is me writing the data to a file on server. I'm adding extremely > high values to my consumer configuration and I'm still getting multiple > files written with very small file sizes. > > As seen below, I wait a long time to retrieve my min bytes. After ~20 > seconds the poll completes with N records and writes a pretty small file. > I'm interpreting that as the wait time not being respected nor is the min > bytes. Why would this be the case? > Code: > > props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, args.enableAutoCommit); > props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, args.minFetchBytes); > props.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, args.maxFetchBytes); > props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, > args.maxPartitionFetchBytes); > props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, args.maxPollRecords); > props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, args.maxFetchWait); > > Consumer configuration: > > --max_fetch_bytes 2147483000--min_fetch_bytes > 2147483000--max_poll_records 2147483000--max_partition_fetch_bytes > 2147483000--enable_auto_commit false--fetch_max_wait 900000 >
