Hello, I'm new to Kafka,I'm learning by tweaking with the different parameters. I am using a Python Producer application for uploading a 3.4mb Csv file into kafka topic with around 6 partitions. Everything is working fine, but since there are so many records amounting to around 35000. There's a log message that i receive and than application terminates stating that there's still some message left in the queue. I have tried all the parameters that i could, like increasing the buffer size, linger.ms<http://linger.ms/> but to no avail I wasnt able to solve this issue. I am already using flush as its mentioned in the log message. The log message that i received was as follow: %4|1688637353.562|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 34866 messages (3035809 bytes) still in queue or transit: use flush() to wait for outstanding message delivery I hope you could help me to what sort of parameters value must be updated to ingest the records into kafka cluster and flush the records from queue to the kafka topic. Waiting for solution from your end. Thanks for reading it.
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows
