Thanks for response , understood, 1. Exception classes isn't available for any fetching related i believe in javadocs TOPIC -> testing, partition count =1, offset count = 2 2. Assuming the message 1st topics 1st message we are trying to read "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."
Under network latency of 1 kbps bandwidth, 1mb offset message with value of fetch.min.bytes = 1 bytes along with fetch.max.wait.ms = 0.5 seconds , it will fetch 1 kilobytes of data but that data if is in avro serialization it will be in binary right, so the fetched data in binary message is upto "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do " for example Then how data is assembled together for remaining potion ? how data is made as whole ? ________________________________ From: Matthias J. Sax <mj...@apache.org> Sent: Wednesday, October 16, 2024 06:23 To: users@kafka.apache.org <users@kafka.apache.org> Subject: [EXT MSG] Re: KAFKA POLL EXTERNAL source. Be CAREFUL with links / attachments 1. As pointed out in the JavaDocs > @param timeoutMs The time, in milliseconds, spent waiting in poll if data is > not available in the buffer. 2. If there is no data with the timeout, `poll()` return an "empty" `ConsumerRecords` object. For error/exception I cannot say, as it depends on the error/exception... The exception which could be thrown are documented in the JavaDocs, so maybe you can start there? > consumerRecords<String, String> records = consumer.poll(10000L); > * whether this records count will be 0 in this case ? Yes, the `ConsumerRecords` object will be empty. > if poll gets 500kb from topic in first poll within 10s after processing > assuming no errors comes up and records count is 0 then in next poll whether > next 500kb is fetched or it will attempt to again fetch full message ? If the message is 1MB in size, it will either be fetched completely not not at all. There will only be one fetch request and one fetch response. If the first poll() send the fetch request, the second poll() would see that a fetch request is already "in progress" and just wait for it to complete (or again just let `poll()` return with empty result if the timeout hits before the fetch request completes) HTH. -Matthias On 10/10/24 5:27 AM, Sathishkumar Palanisamy wrote: > Hi Team, > Does anyone know in the experimental scenario, > JAVA runtime 8 or 11 > > 1. Kafka's topic -> Testing_topic > 2. Kafka's topic partition count -> 1 > 3. Kafka's topic partition offset -> 2 [two messages are available, each > weight about 1 mb ] > 4. Bandwidth of 1 kb and Kafka's topic offset's message of 1 mb is > available each > 5. In consumer scenario > * While polling the consumer using consumer.poll(10000L) deprecated > method supporting the long argument > * With properties (exactly used properties) > > > i. props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, > bootstrap.server.config.url); > > ii. > props.put(ConsumerConfig.GROUP_ID_CONFIG, group-1); > > iii. > props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, > "org.apache.kafka.common.serialization.StringDeserializer"); > > iv. > props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, > "org.apache.kafka.common.serialization.StringDeserializer"); > > v. > props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 1000); > > vi. > props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); > > vii. > props.put("request.timeout.ms", "60000"); > Question > > 1. in poll(10000L ) given , whether this is in seconds or milliseconds or > minutes (Deprecated method) ? > 2. so in bandwidth of 1 kb for experimental purpose -> while polling > assuming the 10000L is milliseconds so if I kafka consumer doesn't get the > offset message which of 1 mb > * what will happens in brief details ? > * whether any errors/exceptions comes up ? > * consumerRecords<String, String> records = consumer.poll(10000L); > * whether this records count will be 0 in this case ? > * if poll gets 500kb from topic in first poll within 10s after > processing assuming no errors comes up and records count is 0 then in next > poll whether next 500kb is fetched or it will attempt to again fetch full > message ? > > The information in this e-mail and any attachments is confidential and may be legally privileged. It is intended solely for the addressee or addressees. Any use or disclosure of the contents of this e-mail/attachments by a not intended recipient is unauthorized and may be unlawful. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of TEMENOS. We recommend that you check this e-mail and any attachments against viruses. TEMENOS accepts no liability for any damage caused by any malicious code or virus transmitted by this e-mail.