Thank you Matthias for the clarification
-----Original Message-----
From: Matthias J. Sax <mj...@mailbox.org.INVALID>
Sent: Thursday, October 17, 2024 8:46 AM
To: users@kafka.apache.org
Subject: Re: [EXT MSG] Re: KAFKA POLL
Not sure if I understand (1).
For (2), you don't need to worry about it. The consumer will handle all this
for you internally, and only return a full message from `poll()`.
-Matthias
On 10/16/24 00:16, Sathishkumar Palanisamy wrote:
> Thanks for response , understood,
>
> 1.
> Exception classes isn't available for any fetching related i believe
> in javadocs TOPIC -> testing, partition count =1, offset count = 2
> 2.
> Assuming the message 1st topics 1st message we are trying to read
> "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
> tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
> quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
> consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
> cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
> proident, sunt in culpa qui officia deserunt mollit anim id est laborum."
>
> Under network latency of 1 kbps bandwidth, 1mb offset message with
> value of fetch.min.bytes = 1 bytes along with fetch.max.wait.ms = 0.5
> seconds , it will fetch 1 kilobytes of data but that data if is in
> avro serialization it will be in binary right, so the fetched data in
> binary message is upto "Lorem ipsum dolor sit amet, consectetur
> adipiscing elit, sed do " for example
>
> Then how data is assembled together for remaining potion ? how data is made
> as whole ?
>
>
>
>
> ________________________________
> From: Matthias J. Sax <mj...@apache.org>
> Sent: Wednesday, October 16, 2024 06:23
> To: users@kafka.apache.org <users@kafka.apache.org>
> Subject: [EXT MSG] Re: KAFKA POLL
>
> EXTERNAL source. Be CAREFUL with links / attachments
>
> 1. As pointed out in the JavaDocs
>
>> @param timeoutMs The time, in milliseconds, spent waiting in poll if data is
>> not available in the buffer.
>
>
>
> 2. If there is no data with the timeout, `poll()` return an "empty"
> `ConsumerRecords` object.
>
> For error/exception I cannot say, as it depends on the
> error/exception... The exception which could be thrown are documented
> in the JavaDocs, so maybe you can start there?
>
>
>> consumerRecords<String, String> records = consumer.poll(10000L);
>> * whether this records count will be 0 in this case ?
>
> Yes, the `ConsumerRecords` object will be empty.
>
>
>
>> if poll gets 500kb from topic in first poll within 10s after processing
>> assuming no errors comes up and records count is 0 then in next poll whether
>> next 500kb is fetched or it will attempt to again fetch full message ?
>
> If the message is 1MB in size, it will either be fetched completely
> not not at all. There will only be one fetch request and one fetch response.
> If the first poll() send the fetch request, the second poll() would
> see that a fetch request is already "in progress" and just wait for it
> to complete (or again just let `poll()` return with empty result if
> the timeout hits before the fetch request completes)
>
>
> HTH.
>
> -Matthias
>
>
> On 10/10/24 5:27 AM, Sathishkumar Palanisamy wrote:
>> Hi Team,
>> Does anyone know in the experimental scenario,
>> JAVA runtime 8 or 11
>>
>> 1. Kafka's topic -> Testing_topic
>> 2. Kafka's topic partition count -> 1
>> 3. Kafka's topic partition offset -> 2 [two messages are available,
>> each weight about 1 mb ]
>> 4. Bandwidth of 1 kb and Kafka's topic offset's message of 1 mb is
>> available each
>> 5. In consumer scenario
>> * While polling the consumer using consumer.poll(10000L) deprecated
>> method supporting the long argument
>> * With properties (exactly used properties)
>>
>>
>> i. props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
>> bootstrap.server.config.url);
>>
>>
>> ii. props.put(ConsumerConfig.GROUP_ID_CONFIG, group-1);
>>
>>
>> iii. props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
>> "org.apache.kafka.common.serialization.StringDeserializer");
>>
>>
>> iv. props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
>> "org.apache.kafka.common.serialization.StringDeserializer");
>>
>>
>> v. props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 1000);
>>
>>
>> vi. props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
>>
>> vii.
>> props.put("request.timeout.ms", "60000");
>> Question
>>
>> 1. in poll(10000L ) given , whether this is in seconds or milliseconds
>> or minutes (Deprecated method) ?
>> 2. so in bandwidth of 1 kb for experimental purpose -> while polling
>> assuming the 10000L is milliseconds so if I kafka consumer doesn't get the
>> offset message which of 1 mb
>> * what will happens in brief details ?
>> * whether any errors/exceptions comes up ?
>> * consumerRecords<String, String> records = consumer.poll(10000L);
>> * whether this records count will be 0 in this case ?
>> * if poll gets 500kb from topic in first poll within 10s after
>> processing assuming no errors comes up and records count is 0 then in next
>> poll whether next 500kb is fetched or it will attempt to again fetch full
>> message ?
>>
>>
>
>
> The information in this e-mail and any attachments is confidential and may be
> legally privileged. It is intended solely for the addressee or addressees.
> Any use or disclosure of the contents of this e-mail/attachments by a not
> intended recipient is unauthorized and may be unlawful. If you have received
> this e-mail in error please notify the sender. Please note that any views or
> opinions presented in this e-mail are solely those of the author and do not
> necessarily represent those of TEMENOS. We recommend that you check this
> e-mail and any attachments against viruses. TEMENOS accepts no liability for
> any damage caused by any malicious code or virus transmitted by this e-mail.
>
The information in this e-mail and any attachments is confidential and may be
legally privileged. It is intended solely for the addressee or addressees. Any
use or disclosure of the contents of this e-mail/attachments by a not intended
recipient is unauthorized and may be unlawful. If you have received this e-mail
in error please notify the sender. Please note that any views or opinions
presented in this e-mail are solely those of the author and do not necessarily
represent those of TEMENOS. We recommend that you check this e-mail and any
attachments against viruses. TEMENOS accepts no liability for any damage caused
by any malicious code or virus transmitted by this e-mail.
