Thanks for response , understood,
1.
Exception classes isn't available for any fetching related i believe in javadocs
TOPIC -> testing, partition count =1, offset count = 2
2.
Assuming the message 1st topics 1st message we are trying to read
"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure
dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit
anim id est laborum."
Under network latency of 1 kbps bandwidth, 1mb offset message with value of
fetch.min.bytes = 1 bytes along with fetch.max.wait.ms = 0.5 seconds , it will fetch 1
kilobytes of data but that data if is in avro serialization it will be in binary right,
so the fetched data in binary message is upto "Lorem ipsum dolor sit amet,
consectetur adipiscing elit, sed do " for example
Then how data is assembled together for remaining potion ? how data is made as
whole ?
________________________________
From: Matthias J. Sax <mj...@apache.org>
Sent: Wednesday, October 16, 2024 06:23
To: users@kafka.apache.org <users@kafka.apache.org>
Subject: [EXT MSG] Re: KAFKA POLL
EXTERNAL source. Be CAREFUL with links / attachments
1. As pointed out in the JavaDocs
@param timeoutMs The time, in milliseconds, spent waiting in poll if data is
not available in the buffer.
2. If there is no data with the timeout, `poll()` return an "empty"
`ConsumerRecords` object.
For error/exception I cannot say, as it depends on the
error/exception... The exception which could be thrown are documented in
the JavaDocs, so maybe you can start there?
consumerRecords<String, String> records = consumer.poll(10000L);
* whether this records count will be 0 in this case ?
Yes, the `ConsumerRecords` object will be empty.
if poll gets 500kb from topic in first poll within 10s after processing
assuming no errors comes up and records count is 0 then in next poll whether
next 500kb is fetched or it will attempt to again fetch full message ?
If the message is 1MB in size, it will either be fetched completely not
not at all. There will only be one fetch request and one fetch response.
If the first poll() send the fetch request, the second poll() would see
that a fetch request is already "in progress" and just wait for it to
complete (or again just let `poll()` return with empty result if the
timeout hits before the fetch request completes)
HTH.
-Matthias
On 10/10/24 5:27 AM, Sathishkumar Palanisamy wrote:
Hi Team,
Does anyone know in the experimental scenario,
JAVA runtime 8 or 11
1. Kafka's topic -> Testing_topic
2. Kafka's topic partition count -> 1
3. Kafka's topic partition offset -> 2 [two messages are available, each
weight about 1 mb ]
4. Bandwidth of 1 kb and Kafka's topic offset's message of 1 mb is
available each
5. In consumer scenario
* While polling the consumer using consumer.poll(10000L) deprecated
method supporting the long argument
* With properties (exactly used properties)
i.
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
bootstrap.server.config.url);
ii.
props.put(ConsumerConfig.GROUP_ID_CONFIG, group-1);
iii.
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
iv.
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
v.
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 1000);
vi.
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
vii.
props.put("request.timeout.ms", "60000");
Question
1. in poll(10000L ) given , whether this is in seconds or milliseconds or
minutes (Deprecated method) ?
2. so in bandwidth of 1 kb for experimental purpose -> while polling
assuming the 10000L is milliseconds so if I kafka consumer doesn't get the offset
message which of 1 mb
* what will happens in brief details ?
* whether any errors/exceptions comes up ?
* consumerRecords<String, String> records = consumer.poll(10000L);
* whether this records count will be 0 in this case ?
* if poll gets 500kb from topic in first poll within 10s after
processing assuming no errors comes up and records count is 0 then in next poll
whether next 500kb is fetched or it will attempt to again fetch full message ?
The information in this e-mail and any attachments is confidential and may be
legally privileged. It is intended solely for the addressee or addressees. Any
use or disclosure of the contents of this e-mail/attachments by a not intended
recipient is unauthorized and may be unlawful. If you have received this e-mail
in error please notify the sender. Please note that any views or opinions
presented in this e-mail are solely those of the author and do not necessarily
represent those of TEMENOS. We recommend that you check this e-mail and any
attachments against viruses. TEMENOS accepts no liability for any damage caused
by any malicious code or virus transmitted by this e-mail.
