Hi Yang,
Thanks for your reply.
Now what should I do to improve my performance?Because the old kafka code
was good in performance
These are the properties:
props.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "");
props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,
"false");
props.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,
"earliest");
props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, "1024");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,"500");
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, "500");
How to decrease this time ,plz suggest .
Thanks,
Giridar
On Sun, Nov 24, 2024 at 3:15 PM PoAn Yang <yangp...@gmail.com> wrote:
> Hi Giridar,
>
> > *Code explanation:Fetching records is taking time for the first poll.*
> > Poll Records Count: 500 diff: 1284
> > Poll Records Count: 500 diff: 3
> >
> > For the first 500 records it took 1284 ms and next 500 records it took 4
> ms
> >
> > *Why this much difference? I would like to improve the performance of the
> > first poll time?*
>
> IIRC, a consumer has FetchBuffer (cache).
> If a FetchResponse can return records morn than `max.poll.records`, the
> extra records will be stored in FetchBuffer.
> That is why the second poll can get data in short time.
> IMO, we don’t need to balance each poll time, because that means we don’t
> want local cache.
>
> > For example if i give "0" as input offset and it is taking time as below
> (6
> > seconds) and not getting 500 records also,it is getting only 200 records
> > per poll and taking lot of time...why this is happening and how to avoid
> > this.
> >
> > Poll Records Count :292 Time taken :1227 ms
> > Poll Records Count :292 Time taken :1181 ms
> > Poll Records Count :296 Time taken:1234 ms
> > Poll Records Count :292 Time taken:1133 ms
> >
> > *If I give an offset as 110999 and it is getting some fast and records
> > getting as 500 each..Why this difference please.*
> >
> > Poll Records Count :500 Time taken:1284 ms
> > Poll Records Count :500 Time taken:3 ms
>
> IIUC, a FetchRequest has a limitation from `fetch.max.bytes`.
> If the record size from offset “0” is bigger than from offset “110999”,
> then a FetchResponse returns less records.
>
> Please correct me if I misunderstood anything.
>
> Thanks,
> PoAn
>
> > On Nov 24, 2024, at 2:09 PM, giri mungi <girimung...@gmail.com> wrote:
> >
> > Hi Team,
> >
> > Good day to you.
> >
> > Iam Giridhar.I need your suggestions in kafka
> > performance improvement please.
> >
> > *Scenario is: The user will give the offset as input and based on the
> > offset we need to give the next 1000 messages from kafka topic and next
> > offset.The kafka topic contains only one partition.*
> >
> > We are trying to migrate from old kafka to new kafka.In the old kafka we
> > were using code like:
> >
> > *old code(kafka clients 0.8 .1):*
> >
> > FetchRequest req = new
> > FetchRequestBuilder().clientId(clientName).addFetch(a_topic,
> > a_partition, readOffset, fetchSize).build();
> > FetchResponse fetchResponse = consumer.fetch(req);
> > ByteBufferMessageSet set = fetchResponse.messageSet(a_topic,
> a_partition);
> >
> > This code is super fast and same we are trying to achieve using
> > KafkaConsumer API and getting slowness
> >
> > *New kafkaconsumer code is using(kafka clients 3.6 .1)*
> >
> > TopicPartition topicPartition = new TopicPartition("Test", 0);
> > consumer.seekToBeginning(Collections.singletonList(topicPartition));
> > long kafkaEarliestOffset = consumer.position(topicPartition);
> > try (KafkaConsumer < String, String > consumer =
> > KafkaConsumerFactory.createConsumer(clientName, fetchSize)) {
> > consumer.assign(Collections.singletonList(topicPartition));
> > consumer.seek(topicPartition, readOffset);
> > do {
> > ConsumerRecords < String, String > records =
> > consumer.poll(Duration.ofMillis(1500));
> > } while (!end)
> >
> > public static KafkaConsumer<String,String> createConsumer(String
> > clientName,int fetchSize) {
> > Properties props = new Properties();
> > String kafkaBrokerStr =
> > Config.getConsumerPropValue("kafkabrokerlist");
> >
> props.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
> > StringDeserializer.class.getName());
> >
> > props.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
> > StringDeserializer.class.getName());
> > props.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "");
> > props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,
> > "false");
> > props.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,
> > "earliest");
> > props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, "1024");
> > props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,"500");
> > props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, "500");
> > return new KafkaConsumer<String,String>(props);
> > }
> >
> > *Code explanation:Fetching records is taking time for the first poll.*
> > Poll Records Count: 500 diff: 1284
> > Poll Records Count: 500 diff: 3
> >
> > For the first 500 records it took 1284 ms and next 500 records it took 4
> ms
> >
> > *Why this much difference? I would like to improve the performance of the
> > first poll time?*
> >
> >
> > 1) How to fetch first 500 records in less time
> >
> > *I am also seeing one strange issue.My kafka topic which has one
> partition
> > contains some 5 lakh records*.*The starting records take more time to
> fetch
> > from kafka.*
> >
> > For example if i give "0" as input offset and it is taking time as below
> (6
> > seconds) and not getting 500 records also,it is getting only 200 records
> > per poll and taking lot of time...why this is happening and how to avoid
> > this.
> >
> > Poll Records Count :292 Time taken :1227 ms
> > Poll Records Count :292 Time taken :1181 ms
> > Poll Records Count :296 Time taken:1234 ms
> > Poll Records Count :292 Time taken:1133 ms
> >
> > *If I give an offset as 110999 and it is getting some fast and records
> > getting as 500 each..Why this difference please.*
> >
> > Poll Records Count :500 Time taken:1284 ms
> > Poll Records Count :500 Time taken:3 ms
> >
> >
> >
> > Please give your suggestion on this.
> >
> > Regards,
> > Giridar
>
>
