FrankChen021 commented on code in PR #19191:
URL: https://github.com/apache/druid/pull/19191#discussion_r3141492785
##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/supervisor/KafkaSupervisor.java:
##########
@@ -265,6 +267,132 @@ protected
List<SeekableStreamIndexTask<KafkaTopicPartition, Long, KafkaRecordEnt
return taskList;
}
+ @Override
+ public void submitBackfillTask(
+ Map<KafkaTopicPartition, Long> startOffsets,
+ Map<KafkaTopicPartition, Long> endOffsets,
+ @Nullable Integer backfillTaskCount
+ )
+ {
+ if (startOffsets == null || startOffsets.isEmpty() || endOffsets == null
|| endOffsets.isEmpty()) {
+ log.info("No offsets to backfill, skipping backfill task submission");
+ return;
+ }
+
+ try {
+ String backfillSupervisorId =
spec.getSpec().getDataSchema().getDataSource() + "_backfill";
+
+ // If backfillTaskCount is not provided, default to taskCount / 2
+ int taskCount = spec.getSpec().getIOConfig().getTaskCount();
+ int numBackfillTasks = backfillTaskCount != null ? backfillTaskCount :
Math.max(1, taskCount / 2);
Review Comment:
[P1] Reject non-positive backfill task counts before resetting offsets
backfillTaskCount comes directly from the new query parameter and can be 0
or negative. In that case numTasks becomes non-positive, the code later divides
by numTasks, catches the exception, and silently skips backfill submission;
because resetSupervisorAndBackfill has already reset the supervisor metadata to
latest, this can acknowledge a reset while leaving the skipped range
un-backfilled. Validate backfillTaskCount > 0 before performing the reset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
