morningman commented on code in PR #69749:
URL: https://github.com/apache/doris/pull/69749#discussion_r4414151472
##########
fe/fe-core/src/main/java/org/apache/doris/load/routineload/kafka/KafkaRoutineLoadJob.java:
##########
@@ -452,6 +453,21 @@ protected boolean refreshKafkaPartitions(boolean
needAutoResume) throws UserExce
}
private boolean updateKafkaPartitions() throws UserException {
+ // The metadata request goes to the backends of the declared compute
group, and after RESUME it runs
+ // before any task, so check the group here as
RoutineLoadTaskScheduler does before each task. A
+ // dropped group would otherwise pause the job with "no alive
backends" and the Kafka advice below.
+ try {
+
ComputeGroupBindingUtil.checkComputeGroupBeforeTask(getUserIdentity(),
getDeclaredComputeGroup());
+ } catch (UserException e) {
+ LOG.warn(new LogBuilder(LogKey.ROUTINE_LOAD_JOB, id)
+ .add("error_msg", e.getMessage())
+ .build());
+ if (this.state == JobState.NEED_SCHEDULE) {
+ unprotectUpdateState(JobState.PAUSED, new
ErrorReason(e.getErrorCode(), e.getMessage()),
Review Comment:
Fixed in aaff9b57e30. The pause now goes through
`RoutineLoadJob.pauseIfNeedSchedule()`, which takes the job write lock, checks
NEED_SCHEDULE again under it, and only then calls `unprotectUpdateState(PAUSED,
...)`. A STOP that completes while the refresh is running therefore stays
STOPPED, and no PAUSED entry is journaled after it. The metadata-failure catch
next to it and `KinesisRoutineLoadJob.updateKinesisShards()` already paused in
the same unlocked way on master, so they use the helper too. `update()` still
runs the metadata request itself without the lock, as before.
Three new `RoutineLoadComputeGroupTest` cases cover the revoked-USAGE check,
the Kafka metadata failure and the Kinesis shard listing failure. In each one,
another thread holds the job lock the way a STOP does. The test asserts that
the refresh waits for that lock and that the STOP, completed in the meantime,
is not turned back into PAUSED. With the helper changed back to the unlocked
check, all three fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
