shounakmk219 commented on code in PR #13584:
URL: https://github.com/apache/pinot/pull/13584#discussion_r1682278381
##########
pinot-controller/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java:
##########
@@ -108,8 +110,20 @@ protected void processTable(String tableNameWithType,
Context context) {
if (context._runSegmentLevelValidation) {
runSegmentLevelValidation(tableConfig, streamConfig);
}
+
+ // IS is updated in below cases
+ // Case 1 -> Table is exceeding storage quota but IS still has
"isQuotaExceeded" as false
+ // This will help prevent consuming segment creation through this task
upon manual zk changes to
+ // "isQuotaExceeded" in IS.
+ // Case 2 -> Table is within quota limits now but IS still has
"isQuotaExceeded" as true
+ // This will help resume the table consumption once the quota is available
due to either quota increase or
+ // segment deletion.
+ // In which case we need to pass "recreateDeletedConsumingSegment" as true
to "ensureAllPartitionsConsuming" below.
+ boolean idealStateUpdated =
_llcRealtimeSegmentManager.updateStorageQuotaExceededInIdealState(tableNameWithType,
+ _storageQuotaChecker.isTableStorageQuotaExceeded(tableConfig));
+
_llcRealtimeSegmentManager.ensureAllPartitionsConsuming(tableConfig,
streamConfig,
- context._recreateDeletedConsumingSegment, context._offsetCriteria);
+ context._recreateDeletedConsumingSegment || idealStateUpdated,
context._offsetCriteria);
Review Comment:
That's right. This will effectively recreate the consuming segment only in
case 2.
Even though `idealStateUpdated` is set in case 1 as well, eventually it will
skip the consuming segment creation as in case 1 the `IS_QUOTA_EXCEEDED` flag
is set which will effectively skip creating consuming segment.
But then I need to ensure the `updateStorageQuotaExceededInIdealState`
should return false incase the IS update fails as in that case
`IS_QUOTA_EXCEEDED` flag is not set which will end up recreating the consuming
segment. Will add that check as well.
Thanks for pointing it out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
