lianetm commented on code in PR #14557:
URL: https://github.com/apache/kafka/pull/14557#discussion_r1427015080
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java:
##########
@@ -190,63 +182,91 @@ private static long findMinTime(final Collection<?
extends RequestState> request
* has elapsed.
*
* @param offsets Offsets to commit
+ * @param expirationTimeMs Time until which the request will continue to
be retried if it fails
+ * with a retriable error. If not present, the
request will be sent
+ * but not retried.
* @return Future that will complete when a response is received for the
request, or a
* completed future if no request is generated.
*/
- public CompletableFuture<Void> maybeAutoCommit(final Map<TopicPartition,
OffsetAndMetadata> offsets) {
- if (!canAutoCommit()) {
+ private CompletableFuture<Void> maybeAutoCommit(final Map<TopicPartition,
OffsetAndMetadata> offsets,
+ final Optional<Long>
expirationTimeMs,
+ boolean checkInterval) {
+ if (!autoCommitEnabled()) {
+ log.debug("Skipping auto-commit because auto-commit config is not
enabled.");
return CompletableFuture.completedFuture(null);
}
AutoCommitState autocommit = autoCommitState.get();
- if (!autocommit.shouldAutoCommit()) {
+ if (checkInterval && !autocommit.shouldAutoCommit()) {
return CompletableFuture.completedFuture(null);
}
- CompletableFuture<Void> result = sendAutoCommit(offsets);
+ CompletableFuture<Void> result = addOffsetCommitRequest(offsets,
expirationTimeMs)
+ .whenComplete(autoCommitCallback(offsets));
autocommit.resetTimer();
autocommit.setInflightCommitStatus(true);
return result;
}
/**
* If auto-commit is enabled, this will generate a commit offsets request
for all assigned
- * partitions and their current positions.
+ * partitions and their current positions. Note on auto-commit timers:
this will reset the
+ * auto-commit timer to the interval before issuing the async commit, and
when the async commit
+ * completes, it will reset the auto-commit timer with the exponential
backoff if the request
+ * failed with a retriable error.
*
* @return Future that will complete when a response is received for the
request, or a
* completed future if no request is generated.
*/
- public CompletableFuture<Void> maybeAutoCommitAllConsumed() {
- return maybeAutoCommit(subscriptions.allConsumed());
- }
+ public CompletableFuture<Void> maybeAutoCommitAllConsumedAsync() {
+ if (!autoCommitEnabled()) {
+ // Early return to ensure that no action/logging is performed.
+ return CompletableFuture.completedFuture(null);
+ }
+ Map<TopicPartition, OffsetAndMetadata> offsets =
subscriptions.allConsumed();
+ CompletableFuture<Void> result = maybeAutoCommit(offsets,
Optional.empty(), true);
+ result.whenComplete((__, error) -> {
+ if (error != null) {
+ if (error instanceof RetriableCommitFailedException) {
+ log.debug("Asynchronous auto-commit of offsets {} failed
due to retriable error.", offsets, error);
+ resetAutoCommitTimer(retryBackoffMs);
+ } else {
Review Comment:
Done,
[this](https://github.com/apache/kafka/pull/14557/commits/18b6c6b4ab8e7c33d8a5606e9fd054507852830e)
is the commit with the fix to make sure we generate the exception for the
async commits that fail with retriable (so that they end up being handled in
the code above). I added 2 tests to illustrate the difference in behaviour here
between sync and async (sync will just timeout after retrying for the time
allowed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
