Boyang Chen created KAFKA-12499:
-----------------------------------
Summary: Adjust transaction timeout according to commit interval
on Streams EOS
Key: KAFKA-12499
URL: https://issues.apache.org/jira/browse/KAFKA-12499
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Boyang Chen
Assignee: Boyang Chen
Fix For: 3.0.0
The transaction timeout is set to 1 minute by default on Producer today, while
the commit interval on the other hand could be set to a very large value, which
makes the stream always hit transaction timeout and drop into rebalance. We
should increase the transaction timeout correspondingly when commit interval is
large.
On the other hand, broker could have a limit on the max transaction timeout to
be set. If we scale up client transaction timeout over the limit, stream will
fail due to INVALID_TRANSACTION_TIMEOUT. To alleviate this problem, user could
define their own customized transaction timeout to avoid hitting the limit, so
we should still respect what user configures in the override.
The new rule for configuring transaction timeout should look like:
1. If transaction timeout is set in streams config, use it
2. if not, transaction_timeout = max(default_transaction_timeout, 10 *
commit_interval)
Additionally if INVALID_TRANSACTION_TIMEOUT was thrown on Streams when calling
initTransaction(), we should wrap the exception to inform user that their
setting for commit interval could potentially be too high and should adjust.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
