GitHub user benstopford opened a pull request:
https://github.com/apache/kafka/pull/1776
KIP-73 - Replication Quotas
This applies to Replication Quotas
based on KIP-73
[()](https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas)
originally motivated by KAFKA-1464.
**This first PR demonstrates the approach**. Main stuff left to do:
- The quota value should be moved to a dynamic, broker-level config. Here
we are making do with a client-level override.
- The fetch size is currently unbounded. This will be addressed in KIP-74,
but we need to ensure this ensures requests donât go beyond the throttle
window.
- There is no specific validation or other utilities in ConfigCommand or
code in ReassignPartitionsCommand for validation etc.
- Tests are largely happy path, the functional tests rely on a single
broker-to-broker set up. There are no system tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/benstopford/kafka rep-quotas-v2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/1776.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1776
----
commit d4964b8905cef3d1e11b20a3ed8964643e214ad3
Author: Ben Stopford <[email protected]>
Date: 2016-08-02T22:44:06Z
CPKAFKA-291: Port of original POC, which used seperate replica fetcher
threads.
commit a3d1a5590df491468a42a481faaa89c7951114d4
Author: Ben Stopford <[email protected]>
Date: 2016-08-10T15:13:34Z
CPKAFKA-296: MINOR: Final checkin .. includes failing test demonstrating
the current linearisibility issue
commit 300ecc43db4bda3adc5ee47e505324a4b16b15df
Author: Ben Stopford <[email protected]>
Date: 2016-08-11T15:59:35Z
CPKAFKA-305: POC of leader, bytes based throttling. This commit includes
the rollback of the previous work.
commit 3fc0444e9d77c65449c08a6c7ac482da09ebc431
Author: Ben Stopford <[email protected]>
Date: 2016-08-11T16:23:08Z
CPKAFKA-305: MINOR: rename and tidy only
commit 0c542440d41d1924a4c8d16f692cd814bcb1a460
Author: Ben Stopford <[email protected]>
Date: 2016-08-11T16:23:33Z
CPKAFKA-305: MINOR: rename and tidy only
commit 627905b4f4083dcc7ec46b1e4d25d50952f294ad
Author: Ben Stopford <[email protected]>
Date: 2016-08-11T18:25:00Z
CPKAFKA-305: Completed POC of follower throttling. Quick and Dirty.
commit 5f4c6aeb30f538b23e8aa53d5783386491648319
Author: Ben Stopford <[email protected]>
Date: 2016-08-12T10:25:18Z
CPKAFKA-305: Added test
shouldReplicateThrottledAndNonThrottledPartitionsConcurrently
commit 239d6901d45518b448c90ccf9b59f5d9d3481361
Author: Ben Stopford <[email protected]>
Date: 2016-08-15T16:39:03Z
CPKAFKA-305: Stabalised tests by replacing Rate with SimpleRate. This is
likely a temporary measure, but that depends on whether we want a graduated
start as defined in the current rate implementation. Altered purgatory check.
Currently not using code for proposing rate changes, instead we validate simple
whether the quota is currently exceeded (boolean)
commit 86425e44d90194ef96f3d666035b9b0977f7e510
Author: Ben Stopford <[email protected]>
Date: 2016-08-15T18:52:16Z
CPKAFKA-305: Added small test around replication quota manager
commit e26c8621ae6c550b901c2f9533960675f03cde22
Author: Ben Stopford <[email protected]>
Date: 2016-08-16T07:10:14Z
CPKAFKA-305: Altered leader algorithm to evaluate bytes per partition
commit 434b1cc3e0326f8dbc6c153b7435b1f83e661747
Author: Ben Stopford <[email protected]>
Date: 2016-08-16T15:17:57Z
CPKAFKA-305: SReplicaManagerQuotasTest - added support for multiple
(subsequent) partitions
commit a6cc2400defaec16308fd3ccfc05410069d42c0b
Author: Ben Stopford <[email protected]>
Date: 2016-08-16T15:36:59Z
CPKAFKA-305: Moved things around in ClientQuotaManager, fixed bug in
ReplicationQuotaTest
commit 2d3ab68b57c205775b8a065ed09e35919aa90820
Author: Ben Stopford <[email protected]>
Date: 2016-08-16T16:55:27Z
CPKAFKA-305: Added a second partition to one of the acceptance tests. Fixed
bug in logging in KafkaApis
commit ab3d846a50a6589d4b860c8afb2bda695a2547e7
Author: Ben Stopford <[email protected]>
Date: 2016-08-17T16:05:59Z
CPKAFKA-305: small changes, nothing functional
commit b8a2fcd71a792f7ae06a7aa2993141f14d0554e4
Author: Ben Stopford <[email protected]>
Date: 2016-08-17T20:44:12Z
CPKAFKA-305: Extracted ReplicationQuotaManager into its own class, and
associated changes.
commit 5cd27799b4e09ece2d11c22de9f6e13f09ad5c11
Author: Ben Stopford <[email protected]>
Date: 2016-08-22T12:26:06Z
CPKAFKA-305: Moved kafka-configs format to use 0-1:1-2 format (i.e. with
colon as seperator) plus extended test
commit 3e525c141c4702615dbac6177aaa3d0704ce2394
Author: Ben Stopford <[email protected]>
Date: 2016-08-22T13:35:03Z
CPKAFKA-305: Ensure the correct config is propagated
commit efb815fde2db6f1423d70ebb701636688625aee7
Author: Ben Stopford <[email protected]>
Date: 2016-08-22T19:17:07Z
CPKAFKA-305: Changed rate to be based on fixed windows which makes it
easier to test. Added some more tests for the ReplicationQuotaManager
commit 726bd34147a7d5caa30b017f742ddc5ff973618d
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T09:51:59Z
CPKAFKA-305: Comments only
commit 62e2b1374b384dde71de9cca7a932572d264f898
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T09:56:07Z
CPKAFKA-305: rolled back minor changes to ClientQuotaManager
commit 0934fdd9d811fe97c941e2865b4bb23304388ccf
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T15:09:25Z
CPKAFKA-305: Fixed bug in ReplicationQuotaManagerTest. Reverted back to
SimpleRate as it gives a better match on the rate in tests (although makes less
difference long term)
commit 5e5948f58667cdbd6f195c200e612797ab574410
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T16:25:13Z
CPKAFKA-305: roll back gradle commit (accidental)
commit cfd5a566340c97f63523ae784d9aa85b9457ac52
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T16:25:57Z
CPKAFKA-305: roll back gradle commit (accidental)
commit 7fb65b942c66542e9ef3110cd8383d53310a2e8e
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T17:03:05Z
CPKAFKA-305: Replaced the FetchQuota class with a Null Object pattern.
commit 9398889d151a84e451224dea20bb448083a29928
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T17:24:24Z
CPKAFKA-305: logging only
commit 66f3f39a438af3589dad5ec04d27ad996b79a1c5
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T18:30:21Z
CPKAFKA-305: Tidy
commit 40c5a5f446d5bd8045b653c362e3ad3d83fef8b7
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T18:33:02Z
CPKAFKA-305: Rename only
commit c91bc40602dc6a31340d20dd558a6dd0191e19bc
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T18:48:26Z
CPKAFKA-305: Rename only
commit bc3b460fe4d1f1dd9ca65b8bba424205d2494456
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T19:19:13Z
CPKAFKA-305: small changes
commit 834060b6f64cf97c2e4dd54221c9e28ef10f370a
Author: Ben Stopford <[email protected]>
Date: 2016-08-23T19:22:31Z
CPKAFKA-305: licenses
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
