GitHub user tcrayford opened a pull request: https://github.com/apache/kafka/pull/3398
allow transactions in producer perf script allow the transactional producer to be enabled in `producer-perf.sh`, with a new flag `--use-transactions` You can merge this pull request into a Git repository by running: $ git pull https://github.com/heroku/kafka allow_transactions_in_producer_perf Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3398.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3398 ---- commit 9170e87b2a154e7478af0ede43cc56b01f75bedd Author: Xavier Léauté <xav...@confluent.io> Date: 2017-05-19T00:02:51Z KAFKA-5192: add WindowStore range scan (KIP-155) Implements range scan for keys in windowed and session stores Modifies caching session and windowed stores to use segmented cache keys. Cache keys are internally prefixed with their segment id to ensure key ordering in the cache matches the ordering in the underlying store for keys spread across multiple segments. This should also result in fewer cache keys getting scanned for queries spanning only some segments. Author: Xavier Léauté <xav...@confluent.io> Reviewers: Damian Guy, Guozhang Wang Closes #3027 from xvrl/windowstore-range-scan (cherry picked from commit e28752357705568219315375c666f8e500db9c12) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit b31df613f2664a8dc699b421b392f0aca9d5f83f Author: Dana Powers <dana.pow...@gmail.com> Date: 2017-05-19T13:03:22Z KAFKA-3878; Support exponential backoff policy via reconnect.backoff.max (KIP-144) Summary: - add `reconnect.backoff.max.ms` common client configuration parameter - if `reconnect.backoff.max.ms` > `reconnect.backoff.ms`, apply an exponential backoff policy - apply +/- 20% random jitter to smooth cluster reconnects Author: Dana Powers <dana.pow...@gmail.com> Reviewers: Ewen Cheslack-Postava <m...@ewencp.org>, Roger Hoover <roger.hoo...@gmail.com>, Ismael Juma <ism...@juma.me.uk> Closes #1523 from dpkp/exp_backoff (cherry picked from commit abe699176babe4f065b67b2b72d20daa0a2e46a1) Signed-off-by: Ismael Juma <ism...@juma.me.uk> commit 64e12d3c641baeb6a4a624b971ba33721d656054 Author: Ewen Cheslack-Postava <m...@ewencp.org> Date: 2017-05-19T18:26:59Z KAFKA-4714; TimestampConverter transformation (KIP-66) Author: Ewen Cheslack-Postava <m...@ewencp.org> Reviewers: Konstantine Karantasis <konstant...@confluent.io>, Jason Gustafson <ja...@confluent.io> Closes #3065 from ewencp/kafka-3209-timestamp-converter (cherry picked from commit 61bab2d875ab5e03d0df4f62217346549a4c64c3) Signed-off-by: Jason Gustafson <ja...@confluent.io> commit 037f63882df854c923bb9de11856318b951ecb32 Author: Matthias J. Sax <matth...@confluent.io> Date: 2017-05-19T23:29:25Z MINOR: improve descriptions of Streams reset tool options Author: Matthias J. Sax <matth...@confluent.io> Reviewers: Bill Bejeck, Guozhang Wang Closes #3107 from mjsax/minor-reset-tool-options (cherry picked from commit 338857e1c2c2bb0c6011fcab477818102b95cb56) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit 55330cc2931cc05cb1172cc207ada60154a6250a Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-20T01:51:37Z KAFKA-5269; Retry on unknown topic/partition error in transactional requests We should retry AddPartitionsToTxnRequest and TxnOffsetCommitRequest when receiving an UNKNOWN_TOPIC_OR_PARTITION error. As described in the JIRA: It turns out that the `UNKNOWN_TOPIC_OR_PARTITION` is returned from the request handler in KafkaAPis for the AddPartitionsToTxn and the TxnOffsetCommitRequest when the broker's metadata doesn't contain one or more partitions in the request. This can happen for instance when the broker is bounced and has not received the cluster metadata yet. We should retry in these cases, as this is the model followed by the consumer when committing offsets, and by the producer with a ProduceRequest. Author: Apurva Mehta <apu...@confluent.io> Reviewers: Guozhang Wang <wangg...@gmail.com>, Jason Gustafson <ja...@confluent.io> Closes #3094 from apurvam/KAFKA-5269-handle-unknown-topic-partition-in-transaction-manager commit a144c5277dd95d8862ea6d9a497e5a087aa491fe Author: Ewen Cheslack-Postava <m...@ewencp.org> Date: 2017-05-22T01:04:32Z HOTFIX: In Connect test with auto topic creation disabled, ensure precreated topic is always used Author: Ewen Cheslack-Postava <m...@ewencp.org> Reviewers: Jason Gustafson <ja...@confluent.io> Closes #3112 from ewencp/hotfix-precreate-topic (cherry picked from commit d190d89dbc3df90e00f5e3c55507f67c5818504e) Signed-off-by: Ewen Cheslack-Postava <m...@ewencp.org> commit c0e789bdcc039a177f99bfecf36191f5be566d63 Author: Magnus Edenhill <mag...@edenhill.se> Date: 2017-05-22T01:23:12Z MINOR: Fix race condition in TestVerifiableProducer sanity test ## Fixes race condition in TestVerifiableProducer sanity test: The test starts a producer, waits for at least 5 acks, and then logs in to the worker to grep for the producer process to figure out what version it is running. The problem was that the producer was set up to produce 1000 messages at a rate of 1000 msgs/s and then exit. This means it will have a typical runtime slightly above 1 second. Logging in to the vagrant instance might take longer than that thus resulting in the process grep to fail, failing the test. This commit doesn't really fix the issue - a proper fix would be to tell the producer to stick around until explicitly killed - but it increases the chances of the test passing, at the expense of a slightly longer runtime. ## Improves error reporting when is_version() fails Author: Magnus Edenhill <mag...@edenhill.se> Reviewers: Apurva Mehta <apu...@confluent.io>, Ewen Cheslack-Postava <e...@confluent.io> Closes #2765 from edenhill/trunk (cherry picked from commit dc10b0ea014974b450f2abcd4a58e09d5b988e9e) Signed-off-by: Ewen Cheslack-Postava <m...@ewencp.org> commit 6d04e412cffabb1976aa935b4fb4767fd255836e Author: Matthias J. Sax <matth...@confluent.io> Date: 2017-05-22T05:16:18Z KAFKA-4923: Modify compatibility test for Exaclty-Once Semantics in Streams - add broker compatibility system tests Author: Matthias J. Sax <matth...@confluent.io> Reviewers: Damian Guy, Eno Thereska, Guozhang Wang Closes #2974 from mjsax/kafka-4923-add-eos-to-streams-add-broker-check-and-system-test (cherry picked from commit 495836a4a27c2219f72a5f3776ba8e5216493b41) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit ef88ee5045ea791d3208c3dcbc8723c4b2092262 Author: Mickael Maison <mickael.mai...@gmail.com> Date: 2017-05-22T10:00:30Z KAFKA-5294: PlainSaslServerFactory should allow a null Map If props is null, use POLICY_NOPLAINTEXT default value: false Author: Mickael Maison <mickael.mai...@gmail.com> Reviewers: Ismael Juma <ism...@juma.me.uk>, Rajini Sivaram <rajinisiva...@googlemail.com> Closes #3105 from mimaison/KAFKA-5294 (cherry picked from commit d3f1407f7bc95b823516e5a1bdaa23c717b6354e) Signed-off-by: Rajini Sivaram <rajinisiva...@googlemail.com> commit e08e76b7166f5690cafdc8d5e52cd81746408a37 Author: Ismael Juma <ism...@juma.me.uk> Date: 2017-05-22T12:37:29Z KAFKA-5289: handleStopReplica should not send a second response `shutdownIdleFetcherThreads()` can throw InterruptedException for example. Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Rajini Sivaram <rajinisiva...@googlemail.com> Closes #3096 from ijuma/kafka-5289-stop-replica-should-not-send-two-responses (cherry picked from commit ceb10c53302bd4824737e03ab86ca19e2aa64c7a) Signed-off-by: Rajini Sivaram <rajinisiva...@googlemail.com> commit 4c29b1544bb058a004c57c3e559c7458810df2b4 Author: Ismael Juma <ism...@juma.me.uk> Date: 2017-05-22T15:08:06Z KAFKA-5303, KAFKA-5305: Improve logging when fetches fail in ReplicaFetcherThread Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Rajini Sivaram <rajinisiva...@googlemail.com> Closes #3115 from ijuma/kafka-5305-missing-log-info-replica-fetcher (cherry picked from commit c2ced5fb51f2a4ec94d158e18836afb7284d26ce) Signed-off-by: Rajini Sivaram <rajinisiva...@googlemail.com> commit 67f691bbd93b06efe58f2ba28f1b85efedf99651 Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Date: 2017-05-22T17:59:38Z KAFKA-5277; Sticky Assignor should not cache previous assignment (KIP-54 follow-up) ... plus some minor cleanup Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Reviewers: Jason Gustafson <ja...@confluent.io> Closes #3092 from vahidhashemian/KAFKA-5277 (cherry picked from commit 65861a712ddf67eb071a00218730926fdeef7084) Signed-off-by: Jason Gustafson <ja...@confluent.io> commit a9c9dbf58f0e9ccf873e100b62f0f811ad5e4659 Author: Jason Gustafson <ja...@confluent.io> Date: 2017-05-22T22:41:26Z KAFKA-5186; Avoid expensive log scan to build producer state when upgrading Author: Jason Gustafson <ja...@confluent.io> Reviewers: Jun Rao <jun...@gmail.com> Closes #3113 from hachikuji/KAFKA-5186 (cherry picked from commit fcdbb71953fc4c92559a9c7adb4cb8ad4a74acd6) Signed-off-by: Jason Gustafson <ja...@confluent.io> commit 9ae4deaa5724220569b0af312500130079b7fdb0 Author: Ewen Cheslack-Postava <m...@ewencp.org> Date: 2017-05-23T00:46:23Z HOTFIX: Replace JDK download and fix missing argument in Vagrant provisioning script Author: Ewen Cheslack-Postava <m...@ewencp.org> Reviewers: Jason Gustafson <ja...@confluent.io>, Ismael Juma <ism...@juma.me.uk> Closes #3121 from ewencp/hotfix-vagrant-provisioning (cherry picked from commit cca9ad42434265e388ec0100e0080250120e2cf9) Signed-off-by: Ewen Cheslack-Postava <m...@ewencp.org> commit 33216846fe070dfb5dd09eab9f53490bae92c357 Author: Jason Gustafson <ja...@confluent.io> Date: 2017-05-23T03:00:07Z MINOR: Broker should disallow downconversion of transactional/idempotent records Author: Jason Gustafson <ja...@confluent.io> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #3118 from hachikuji/disallow-transactional-idempotent-downconversion (cherry picked from commit e3e2f1d22d17a20ccccf67218c600e7e1647e1ca) Signed-off-by: Jason Gustafson <ja...@confluent.io> commit 2b326faf5f5a2bc632b842c26d2f48a07fa8a15e Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-23T16:45:31Z KAFKA-5247; Materialize committed offsets in offset order With this patch, offset commits are always materialized according to the order of the commit records in the offsets topic. Before this patch, transactional offset commits were materialized in transaction order. However, the log cleaner will always preserve the record with the greatest offset. This meant that if there was a mix of offset commits from a consumer and a transactional producer, then it we would switch from transactional order to offset order after cleaning, resulting in an inconsistent state. Author: Apurva Mehta <apu...@confluent.io> Reviewers: Guozhang Wang <wangg...@gmail.com>, Jason Gustafson <ja...@confluent.io> Closes #3108 from apurvam/KAFKA-5247-materialize-committed-offsets-in-offset-order (cherry picked from commit d1853f791147d66303c2700a8e58af762518188c) Signed-off-by: Jason Gustafson <ja...@confluent.io> commit 9e0581df513f2953bb7a899f784410bc99407465 Author: Damian Guy <damian....@gmail.com> Date: 2017-05-23T16:50:45Z MINOR: remove TransactionCoordinatorIntegrationTest `TransactionCoordinatorIntegrationTest` is not covering anything that isn't already covered by the more complete `TransactionsTest` Author: Damian Guy <damian....@gmail.com> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #3128 from dguy/minor-remove-test (cherry picked from commit 5a6676bfca9f56c467ff46aefffc6f344ff4bf65) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit fdbf2e5f7ad99b25feff2122cbcf4c51697250e3 Author: Jason Gustafson <ja...@confluent.io> Date: 2017-05-23T16:53:18Z MINOR: Log transaction metadata state transitions plus a few cleanups Author: Jason Gustafson <ja...@confluent.io> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #3081 from hachikuji/minor-add-txn-transition-logging (cherry picked from commit 70ec4b1d927cad7373eda2c54ef44bfc4275832f) Signed-off-by: Jason Gustafson <ja...@confluent.io> commit 49a2482c22ca75516038ce961b5d053e9d2def56 Author: Matthias J. Sax <matth...@confluent.io> Date: 2017-05-23T18:53:13Z MINOR: improve EmbeddedKafkaCluster test utility for deleting topics Author: Matthias J. Sax <matth...@confluent.io> Reviewers: Bill Bejeck, Damian Guy, Eno Thereska, Guozhang Wang Closes #3104 from mjsax/minor-improve-embedded-kafka-cluster (cherry picked from commit 49587750570a2aaf332eb73f5aad373196f984ce) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit e6600a36816718c90f896ab89bda6650de60ad78 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-05-23T20:34:43Z KAFKA-5280: Protect txn metadata map with read-write lock Two major changes plus one minor change: 0. change stateLock to a read-write lock. 1. Put the check of "isCoordinator" and "coordinatorLoading" together with the return of the metadata, under one read lock block, since otherwise we can get incorrect behavior if there is a change in the metadata cache after the check but before the accessing of the metadata. 2. Grab the read lock right before trying to append to local txn log, and until the local append returns; this is to avoid the scenario that the epoch has actually changed when we are appending to local log (e.g. emigration followed by immigration). 3. only watch on txnId instead of txnId and txnPartitionId in the txn marker purgatory, and disable reaper thread, as we can now safely clear all the delayed operations by traversing the marker queues. Author: Guozhang Wang <wangg...@gmail.com> Reviewers: Jason Gustafson, Jun Rao Closes #3082 from guozhangwang/K5231-read-write-lock (cherry picked from commit 1bf64833168e81015e1f30a30c64eb6849c15422) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit 69970630ba91de4a48b83a9101c487cd4eff3b46 Author: Onur Karaman <okara...@linkedin.com> Date: 2017-05-23T23:29:55Z KAFKA-5310; reset ControllerContext during resignation This ticket is all about ControllerContext initialization and teardown. The key points are: 1. we should teardown ControllerContext during resignation instead of waiting on election to fix it up. A heapdump shows that the former controller keeps pretty much all of its ControllerContext state laying around. 2. we don't properly teardown/reset ControllerContext.partitionsBeingReassigned. This can cause problems when the former controller becomes re-elected as controller at a later point in time. Suppose a partition assignment is initially R0. Now suppose a reassignment R1 gets stuck during controller C0 and an admin tries to "undo" R1 (by deleting /admin/partitions_reassigned, deleting /controller, and submitting another reassignment specifying R0). The new controller C1 may succeed with R0. If the controller moves back to C0, it will then reattempt R1 even though that partition reassignment has been cleared from zookeeper prior to shifting the controller back to C0. This results in the actual partition reassignment in zookeeper being unexpectedly changed back to R1. Author: Onur Karaman <okara...@linkedin.com> Reviewers: Jun Rao <jun...@gmail.com> Closes #3122 from onurkaraman/KAFKA-5310 (cherry picked from commit beeddc25d6b443ed344658a2562a162fb03048ef) Signed-off-by: Jun Rao <jun...@gmail.com> commit 999c247e076cbf7e84141533c5a9042a844d57ae Author: Ismael Juma <ism...@juma.me.uk> Date: 2017-05-23T23:36:52Z KAFKA-5135; Controller Health Metrics (KIP-143) Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Jun Rao <jun...@gmail.com>, Onur Karaman <okara...@linkedin.com> Closes #2983 from ijuma/kafka-5135-controller-health-metrics-kip-143 (cherry picked from commit 516d8457d8142111a91af94cab918c84990685da) Signed-off-by: Ismael Juma <ism...@juma.me.uk> commit 44294a8941145fec2d973fd448ccbf97c86f0dd4 Author: dejan2609 <dejan2...@gmail.com> Date: 2017-05-24T05:07:17Z KAFKA-5081: Force jackson-annotations to a single version matching expected Jackson version **JIRA ticket:** [KAFKA-5081 two versions of jackson-annotations-xxx.jar in distribution tgz](https://issues.apache.org/jira/browse/KAFKA-5081) **Solutions:** 1. accept this merge request **_OR_** 2. upgrade jackson libraries to version **_2.9.x_** (currently available as a pre-release only) **Related jackson issue:** [Add explicit \`jackson-annotations\` dependency version for \`jackson-databind\`](https://github.com/FasterXML/jackson-databind/issues/1545) **Note:** previous (equivalent) merge request #2900 ended up deep in the sand with swarm of messages due to flaky test, so I opted to close it and to open this one. ijuma: FYI Author: dejan2609 <dejan2...@gmail.com> Reviewers: Ewen Cheslack-Postava <e...@confluent.io> Closes #3116 from dejan2609/KAFKA-5081 (cherry picked from commit 90dc2d16fbed59fc65e6e190380c8b9c43b6d95a) Signed-off-by: Ewen Cheslack-Postava <m...@ewencp.org> commit 23de4f4cdda23181fe0b925dd9feb2444dc06e3a Author: Apurva Mehta <apu...@confluent.io> Date: 2017-05-24T06:08:57Z KAFKA-5273: Make KafkaConsumer.committed query the server for all partitions Before this patch the consumer would return the cached offsets for partitions in its current assignment. This worked when all the offset commits went through the consumer. With KIP-98, offsets can be committed transactionally through the producer. This means that relying on cached positions in the consumer returns incorrect information: since commits go through the producer, the cache is never updated. Hence we need to update the `KafkaConsumer.committed` method to always lookup the server for the last committed offset to ensure it gets the correct information every time. Author: Apurva Mehta <apu...@confluent.io> Reviewers: Jason Gustafson, Guozhang Wang Closes #3119 from apurvam/KAFKA-5273-kafkaconsumer-committed-should-always-hit-server (cherry picked from commit 4d89db968257c0c3273ece7e5546cb95ddcbeb46) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit 308c7e47c3b8c4fa2835534b9d28dd9c2e2dc1a3 Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Date: 2017-05-24T16:14:10Z MINOR: Improve the help doc of consumer group command Clarify the consumer group command help message around `zookeeper`, `bootstrap-server`, and `new-consumer` options. Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Reviewers: Jason Gustafson <ja...@confluent.io> Closes #2046 from vahidhashemian/minor/improve_consumer_group_command_doc (cherry picked from commit 88200938f0c3ace5ec0bdb65a315ebd9b6adb4d0) Signed-off-by: Jason Gustafson <ja...@confluent.io> commit 7cb57dc99bc775b8a195b5ba507cb2a31a00f553 Author: Jiangjie Qin <becket....@gmail.com> Date: 2017-05-22T00:31:31Z KAFKA-3995; KIP-126 Allow KafkaProducer to split and resend oversized batches Author: Jiangjie Qin <becket....@gmail.com> Reviewers: Joel Koshy <jjkosh...@gmail.com>, Ismael Juma <ism...@juma.me.uk> Closes #2638 from becketqin/KAFKA-3995 (cherry picked from commit 7fad45557e4cb7b345f34cec32f910b437c59bc2) Signed-off-by: Jiangjie Qin <becket....@gmail.com> commit 9a21bf20b623b790c9910813e373d2d61fe84c2c Author: Jason Gustafson <ja...@confluent.io> Date: 2017-05-24T22:26:46Z KAFKA-5259; TransactionalId auth implies ProducerId auth Author: Jason Gustafson <ja...@confluent.io> Reviewers: Apurva Mehta <apu...@confluent.io>, Jun Rao <jun...@gmail.com> Closes #3075 from hachikuji/KAFKA-5259-FIXED (cherry picked from commit 38f6cae9e879baa35c5dbc5829bf09ecd59930c2) Signed-off-by: Jason Gustafson <ja...@confluent.io> commit a8dbce47f5a39ebc0fb21b7e0ef43f837b252313 Author: Jeyhun Karimov <je.kari...@gmail.com> Date: 2017-05-25T02:00:37Z KAFKA-4144 Follow-up: add one missing overload function to maintain backward compatibility A follow up RP to fix [issue](https://github.com/confluentinc/examples/commit/2cd0b87bc8a7eab0e7199fa0079db6417f0e6b63#commitcomment-22200864) Author: Jeyhun Karimov <je.kari...@gmail.com> Reviewers: Matthias J. Sax, Eno Thereska, Bill Bejeck, Guozhang Wang Closes #3109 from jeyhunkarimov/KAFKA-4144-follow-up (cherry picked from commit c5d44af77473abb36cb9bf7ca3dead36490b8320) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit 8dfc098ee5c5024a580dab50281e19fec2dd6cbf Author: Jason Gustafson <ja...@confluent.io> Date: 2017-05-25T04:00:44Z MINOR: GroupCoordinator can append with group lock Author: Jason Gustafson <ja...@confluent.io> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #3133 from hachikuji/minor-replica-manager-append-refactor (cherry picked from commit fdcee8b8b3c027cdc1b13031fa19fcfc7de5609f) Signed-off-by: Jason Gustafson <ja...@confluent.io> commit a89200a585593cebb4a3a1af0c2af9892babcd5b Author: Michael André Pearce <michael.andre.pea...@me.com> Date: 2017-05-21T21:34:19Z KAFKA-4208; Add record headers upgrade doc Update upgrade.html Raising this now, as KIP-118 is pulled from release as such submitting this without java 8 changes. As per remaining review comment from https://github.com/apache/kafka/pull/2772, updating the upgrade notes. Author: Michael André Pearce <michael.andre.pea...@me.com> Author: Michael Andre Pearce <michael.andre.pea...@me.com> Reviewers: Jiangjie Qin <becket....@gmail.com>, Ismael Juma <ism...@juma.me.uk> Closes #2991 from michaelandrepearce/KIP-82 ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---