GitHub user stathmarxis opened a pull request: https://github.com/apache/kafka/pull/1558
org.apache.kafka.streams.errors.StreamsException: Failed to rebalance <p>I am running the below simple example in kafka streams and i got a weird exception which i cannot handle</p> Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-pipe"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.1.3:9092"); props.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass()); props.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass()); // setting offset reset to earliest so that we can re-run the demo code with the same pre-loaded data props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); KStreamBuilder builder = new KStreamBuilder(); builder.stream("streams-file-input").to("streams-pipe-output"); KafkaStreams streams = new KafkaStreams(builder, props); streams.start(); // usually the stream application would be running forever, // in this example we just let it run for some time and stop since the input data is finite. Thread.sleep(5000L); streams.close(); [Exception in thread "StreamThread-1" org.apache.kafka.streams.errors.StreamsException: Failed to rebalance at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:299) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:218) Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error while creating the state manager at org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:71) at org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:86) at org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:550) at org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:577) at org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:68) at org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:123) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:222) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:232) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:227) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) at org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:436) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:422) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658) at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:243) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:345) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:977) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:937) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:295) ... 1 more Caused by: java.io.FileNotFoundException: C:\tmp\kafka-streams\my-streapplication\0_0\.lock (The system cannot find the path specified) at java.io.RandomAccessFile.open0(Native Method) at java.io.RandomAccessFile.open(RandomAccessFile.java:316) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) at org.apache.kafka.streams.processor.internals.ProcessorStateManager.lockStateDirectory(ProcessorStateManager.java:125) at org.apache.kafka.streams.processor.internals.ProcessorStateManager.<init>(ProcessorStateManager.java:93) at org.apache.kafka.streams.processor.internals.AbstractTask.<init>(AbstractTask.java:69)](url) You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/kafka 0.10.0 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1558.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1558 ---- commit 1b5879653e0d956c79556301d1d11987baf6f2d7 Author: Gwen Shapira <csh...@gmail.com> Date: 2016-03-21T16:48:02Z Changing version to 0.10.0.0 commit 4f8b3aed8dc6a77293c526253123d999a23af149 Author: Guozhang Wang <wangg...@gmail.com> Date: 2016-03-21T19:06:07Z MINOR: Add InterfaceStability.Unstable annotations to some Kafka Streams public APIs Also improves Java docs for the Streams high-level DSL. Author: Guozhang Wang <wangg...@gmail.com> Reviewers: Ismael Juma, Michael G. Noll Closes #1097 from guozhangwang/KNewJavaDoc (cherry picked from commit b6c29e3810bd59f39fa93c429817396cf8c324b7) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit 3710b12e6e6a6d8dddb7402087ee550cd14909db Author: Gwen Shapira <csh...@gmail.com> Date: 2016-03-21T20:15:53Z MINOR: fix documentation version This will need to be double-committed. Author: Gwen Shapira <csh...@gmail.com> Reviewers: Ismael Juma <ism...@juma.me.uk>, Ewen Cheslack-Postava <e...@confluent.io> Closes #1107 from gwenshap/fix-doc-version (cherry picked from commit 69df734470f1db9ba1732cb4bedaa1541d721103) Signed-off-by: Ewen Cheslack-Postava <m...@ewencp.org> commit 3cfd20b7b9d609c1055f6c23de86b6133e5cac75 Author: Gwen Shapira <csh...@gmail.com> Date: 2016-03-22T01:58:45Z MINOR: update new version in additional places matching set of version fixes. ewencp junrao Author: Gwen Shapira <csh...@gmail.com> Reviewers: Ismael Juma <ism...@juma.me.uk>, Geoff Anderson <ge...@confluent.io>, Ewen Cheslack-Postava <e...@confluent.io> Closes #1110 from gwenshap/minor-fix-version-010 commit 808f85f03d8f69047914eb21438d1458e23c4325 Author: Jason Gustafson <ja...@confluent.io> Date: 2016-03-22T03:47:25Z KAFKA-3412: multiple asynchronous commits causes send failures Author: Jason Gustafson <ja...@confluent.io> Reviewers: Ismael Juma <ism...@juma.me.uk>, Ewen Cheslack-Postava <e...@confluent.io> Closes #1108 from hachikuji/KAFKA-3412 (cherry picked from commit 8d8e3aaa6172d314230a8d61e6892e9c09dc45b6) Signed-off-by: Ewen Cheslack-Postava <m...@ewencp.org> commit c0f870f8509253564c73ca76cd9fabb7df4ce6ad Author: Grant Henke <granthe...@gmail.com> Date: 2016-03-22T18:53:12Z KAFKA-3435: Remove `Unstable` annotation from new Java Consumer Author: Grant Henke <granthe...@gmail.com> Reviewers: Gwen Shapira Closes #1113 from granthenke/remove-unstable (cherry picked from commit 99cfb99aa161007f2ceae33945d11b968d932175) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit 66ac5157d914aacf9b544fe7de91020f445cef3c Author: gaob13 <gao...@mails.tsinghua.edu.cn> Date: 2016-03-22T18:58:01Z MINOR: Remove the very misleading comment lines It is not true in practice. Maybe the implied feature is not yet implemented or removed. These lines can be super misleading. Please merge. Thank you. Author: gaob13 <gao...@mails.tsinghua.edu.cn> Reviewers: Ismael Juma, Ewen Cheslack-Postava Closes #793 from gaob13/trunk (cherry picked from commit 73470b028cd659c1c405e6b6ba52bf483113b90b) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit e2d7f9e445a2cbb5e1e0ed8437af849b731677bf Author: Ismael Juma <ism...@juma.me.uk> Date: 2016-03-22T19:39:04Z KAFKA-3426; Improve protocol type errors when invalid sizes are received Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Grant Henke, Gwen Shapira Closes #1100 from ijuma/kafka-3426-invalid-protocol-type-errors-invalid-sizes (cherry picked from commit 73c79000edddd929cd0af25f4a29fcc682a1c9c0) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit 687d2494ff27b05f21ea29e645a0b1b000334db6 Author: Jason Gustafson <ja...@confluent.io> Date: 2016-03-22T20:09:13Z KAFKA-3319: improve session timeout broker/client config documentation Author: Jason Gustafson <ja...@confluent.io> Reviewers: Grant Henke, Ismael Juma, Guozhang Wang Closes #1106 from hachikuji/KAFKA-3319 (cherry picked from commit ca77d67058726fc9df9bdd7cc0217ee62ccc5106) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit 05984de69ee8eb06f214a2e42abd789a4fbc8cef Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Date: 2016-03-22T20:10:07Z KAFKA-3219: Fix long topic name validation This fixes an issue with long topic names by considering, during topic validation, the '-' and the partition id that is appended to the log folder created for each topic partition. Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Reviewers: Gwen Shapira, Grant Henke Closes #898 from vahidhashemian/KAFKA-3219 (cherry picked from commit ad3dfc6ab25c3f80d2425e24e72ae732b850dc60) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit a81e5d05e6fea7fb74dfb19b67729b6dd4ca2423 Author: Ewen Cheslack-Postava <m...@ewencp.org> Date: 2016-03-22T21:09:10Z KAFKA-3435: Follow up to fix checkstyle Author: Ewen Cheslack-Postava <m...@ewencp.org> Reviewers: Gwen Shapira Closes #1116 from ewencp/kafka-3435-follow-up (cherry picked from commit d6b450ced57bb5ab8a7164358e6a360f97a7b5cc) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit d9cbe566371ed1f519fa983c14fc48d088f66b6a Author: Grant Henke <granthe...@gmail.com> Date: 2016-03-22T22:01:05Z KAFKA-3301; CommonClientConfigs.METRICS_SAMPLE_WINDOW_MS_DOC is incor⦠â¦rect Author: Grant Henke <granthe...@gmail.com> Reviewers: Jun Rao <jun...@gmail.com> Closes #1114 from granthenke/window-docs (cherry picked from commit 983e9486ef139f8c9797679732d09feb3f7bfb87) Signed-off-by: Jun Rao <jun...@gmail.com> commit d1e74890caaf3b76b0d9a888742a2cf1c6e6e883 Author: Ismael Juma <ism...@juma.me.uk> Date: 2016-03-23T01:20:07Z KAFKA-3447; partitionState in UpdateMetadataRequest not logged properly state-change log Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Jun Rao <jun...@gmail.com> Closes #1117 from ijuma/kafka-3447-metadata-cache-logging (cherry picked from commit adca4d7df3155ac29ae510c237adf06c9521d221) Signed-off-by: Jun Rao <jun...@gmail.com> commit aead28a54e55ca0541d696abced523b453bb50f7 Author: Ismael Juma <ism...@juma.me.uk> Date: 2016-03-23T02:13:26Z KAFKA-3431: Remove `o.a.k.common.BrokerEndPoint` in favour of `Node` Also included a minor efficiency improvement in kafka.cluster.EndPoint. Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Gwen Shapira Closes #1105 from ijuma/kafka-3431-replace-broker-end-point-with-node (cherry picked from commit 255b5e13863a95cfc327236856db2df188f04d49) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit e5a1f704d7ed59f9a6a423bbc36510d13a5df1de Author: Jiangjie Qin <becket....@gmail.com> Date: 2016-03-23T14:15:59Z KAFKA-3442; Fix FileMessageSet iterator. Author: Jiangjie Qin <becket....@gmail.com> Reviewers: Ismael Juma <ism...@juma.me.uk>, Jun Rao <jun...@gmail.com> Closes #1112 from becketqin/KAFKA-3442 (cherry picked from commit 7af67ce22aa02121d6b82dc54dad42358282e524) Signed-off-by: Jun Rao <jun...@gmail.com> commit 2835e73000a44c9586d085275b62eeeb2ce1fcec Author: Jason Gustafson <ja...@confluent.io> Date: 2016-03-23T16:47:48Z KAFKA-3409: handle CommitFailedException in MirrorMaker Author: Jason Gustafson <ja...@confluent.io> Reviewers: Ismael Juma, Ashish Singh, Guozhang Wang Closes #1115 from hachikuji/KAFKA-3409 (cherry picked from commit 20c313526a0518a51142d3abc5ee2a4d2ef3cb34) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit f5e1ca625332572923b42a9ae8d660f2452ae92f Author: Grant Henke <granthe...@gmail.com> Date: 2016-03-23T19:54:26Z KAFKA-3441: 0.10.0 documentation still says "0.9.0" Author: Grant Henke <granthe...@gmail.com> Reviewers: Gwen Shapira Closes #1122 from granthenke/docs-10 (cherry picked from commit d57847641037823a7306f53251b063869f7affb6) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit 63e9d246b7978152e12aa4ea8b1ba6bcc40f4498 Author: Ismael Juma <ism...@juma.me.uk> Date: 2016-03-23T20:53:37Z KAFKA-3432; Cluster.update() thread-safety Replace `update` with `withPartitions`, which returns a copy instead of mutating the instance. Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #1118 from ijuma/kafka-3432-cluster-update-thread-safety (cherry picked from commit d4d5920ed40736d21f056188efa8a86c93e22506) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit b31a2f06c80c2c2cacf34d75998d401bddcca374 Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-23T21:25:08Z HOTFIX: fix NPE in changelogger Fix NPE in StoreChangeLogger caused by a record out of window retention period. guozhangwang Author: Yasuhiro Matsuda <yasuh...@confluent.io> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #1124 from ymatsuda/logger_npe (cherry picked from commit 80d78f81470f109dc6d221f755b039c7332bb93b) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit 8525d8279b7798db90f752c329aa3b5936cb5f7f Author: Yasuhiro Matsuda <yasuh...@confluent.io> Date: 2016-03-23T21:57:03Z MINOR: remove streams-smoke-test.sh guozhangwang Author: Yasuhiro Matsuda <yasuh...@confluent.io> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #1125 from ymatsuda/remove_smoketest_shell_script (cherry picked from commit de062443381df84ee0d65acc20e44ffca2b2552b) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit 0cc997ae76010cf85b55d057dea9996fa1931e5b Author: Gwen Shapira <csh...@gmail.com> Date: 2016-03-23T22:41:07Z MINOR: Revert 0.10.0 branch to SNAPSHOT per change in release process Author: Gwen Shapira <csh...@gmail.com> Reviewers: Ewen Cheslack-Postava Closes #1126 from gwenshap/minor-release-version commit 4e557f8ef60d46a8870704655c9a35092f74d125 Author: Jason Gustafson <ja...@confluent.io> Date: 2016-03-24T05:36:19Z KAFKA-3434; add old constructor to ConsumerRecord Author: Jason Gustafson <ja...@confluent.io> Reviewers: Grant Henke <granthe...@gmail.com>, Ismael Juma <ism...@juma.me.uk>, Ewen Cheslack-Postava <e...@confluent.io> Closes #1123 from hachikuji/KAFKA-3434 (cherry picked from commit cb78223bf90aca4f75699f36c1a82db7661a62f3) Signed-off-by: Ewen Cheslack-Postava <m...@ewencp.org> commit 4e5653d77359214452b94a65c9db29eaa57a3c31 Author: Grant Henke <granthe...@gmail.com> Date: 2016-03-25T17:07:05Z KAFKA-3460: Remove old 0.7 KafkaMigrationTool Author: Grant Henke <granthe...@gmail.com> Reviewers: Gwen Shapira Closes #1136 from granthenke/remove-07-migration (cherry picked from commit d3a66a65365579e9320347663969b30c1148e497) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit 3f930cd84c1a889f0df2bc5057e073e683860171 Author: Jason Gustafson <ja...@confluent.io> Date: 2016-03-25T19:51:54Z KAFKA-3463: change default receive buffer size for consumer to 64K Author: Jason Gustafson <ja...@confluent.io> Reviewers: Gwen Shapira Closes #1140 from hachikuji/KAFKA-3463 (cherry picked from commit d691faf98cb573c4e92748d95d5c8afc492db806) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit 1b1b949b7fbd10fe4b08e97769ab024fad8459fb Author: Andrea Cosentino <anco...@gmail.com> Date: 2016-03-25T22:00:45Z KAFKA-3449: Rename filterOut() to filterNot() to achieve better terminology â¦nology Hi all, This is my first contribution and I hope it will be good. The PR is related to this issue: https://issues.apache.org/jira/browse/KAFKA-3449 Thanks a lot, Andrea Author: Andrea Cosentino <anco...@gmail.com> Reviewers: Yasuhiro Matsuda, Guozhang Wang Closes #1134 from oscerd/KAFKA-3449 (cherry picked from commit c1d8c38345e0a1e04ced143ed07e63fe02ceb8b0) Signed-off-by: Guozhang Wang <wangg...@gmail.com> commit 496bd3fd41c0272b7ef77652b85cbaae748e100e Author: Guozhang Wang <wangg...@gmail.com> Date: 2016-03-25T23:04:58Z KAFKA-3454: add Kafka Streams web docs Author: Guozhang Wang <wangg...@gmail.com> Reviewers: Gwen Shapira Closes #1127 from guozhangwang/KStreamsDocs (cherry picked from commit 23b50093f4100ce7fbff325cdc92ee6cf3c54102) Signed-off-by: Gwen Shapira <csh...@gmail.com> commit 206757eeb18589952291ce1a5578c66de3669f4c Author: Liquan Pei <liquan...@gmail.com> Date: 2016-03-25T23:46:53Z KAFKA-3316: Add REST API for listing connector plugins Author: Liquan Pei <liquan...@gmail.com> Reviewers: Ewen Cheslack-Postava <e...@confluent.io> Closes #1090 from Ishiihara/kafka-3316 (cherry picked from commit 78fa20eb58a948abd9ad4e44acfed606400a47f3) Signed-off-by: Ewen Cheslack-Postava <m...@ewencp.org> commit b45fe77791560a2171e0bbd7df09f648a01b016f Author: Jiangjie Qin <becket....@gmail.com> Date: 2016-03-26T16:22:59Z KAFKA-3388; Fix expiration of batches sitting in the accumulator Author: Jiangjie Qin <becket....@gmail.com> Reviewers: Ismael Juma <ism...@juma.me.uk>, Jun Rao <jun...@gmail.com> Closes #1056 from becketqin/KAFKA-3388 (cherry picked from commit 1fbe445dde71df0023a978c5e54dd229d3d23e1b) Signed-off-by: Jun Rao <jun...@gmail.com> commit f37b550502f6e3429cb3582db7e1c93614e84c58 Author: Ismael Juma <ism...@juma.me.uk> Date: 2016-03-28T16:00:03Z MINOR: Fix typo and tweak wording in `RecordAccumulator` comments This was recently introduced in: https://github.com/apache/kafka/commit/1fbe445dde71df0023a978c5e54dd229d3d23e1b Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Jun Rao <jun...@gmail.com> Closes #1152 from ijuma/fix-typos-in-record-accumulator (cherry picked from commit 4c0660bf3da9879cb405a0f85cf1524511e091e8) Signed-off-by: Jun Rao <jun...@gmail.com> commit afe82512ba5ec3c22ce0e99acfea9ce4d5b2bb2e Author: Ismael Juma <ism...@juma.me.uk> Date: 2016-03-28T21:35:31Z MINOR: Remove a couple of redundant `CoreUtils.rm` methods Also: * Rename remaining `CoreUtils.rm` to `delete` for consistency * Use `try with resources` in `Utils` to simplify code * Silence compiler warning due to exception catch clause in `TestUtils` Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #1153 from ijuma/remove-redundant-core-utils-rm (cherry picked from commit 43d5078e981bbb25fd81cdc8ba4c339cd2d3f3d2) Signed-off-by: Guozhang Wang <wangg...@gmail.com> ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---