GitHub user voronenko-da opened a pull request: https://github.com/apache/samza/pull/631
Invalid partition calculation in KafkaSystemProducer 0.14.1 If you use a byte array as the key, the key.hashCode() will be different all the time. Proposal to remove the preliminary calculation of the partition. https://github.com/apache/samza/blob/master/samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala val partitionKey = if (envelope.getPartitionKey != null) KafkaUtil.getIntegerPartitionKey(envelope, partitions) else null https://github.com/apache/samza/blob/master/samza-kafka/src/main/scala/org/apache/samza/util/KafkaUtil.scala def getIntegerPartitionKey(envelope: OutgoingMessageEnvelope, partitions: java.util.List[PartitionInfo]): Integer = { val numPartitions = partitions.size abs(envelope.getPartitionKey.hashCode()) % numPartitions } You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/samza 0.14.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/samza/pull/631.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #631 ---- commit 3f2c0775bfc69e7464f45343503c5e51310d55dd Author: xiliu <xiliu@...> Date: 2018-05-07T22:25:35Z Trigger notification to github of 0.14.1 branch commit 5ca7fac5767496a5c22ed6cc3096f366c1e304a9 Author: xiliu <xiliu@...> Date: 2018-05-07T23:11:59Z SAMZA-1702: Prepare 0.14.1 release on the 0.14.1 branch Author: xiliu <xi...@linkedin.com> Reviewers: Prateek M <prate...@apache.org> Closes #505 from xinyuiscool/SAMZA-1702 commit 06039b789b8134c4dbbe7883b34722427b3f46c8 Author: xiliu <xiliu@...> Date: 2018-05-08T00:06:35Z SAMZA-1703: Disable flaky test TestEmbeddedTaggedRateLimiter.testAcquireWithTimeout Author: xiliu <xi...@linkedin.com> Reviewers: Boris S <sbo...@apache.org> Closes #507 from xinyuiscool/SAMZA-1703 (cherry picked from commit 9ac221e4e99d3e576c7fad03d0b83eeb489564e9) Signed-off-by: xiliu <xi...@linkedin.com> commit 4b19efefec78ec6d2f7352dfa1972a9d04181861 Author: xiliu <xiliu@...> Date: 2018-05-08T01:14:29Z SAMZA-1704: Fix compatibility issues with scala 2.12 Need to add override keyword for overriding a method in scala 2.12. Author: xiliu <xi...@linkedin.com> Reviewers: Prateek M <prate...@apache.org> Closes #508 from xinyuiscool/SAMZA-1704 (cherry picked from commit 31a9e4aea8a8233853334b7ed70da8694f42bd52) Signed-off-by: xiliu <xi...@linkedin.com> commit 44d6f40f7d8d331e40752fc369a83fc69edf7867 Author: xinyuiscool <xiliu@...> Date: 2018-05-09T00:48:55Z SAMZA-1705: Switch to use snapshot in iterable impl of RocksDb We should use rocksDb.snapshot() method to keep the snapshot and creates a new iterator with it all the time. The perf shows a little bit more expensive but mostly on par with range iterator query. Author: xinyuiscool <xi...@linkedin.com> Reviewers: Jagadish V <vjagad...@apache.org> Closes #510 from xinyuiscool/SAMZA-1705 (cherry picked from commit 89beb1fccb01c781a4de905d57a4bd99df25577a) Signed-off-by: xiliu <xi...@linkedin.com> commit 64ea33d7d89c5f7f934a0cc8290f3ad3f925e98a Author: Hai Lu <halu@...> Date: 2018-05-09T22:32:44Z SAMZA-1706: lazy initialization for eventhub system producer We are seeing slow shutdown issue for eventhub system producers for users who only use eventhub consumer (but then Samza system creates both consumer and producer together no matter what). As a workaround, add lazy initialization to the producer to avoid the slow shutdown Author: Hai Lu <h...@linkedin.com> Reviewers: Jagadish <jagad...@apache.org> Closes #511 from lhaiesp/master commit adb514aaee046bf760e4f992c45e1f4582bb759d Author: Prateek Maheshwari <pmaheshw@...> Date: 2018-05-10T16:21:12Z Fixed test failure for TestRocksDbKeyValueStoreJava#testIterate RocksDB Snapshots and any iterators obtained from them need to be closed before the store is closed. Otherwise the process aborts with the following message (at least on OSX): `Assertion failed: (is_last_reference), function ~ColumnFamilyData, file db/column_family.cc, line 457.` Author: Prateek Maheshwari <pmahe...@linkedin.com> Reviewers: Jagadish V <vjagadish1...@gmail.com> Closes #513 from prateekm/rocksdb-test-fi commit 1e1330cefbcfc7f2ff8e20c8ff19c465f7df4b66 Author: Prateek Maheshwari <pmaheshw@...> Date: 2018-05-10T19:16:33Z Fixed test failure for TestRocksDbKeyValueStoreJava#testPerf Iterators (incl. those obtained from snapshots) must be closed before store close. Author: Prateek Maheshwari <pmahe...@linkedin.com> Reviewers: Xinyu Liu <xinyuliu...@gmail.com> Closes #514 from prateekm/rocksdb-test-fi (cherry picked from commit 54c690e6fc52ac13d1c59fe1bd1c34f35f710164) Signed-off-by: xiliu <xi...@linkedin.com> commit b2ca67ea19becac1ea8c7a73d544438e75f94784 Author: xinyuiscool <xiliu@...> Date: 2018-05-10T21:19:08Z Remove the iterable interface from KeyValueSnapshot The iterable interface makes it hard for the users to close it after using. Author: xinyuiscool <xi...@linkedin.com> Reviewers: Prateek M <prate...@apache.org> Closes #516 from xinyuiscool/kv-snapshot (cherry picked from commit e204d298e84f549ecf4d8d9d642304502f046c29) Signed-off-by: xiliu <xi...@linkedin.com> commit 973e0735e076101fa9c05e88df999cfdd289fd0a Author: Yi Pan (Data Infrastructure) <yipan@...> Date: 2018-05-17T05:19:37Z SAMZA-1715: Unit test for Kafka admin deletedMessagesCalled() fail Test locally and works. Author: Yi Pan (Data Infrastructure) <yi...@yipan-mn1.linkedin.biz> Reviewers: Jagadish <jagad...@apache.org> Closes #523 from nickpan47/fix-unittest-deleted-messages commit 69e63d8153b89cf47c618b3f4e871609d7b18a02 Author: Shanthoosh Venkataraman <santhoshvenkat1988@...> Date: 2018-05-18T19:29:24Z SAMZA-1720: Remove javafx.util dependency from samza-sql tests. In samza-sql module, currently few test classes(`TestSamzaSqlRelMessageSerde` and `TestSamzaSqlRelRecordSerde`) are dependent upon `javafx.util.Pair` class(coming from `javafx` module). `javafx.util.Pair` is not supported by default in all JDK builds(example; open-jdk java-8 doesn't support `javafx` module) and it belongs to `javafx` package which is primarily used for developing GUI applications. This dependency is removed and replaced with `Pair` class from `apache-commons`. Author: Shanthoosh Venkataraman <santhoshvenkat1...@gmail.com> Reviewers: Jagadish V <vjagad...@gmail.com> Closes #527 from shanthoosh/SAMZA-1720 (cherry picked from commit 3b9e14be3aa5fc2b6003b7e244fba466db34236f) Signed-off-by: xiliu <xi...@linkedin.com> commit 9bc03f7ab32199a15af157cc515d897917694cb5 Author: Srinivasulu Punuru <spunuru@...> Date: 2018-05-18T19:31:44Z Implementing the fetchSinkInfo in ConfigBasedIOResolver 1. I think we missed implementing the fetchSinkInfo method in the ConfigBasedResolver when the API was introduced which is breaking the samza sql console tool. This fixes it. 2. latest release of mac removed realpath so the command line tools are broken. Removed the usage of realpath to fix these tools. Thanks to nickpan47 for identifying these problems. Author: Srinivasulu Punuru <spun...@linkedin.com> Reviewers: Yi Pan <nickpa...@gmail.com> Closes #528 from srinipunuru/release-fix.1 (cherry picked from commit 171793b69b33081fc6277c9505b3055f79fcb4b7) Signed-off-by: xiliu <xi...@linkedin.com> commit a652e6f5230a297f3d991c6441311ef55e562d9a Author: xiliu <xiliu@...> Date: 2018-05-25T22:18:38Z Remove git checkout latest from the docs in 0.14.1 branch ---- ---