[jira] [Commented] (KAFKA-559) Garbage collect old consumer metadata entries
[
https://issues.apache.org/jira/browse/KAFKA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13699819#comment-13699819
]
Swapnil Ghike commented on KAFKA-559:
-
Some feedback:
1. Passing a groupId for cleanup will make the cleanup job tedious since we
tend to have hundreds of console-consumer group ids in ZK that are stale.
Running the tool for a particular topic or all topics probably makes more
sense.
2. I would suggest accepting a date param "mm-dd- hh:mm:ss,SSS" as a String
instead of accepting a timestamp value, and deleting the group only if it has
had no updates to its offsets since that date, as described above.
3. It's dangerous to delete the entire group if the date/"since" is not
provided. It's very easy for user to specify only two arguments (topic and
zkconnect) and not specify the date. Let's also make sure that the user always
specifies a date.
4. "dry-run" does not need to accept any value. You can simply use
parser.accepts("dry-run", "") and then use if (options.has(dryRunOpt)) {
yeay } else { nay }.
5. We can inline exitIfNoPathExists, the implementation is small and clear
enough.
6. We should have an info statement when the group ids are deleted in the non
dry-run mode.
7. info("Removal has successfully completed.") can probably be refactored to
something more specific to this tool.
8. Instead of writing a different info statement for dry-run mode, I think you
should be able to set logIdent of Logging to "[dry-run]" or "" depending on
which mode the tool is working in. This will let you have a single info
statement for both modes.
Minor stuff:
1. I think we tend to use camelCase in variable names instead of underscores.
2. Whitespaces can be made more consistent.
> Garbage collect old consumer metadata entries
> -
>
> Key: KAFKA-559
> URL: https://issues.apache.org/jira/browse/KAFKA-559
> Project: Kafka
> Issue Type: New Feature
>Reporter: Jay Kreps
>Assignee: Tejas Patil
> Labels: project
> Attachments: KAFKA-559.v1.patch
>
>
> Many use cases involve tranient consumers. These consumers create entries
> under their consumer group in zk and maintain offsets there as well. There is
> currently no way to delete these entries. It would be good to have a tool
> that did something like
> bin/delete-obsolete-consumer-groups.sh [--topic t1] --since [date]
> --zookeeper [zk_connect]
> This would scan through consumer group entries and delete any that had no
> offset update since the given date.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings
[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13699857#comment-13699857 ] Ashwanth Fernando commented on KAFKA-718: - I checked out 0.8, committed my changes and while rebasing with the trunk (which I believe holds 0.7), there are a whole lot of inbound patches which have conflicts with 0.8 code. Do I try to manual merge these conflicts? I have no clue what those patches are. Followed the simple contributor workflow here - https://cwiki.apache.org/confluence/display/KAFKA/Git+Workflow except that for "git checkout -b xyz remotes/origin/trunk" I did git checkout -b 718 remotes/origin/0.8 > kafka-run-class.sh should use reasonable gc settings > > > Key: KAFKA-718 > URL: https://issues.apache.org/jira/browse/KAFKA-718 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Jay Kreps >Assignee: Neha Narkhede > > Our start script seems to use the default "stop the world" collector. It > would be good to default to well tuned gc settings including gc logging, CMS, > etc. Whatever we are using in prod and perf lab... > Many people who want to use kafka basically don't know java well so they > won't succeed in figuring this stuff out on their own and just think it is > broken and timing out if we don't have good defaults. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Subscription: outstanding kafka patches
Issue Subscription Filter: outstanding kafka patches (73 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-956 High-level consumer fails to check topic metadata response for errors https://issues.apache.org/jira/browse/KAFKA-956 KAFKA-946 Kafka Hadoop Consumer fails when verifying message checksum https://issues.apache.org/jira/browse/KAFKA-946 KAFKA-943 Move all configuration key string to constants https://issues.apache.org/jira/browse/KAFKA-943 KAFKA-932 System Test - set retry.backoff.ms=300 to all test cases https://issues.apache.org/jira/browse/KAFKA-932 KAFKA-925 Add optional partition key override in producer https://issues.apache.org/jira/browse/KAFKA-925 KAFKA-923 Improve controller failover latency https://issues.apache.org/jira/browse/KAFKA-923 KAFKA-922 System Test - set retry.backoff.ms=300 to testcase_0119 https://issues.apache.org/jira/browse/KAFKA-922 KAFKA-917 Expose zk.session.timeout.ms in console consumer https://issues.apache.org/jira/browse/KAFKA-917 KAFKA-915 System Test - Mirror Maker testcase_5001 failed https://issues.apache.org/jira/browse/KAFKA-915 KAFKA-898 Add a KafkaMetricsReporter that wraps Librato's reporter https://issues.apache.org/jira/browse/KAFKA-898 KAFKA-896 merge 0.8 (988d4d8e65a14390abd748318a64e281e4a37c19) to trunk https://issues.apache.org/jira/browse/KAFKA-896 KAFKA-885 sbt package builds two kafka jars https://issues.apache.org/jira/browse/KAFKA-885 KAFKA-883 System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID https://issues.apache.org/jira/browse/KAFKA-883 KAFKA-882 Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 https://issues.apache.org/jira/browse/KAFKA-882 KAFKA-881 Kafka broker not respecting log.roll.hours https://issues.apache.org/jira/browse/KAFKA-881 KAFKA-879 In system test, read the new leader from zookeeper instead of broker log on completion of become-leader state transition https://issues.apache.org/jira/browse/KAFKA-879 KAFKA-877 Still getting kafka.common.NotLeaderForPartitionException https://issues.apache.org/jira/browse/KAFKA-877 KAFKA-873 Consider replacing zkclient with curator (with zkclient-bridge) https://issues.apache.org/jira/browse/KAFKA-873 KAFKA-868 System Test - add test case for rolling controlled shutdown https://issues.apache.org/jira/browse/KAFKA-868 KAFKA-863 System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases https://issues.apache.org/jira/browse/KAFKA-863 KAFKA-859 support basic auth protection of mx4j console https://issues.apache.org/jira/browse/KAFKA-859 KAFKA-855 Ant+Ivy build for Kafka https://issues.apache.org/jira/browse/KAFKA-855 KAFKA-854 Upgrade dependencies for 0.8 https://issues.apache.org/jira/browse/KAFKA-854 KAFKA-852 Remove clientId from OffsetFetchResponse and OffsetCommitResponse https://issues.apache.org/jira/browse/KAFKA-852 KAFKA-836 Update quickstart for Kafka 0.8 https://issues.apache.org/jira/browse/KAFKA-836 KAFKA-815 Improve SimpleConsumerShell to take in a max messages config option https://issues.apache.org/jira/browse/KAFKA-815 KAFKA-745 Remove getShutdownReceive() and other kafka specific code from the RequestChannel https://issues.apache.org/jira/browse/KAFKA-745 KAFKA-735 Add looping and JSON output for ConsumerOffsetChecker https://issues.apache.org/jira/browse/KAFKA-735 KAFKA-717 scala 2.10 build support https://issues.apache.org/jira/browse/KAFKA-717 KAFKA-705 Controlled shutdown doesn't seem to work on more than one broker in a cluster https://issues.apache.org/jira/browse/KAFKA-705 KAFKA-686 0.8 Kafka broker should give a better error message when running against 0.7 zookeeper https://issues.apache.org/jira/browse/KAFKA-686 KAFKA-682 java.lang.OutOfMemoryError: Java heap space https://issues.apache.org/jira/browse/KAFKA-682 KAFKA-677 Retention process gives exception if an empty segment is chosen for collection https://issues.apache.org/jira/browse/KAFKA-677 KAFKA-674 Clean Shutdown Testing - Log segments checksums mismatch https://issues.apache.org/jira/browse/KAFKA-674 KAFKA-652 Create testcases for clean shut-down https://issues.apache.org/jira/browse/KAFKA-652 KAFKA-649 Cleanup log4j logging https://issues.apache.org/jira/browse/KAFKA-649 KAFKA-645 Create a shell script to run System Test with DEBUG details and "tee" console output to a file https://issues.apache.org/jira/browse/KAFKA-645 KAFKA-637 Sep
