[jira] [Commented] (KAFKA-649) Cleanup log4j logging

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13743888#comment-13743888
 ] 

Jun Rao commented on KAFKA-649:
---

Thanks for patch v6. Committed to 0.8.

> Cleanup log4j logging
> -
>
> Key: KAFKA-649
> URL: https://issues.apache.org/jira/browse/KAFKA-649
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8
>Reporter: Jay Kreps
>Assignee: Jun Rao
>Priority: Blocker
> Attachments: kafka-649_extra.patch, kafka-649.patch, 
> KAFKA-649.v3.patch, KAFKA-649.v4.patch, KAFKA-649.v5.patch, KAFKA-649.v6.patch
>
>
> Review the logs and do the following:
> 1. Fix confusing or duplicative messages
> 2. Assess that messages are at the right level (TRACE/DEBUG/INFO/WARN/ERROR)
> It would also be nice to add a log4j logger for the request logging (i.e. the 
> access log) and another for the controller state change log, since these 
> really have their own use cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-990) Fix ReassignPartitionCommand and improve usability

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13743930#comment-13743930
 ] 

Jun Rao commented on KAFKA-990:
---

Thanks for patch v2. A few more comments.

2.1 I think it's better to guard this in the command line. The issue is that if 
a user provided both options, it's not clear which one takes precedence.
2.2 In that case, we should make sure that brokerList is a mandatory field 
(like zkConnect).

30. KafkaController.initializeAndMaybeTriggerPartitionReassignment(): The 
following comment is weird.
// need to call method

31. Related to Swapnil's comment in #11, currently, the tool finishes after the 
ZK path is created. It would be useful to add an option to check the state of 
partition reassignment so that we know either all assignments have completed or 
the set of partitions that are remaining.


> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sriram Subramanian
>Assignee: Sriram Subramanian
> Attachments: KAFKA-990-v1.patch, KAFKA-990-v1-rebased.patch, 
> KAFKA-990-v2.patch
>
>
> 1. The tool does not register for IsrChangeListener on controller failover.
> 2. There is a race condition where the previous listener can fire on 
> controller failover and the replicas can be in ISR. Even after re-registering 
> the ISR listener after failover, it will never be triggered.
> 3. The input the tool is a static list which is very hard to use. To improve 
> this, as a first step the tool needs to take a list of topics and list of 
> brokers to do the assignment to and then generate the reassignment plan.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1009) DumpLogSegments tool should return error on non-existing files

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1009:
-

Attachment: KAFKA-1009.v2.patch

Thanks for the comments Neha.

1. We do set default mutable to true, what I said is that we add another 
function that do accept the mutable parameter and pass it instead of always 
using the default.

2. Rebased on trunk for v2.

> DumpLogSegments tool should return error on non-existing files
> --
>
> Key: KAFKA-1009
> URL: https://issues.apache.org/jira/browse/KAFKA-1009
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Minor
> Fix For: 0.8.1
>
> Attachments: KAFKA-1009.v1.patch, KAFKA-1009.v2.patch
>
>
> If we run the tool on an non-existing file, we get the following
> bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00.log
> Dumping 00.log
> Starting offset: 0
> The tool should return an error message instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-990) Fix ReassignPartitionCommand and improve usability

[Sriram Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13743944#comment-13743944
 ] 

Sriram Subramanian commented on KAFKA-990:
--

2.1 will do so.
2.2 We cannot make it mandatory. It is not required when explicit list is 
specified. In the case when only topics are specified we do make it mandatory.

31. There is already a tool for that. It is called CheckReassignmentStatus.

> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sriram Subramanian
>Assignee: Sriram Subramanian
> Attachments: KAFKA-990-v1.patch, KAFKA-990-v1-rebased.patch, 
> KAFKA-990-v2.patch
>
>
> 1. The tool does not register for IsrChangeListener on controller failover.
> 2. There is a race condition where the previous listener can fire on 
> controller failover and the replicas can be in ISR. Even after re-registering 
> the ISR listener after failover, it will never be triggered.
> 3. The input the tool is a static list which is very hard to use. To improve 
> this, as a first step the tool needs to take a list of topics and list of 
> brokers to do the assignment to and then generate the reassignment plan.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-990) Fix ReassignPartitionCommand and improve usability

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13743970#comment-13743970
 ] 

Guozhang Wang commented on KAFKA-990:
-

Just one more comment: as for Swapnil's 16, currently partition id 0 is also 
used for AddPartitionCommand. Shall we change that also?

> Fix ReassignPartitionCommand and improve usability
> --
>
> Key: KAFKA-990
> URL: https://issues.apache.org/jira/browse/KAFKA-990
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sriram Subramanian
>Assignee: Sriram Subramanian
> Attachments: KAFKA-990-v1.patch, KAFKA-990-v1-rebased.patch, 
> KAFKA-990-v2.patch
>
>
> 1. The tool does not register for IsrChangeListener on controller failover.
> 2. There is a race condition where the previous listener can fire on 
> controller failover and the replicas can be in ISR. Even after re-registering 
> the ISR listener after failover, it will never be triggered.
> 3. The input the tool is a static list which is very hard to use. To improve 
> this, as a first step the tool needs to take a list of topics and list of 
> brokers to do the assignment to and then generate the reassignment plan.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1009) DumpLogSegments tool should return error on non-existing files

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1009:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk

> DumpLogSegments tool should return error on non-existing files
> --
>
> Key: KAFKA-1009
> URL: https://issues.apache.org/jira/browse/KAFKA-1009
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Minor
> Fix For: 0.8.1
>
> Attachments: KAFKA-1009.v1.patch, KAFKA-1009.v2.patch
>
>
> If we run the tool on an non-existing file, we get the following
> bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00.log
> Dumping 00.log
> Starting offset: 0
> The tool should return an error message instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-1009) DumpLogSegments tool should return error on non-existing files

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13743981#comment-13743981
 ] 

Neha Narkhede commented on KAFKA-1009:
--

1. I see what you are saying. 

The patch looks good. Thanks for fixing the bug. +1

> DumpLogSegments tool should return error on non-existing files
> --
>
> Key: KAFKA-1009
> URL: https://issues.apache.org/jira/browse/KAFKA-1009
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Minor
> Fix For: 0.8.1
>
> Attachments: KAFKA-1009.v1.patch, KAFKA-1009.v2.patch
>
>
> If we run the tool on an non-existing file, we get the following
> bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00.log
> Dumping 00.log
> Starting offset: 0
> The tool should return an error message instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1009) DumpLogSegments tool should return error on non-existing files

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1009:
-

Component/s: (was: core)
 log

> DumpLogSegments tool should return error on non-existing files
> --
>
> Key: KAFKA-1009
> URL: https://issues.apache.org/jira/browse/KAFKA-1009
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Minor
> Fix For: 0.8.1
>
> Attachments: KAFKA-1009.v1.patch, KAFKA-1009.v2.patch
>
>
> If we run the tool on an non-existing file, we get the following
> bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00.log
> Dumping 00.log
> Starting offset: 0
> The tool should return an error message instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-1009) DumpLogSegments tool should return error on non-existing files

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13743981#comment-13743981
 ] 

Neha Narkhede commented on KAFKA-1009:
--

1. I see what you are saying. 

The patch looks good. Thanks for fixing the bug. +1

> DumpLogSegments tool should return error on non-existing files
> --
>
> Key: KAFKA-1009
> URL: https://issues.apache.org/jira/browse/KAFKA-1009
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8
>Reporter: Jun Rao
>Assignee: Guozhang Wang
>Priority: Minor
> Fix For: 0.8.1
>
> Attachments: KAFKA-1009.v1.patch, KAFKA-1009.v2.patch
>
>
> If we run the tool on an non-existing file, we get the following
> bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 00.log
> Dumping 00.log
> Starting offset: 0
> The tool should return an error message instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

review board

[Jay Kreps]

I would like to get review board set up for Kafka. We have gone around and
around on different tools, but never actually did anything. I find the JIRA
bullet list style of code review pretty unusable. Any objections?

-Jay

Re: review board

[Neha Narkhede]

I personally like the phabricator tool for code reviews and accepting
patches - http://phabricator.org/
Some big top level Apache projects have adopted phabricator as well -
Hiveand
HBase 

If we can't do this for some reason, I don't mind review board which is
better than what we do today.

Thanks,
Neha


On Mon, Aug 19, 2013 at 10:46 AM, Jay Kreps  wrote:

> I would like to get review board set up for Kafka. We have gone around and
> around on different tools, but never actually did anything. I find the JIRA
> bullet list style of code review pretty unusable. Any objections?
>
> -Jay
>

[jira] Subscription: outstanding kafka patches

[jira]

Issue Subscription
Filter: outstanding kafka patches (69 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1008  Unmap before resizing
https://issues.apache.org/jira/browse/KAFKA-1008
KAFKA-1005  kafka.perf.ConsumerPerformance not shutting down consumer
https://issues.apache.org/jira/browse/KAFKA-1005
KAFKA-1004  Handle topic event for trivial whitelist topic filters
https://issues.apache.org/jira/browse/KAFKA-1004
KAFKA-998   Producer should not retry on non-recoverable error codes
https://issues.apache.org/jira/browse/KAFKA-998
KAFKA-997   Provide a strict verification mode when reading configuration 
properties
https://issues.apache.org/jira/browse/KAFKA-997
KAFKA-996   Capitalize first letter for log entries
https://issues.apache.org/jira/browse/KAFKA-996
KAFKA-995   Enforce that the value for replica.fetch.max.bytes is always >= the 
value for message.max.bytes
https://issues.apache.org/jira/browse/KAFKA-995
KAFKA-990   Fix ReassignPartitionCommand and improve usability
https://issues.apache.org/jira/browse/KAFKA-990
KAFKA-984   Avoid a full rebalance in cases when a new topic is discovered but 
container/broker set stay the same
https://issues.apache.org/jira/browse/KAFKA-984
KAFKA-982   Logo for Kafka
https://issues.apache.org/jira/browse/KAFKA-982
KAFKA-981   Unable to pull Kafka binaries with Ivy
https://issues.apache.org/jira/browse/KAFKA-981
KAFKA-976   Order-Preserving Mirror Maker Testcase
https://issues.apache.org/jira/browse/KAFKA-976
KAFKA-974   can't use public release maven repo because of failure of 
downloaded dependency
https://issues.apache.org/jira/browse/KAFKA-974
KAFKA-967   Use key range in ProducerPerformance
https://issues.apache.org/jira/browse/KAFKA-967
KAFKA-956   High-level consumer fails to check topic metadata response for 
errors
https://issues.apache.org/jira/browse/KAFKA-956
KAFKA-955   After a leader change, messages sent with ack=0 are lost
https://issues.apache.org/jira/browse/KAFKA-955
KAFKA-946   Kafka Hadoop Consumer fails when verifying message checksum
https://issues.apache.org/jira/browse/KAFKA-946
KAFKA-923   Improve controller failover latency
https://issues.apache.org/jira/browse/KAFKA-923
KAFKA-917   Expose zk.session.timeout.ms in console consumer
https://issues.apache.org/jira/browse/KAFKA-917
KAFKA-885   sbt package builds two kafka jars
https://issues.apache.org/jira/browse/KAFKA-885
KAFKA-881   Kafka broker not respecting log.roll.hours
https://issues.apache.org/jira/browse/KAFKA-881
KAFKA-873   Consider replacing zkclient with curator (with zkclient-bridge)
https://issues.apache.org/jira/browse/KAFKA-873
KAFKA-868   System Test - add test case for rolling controlled shutdown
https://issues.apache.org/jira/browse/KAFKA-868
KAFKA-863   System Test - update 0.7 version of kafka-run-class.sh for 
Migration Tool test cases
https://issues.apache.org/jira/browse/KAFKA-863
KAFKA-859   support basic auth protection of mx4j console
https://issues.apache.org/jira/browse/KAFKA-859
KAFKA-855   Ant+Ivy build for Kafka
https://issues.apache.org/jira/browse/KAFKA-855
KAFKA-854   Upgrade dependencies for 0.8
https://issues.apache.org/jira/browse/KAFKA-854
KAFKA-815   Improve SimpleConsumerShell to take in a max messages config option
https://issues.apache.org/jira/browse/KAFKA-815
KAFKA-745   Remove getShutdownReceive() and other kafka specific code from the 
RequestChannel
https://issues.apache.org/jira/browse/KAFKA-745
KAFKA-735   Add looping and JSON output for ConsumerOffsetChecker
https://issues.apache.org/jira/browse/KAFKA-735
KAFKA-717   scala 2.10 build support
https://issues.apache.org/jira/browse/KAFKA-717
KAFKA-686   0.8 Kafka broker should give a better error message when running 
against 0.7 zookeeper
https://issues.apache.org/jira/browse/KAFKA-686
KAFKA-677   Retention process gives exception if an empty segment is chosen for 
collection
https://issues.apache.org/jira/browse/KAFKA-677
KAFKA-674   Clean Shutdown Testing - Log segments checksums mismatch
https://issues.apache.org/jira/browse/KAFKA-674
KAFKA-652   Create testcases for clean shut-down
https://issues.apache.org/jira/browse/KAFKA-652
KAFKA-649   Cleanup log4j logging
https://issues.apache.org/jira/browse/KAFKA-649
KAFKA-645   Create a shell script to run System Test with DEBUG details and 
"tee" console output to a file
https://issues.apache.org/jira/browse/KAFKA-645
KAFKA-598   decouple fetch size from max message size
https://issues.apache.org/jira/browse/KAFKA-598
KAFKA-583   Simple

[jira] [Updated] (KAFKA-1000) Inbuilt consumer offset management

[Tejas Patil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tejas Patil updated KAFKA-1000:
---

Description: 
This Jira is for tracking the phase #2 of Offset Management [0]. After 
KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets 
from brokers. Currently, consumers are not using this API and directly talking 
with Zookeeper. 
Joel and I have been working on this.

[0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
[1] : https://issues.apache.org/jira/browse/KAFKA-657

  was:
This Jitra corresponds to the phase #2 of Offset Management [0]. After 
KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets 
from brokers. Currently, consumers are not using this API and directly talking 
with Zookeeper. 

This Jira will involve following:
1. Consumers should now use the offset commit API
2. Add a special topic in kafka for storing offsets
3. Pluggable offset commit logic so that users can pick the existing ZK based 
storage or inbuilt storage for offsets.
4. Leader brokers would now maintain an additional hash table of offsets for 
the group-topic-partitions that they lead
5. Consumers send their offset fetch requests to brokers. Receiving broker may 
or may not be the leader for the requested group-topic-partitions so would have 
to forward the request to respective leader brokers.

Joel and I have been working on this and I would share the initial patch and a 
writeup covering the nitty-gritty of the implementation.

[0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
[1] : https://issues.apache.org/jira/browse/KAFKA-657


> Inbuilt consumer offset management
> --
>
> Key: KAFKA-1000
> URL: https://issues.apache.org/jira/browse/KAFKA-1000
> Project: Kafka
>  Issue Type: New Feature
>  Components: consumer
>Affects Versions: 0.8.1
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
>  Labels: features
>
> This Jira is for tracking the phase #2 of Offset Management [0]. After 
> KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets 
> from brokers. Currently, consumers are not using this API and directly 
> talking with Zookeeper. 
> Joel and I have been working on this.
> [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> [1] : https://issues.apache.org/jira/browse/KAFKA-657

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1000) Inbuilt consumer offset management feature for kakfa

[Tejas Patil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tejas Patil updated KAFKA-1000:
---

Summary: Inbuilt consumer offset management feature for kakfa  (was: 
Inbuilt consumer offset management)

> Inbuilt consumer offset management feature for kakfa
> 
>
> Key: KAFKA-1000
> URL: https://issues.apache.org/jira/browse/KAFKA-1000
> Project: Kafka
>  Issue Type: New Feature
>  Components: consumer
>Affects Versions: 0.8.1
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
>  Labels: features
>
> This Jira is for tracking the phase #2 of Offset Management [0]. After 
> KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets 
> from brokers. Currently, consumers are not using this API and directly 
> talking with Zookeeper. 
> Joel and I have been working on this.
> [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> [1] : https://issues.apache.org/jira/browse/KAFKA-657

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1000) Inbuilt consumer offset management feature for kakfa

[Tejas Patil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tejas Patil updated KAFKA-1000:
---

Description: 
This Jira is for tracking the phase #2 of Offset Management [0]. Joel and I 
have been working on this. [1] is the overall design of the feature.

[0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
[1] : 
https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management


  was:
This Jira is for tracking the phase #2 of Offset Management [0]. After 
KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets 
from brokers. Currently, consumers are not using this API and directly talking 
with Zookeeper. 
Joel and I have been working on this.

[0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
[1] : https://issues.apache.org/jira/browse/KAFKA-657


> Inbuilt consumer offset management feature for kakfa
> 
>
> Key: KAFKA-1000
> URL: https://issues.apache.org/jira/browse/KAFKA-1000
> Project: Kafka
>  Issue Type: New Feature
>  Components: consumer
>Affects Versions: 0.8.1
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
>  Labels: features
>
> This Jira is for tracking the phase #2 of Offset Management [0]. Joel and I 
> have been working on this. [1] is the overall design of the feature.
> [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> [1] : 
> https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

[Tejas Patil (JIRA)]

Tejas Patil created KAFKA-1012:
--

 Summary: Implement an Offset Manager and hook offset requests to it
 Key: KAFKA-1012
 URL: https://issues.apache.org/jira/browse/KAFKA-1012
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Reporter: Tejas Patil
Assignee: Tejas Patil
Priority: Minor


After KAFKA-657, we have a protocol for consumers to commit and fetch offsets 
from brokers. Currently, consumers are not using this API and directly talking 
with Zookeeper. 

This Jira will involve following:
1. Add a special topic in kafka for storing offsets
2. Add an OffsetManager interface which would handle storing, accessing, 
loading and maintaining consumer offsets
3. Implement offset managers for both of these 2 choices : existing ZK based 
storage or inbuilt storage for offsets.
4. Leader brokers would now maintain an additional hash table of offsets for 
the group-topic-partitions that they lead
5. Consumers should now use the OffsetCommit and OffsetFetch API

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000

[Tejas Patil (JIRA)]

Tejas Patil created KAFKA-1013:
--

 Summary: Modify existing tools as per the changes in KAFKA-1000
 Key: KAFKA-1013
 URL: https://issues.apache.org/jira/browse/KAFKA-1013
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Reporter: Tejas Patil
Assignee: Tejas Patil
Priority: Minor


Modify existing tools as per the changes in KAFKA-1000. AFAIK, the tools below 
would be affected:
- ConsumerOffsetChecker
- ExportZkOffsets
- ImportZkOffsets
- UpdateOffsetsInZK


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (KAFKA-1014) Junits and system tests for inbuilt offset management

[Tejas Patil (JIRA)]

Tejas Patil created KAFKA-1014:
--

 Summary: Junits and system tests for inbuilt offset management
 Key: KAFKA-1014
 URL: https://issues.apache.org/jira/browse/KAFKA-1014
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Reporter: Tejas Patil
Assignee: Tejas Patil
Priority: Minor


This Jira is for 2 things:
1. Fix tests (junits & system tests) which fail due to inbuilt offset 
management feature.
2. Additional tests must be added which includes broker loading logs on 
leadership change, perform cleanup when a leader becomes follower for offsets 
topic, and other such possible scenarios.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (KAFKA-1015) documentation for inbuilt offset management

[Tejas Patil (JIRA)]

Tejas Patil created KAFKA-1015:
--

 Summary: documentation for inbuilt offset management
 Key: KAFKA-1015
 URL: https://issues.apache.org/jira/browse/KAFKA-1015
 Project: Kafka
  Issue Type: Sub-task
  Components: consumer
Reporter: Tejas Patil
Assignee: Tejas Patil
Priority: Minor


Add documentation for inbuilt offset management and update existing documents 
if needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: review board

[Joe Stein]

Phabricator looks pretty awesome but agree with Neha and Jay that review board 
is definitely better than what is in place now.


/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
/


On Aug 19, 2013, at 2:58 PM, Neha Narkhede  wrote:

> I personally like the phabricator tool for code reviews and accepting
> patches - http://phabricator.org/
> Some big top level Apache projects have adopted phabricator as well -
> Hiveand
> HBase 
> 
> If we can't do this for some reason, I don't mind review board which is
> better than what we do today.
> 
> Thanks,
> Neha
> 
> 
> On Mon, Aug 19, 2013 at 10:46 AM, Jay Kreps  wrote:
> 
>> I would like to get review board set up for Kafka. We have gone around and
>> around on different tools, but never actually did anything. I find the JIRA
>> bullet list style of code review pretty unusable. Any objections?
>> 
>> -Jay
>> 

[jira] [Comment Edited] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

[Tejas Patil (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744186#comment-13744186
 ] 

Tejas Patil edited comment on KAFKA-1012 at 8/19/13 8:38 PM:
-

Attached KAFKA-1012-v1.patch for trunk. An overview of the implementation can 
be found at [0]. Would be happy to hear any suggestions / concerns wrt the 
design and implementation.

Things that are planned but not included in this patch

1. Use a binary format for the offsets topic messages and incorporate 
versioning to support changes over time if required.
2. Cleanup of offsets table when there is a change in ownership of any 
partition of the offsets' topic.
3. Apart from the normal use case of clearing off the unwanted offsets from 
offsets table, support is needed to cleanup all entries specific to a topic or 
group and also based on their age. (This could be even moved to KAFKA-559)

Known limitations / caveats

1. Re-assign partitions: We are assuming that there is no reassignment of the 
offsets' topic partitions
2. Atomicity of offset commit is not currently implemented since we are using a 
concurrent hashmap.

PS: If you plan to run junits, use the v1 patch in KAFKA-1014 on top of this 
one.

[0] : 
https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management

  was (Author: tejasp):
Attached KAFKA-1012-v1.patch for trunk. An overview of the implementation 
can be found at [0]. Would be happy to hear any suggestions / concerns wrt the 
design and implementation.

Things that are planned but not included in this patch
--
1. Use a binary format for the offsets topic messages and incorporate 
versioning to support changes over time if required.
2. Cleanup of offsets table when there is a change in ownership of any 
partition of the offsets' topic.
3. Apart from the normal use case of clearing off the unwanted offsets from 
offsets table, support is needed to cleanup all entries specific to a topic or 
group and also based on their age. (This could be even moved to KAFKA-559)

Known limitations / caveats
-
1. Re-assign partitions: We are assuming that there is no reassignment of the 
offsets' topic partitions
2. Atomicity of offset commit is not currently implemented since we are using a 
concurrent hashmap.

PS: If you plan to run junits, use the v1 patch in KAFKA-1014 on top of this 
one.

[0] : 
https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
  
> Implement an Offset Manager and hook offset requests to it
> --
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
> Attachments: KAFKA-1012.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets 
> from brokers. Currently, consumers are not using this API and directly 
> talking with Zookeeper. 
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing, 
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based 
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for 
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

[Tejas Patil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tejas Patil updated KAFKA-1012:
---

Attachment: KAFKA-1012.patch

Attached KAFKA-1012-v1.patch for trunk. An overview of the implementation can 
be found at [0]. Would be happy to hear any suggestions / concerns wrt the 
design and implementation.

Things that are planned but not included in this patch
--
1. Use a binary format for the offsets topic messages and incorporate 
versioning to support changes over time if required.
2. Cleanup of offsets table when there is a change in ownership of any 
partition of the offsets' topic.
3. Apart from the normal use case of clearing off the unwanted offsets from 
offsets table, support is needed to cleanup all entries specific to a topic or 
group and also based on their age. (This could be even moved to KAFKA-559)

Known limitations / caveats
-
1. Re-assign partitions: We are assuming that there is no reassignment of the 
offsets' topic partitions
2. Atomicity of offset commit is not currently implemented since we are using a 
concurrent hashmap.

PS: If you plan to run junits, use the v1 patch in KAFKA-1014 on top of this 
one.

[0] : 
https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management

> Implement an Offset Manager and hook offset requests to it
> --
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
> Attachments: KAFKA-1012.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets 
> from brokers. Currently, consumers are not using this API and directly 
> talking with Zookeeper. 
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing, 
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based 
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for 
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: review board

[Jay Kreps]

Personally I would prefer review board as we are using it for Samza and
also at LinkedIn and switching between tools drives me nuts. Does
phabricator have any particularly good features?

If someone else is willing to do the leg work, though, I will take whatever
they set up.

-Jay


On Mon, Aug 19, 2013 at 11:58 AM, Neha Narkhede wrote:

> I personally like the phabricator tool for code reviews and accepting
> patches - http://phabricator.org/
> Some big top level Apache projects have adopted phabricator as well -
> Hiveand
> HBase 
>
> If we can't do this for some reason, I don't mind review board which is
> better than what we do today.
>
> Thanks,
> Neha
>
>
> On Mon, Aug 19, 2013 at 10:46 AM, Jay Kreps  wrote:
>
> > I would like to get review board set up for Kafka. We have gone around
> and
> > around on different tools, but never actually did anything. I find the
> JIRA
> > bullet list style of code review pretty unusable. Any objections?
> >
> > -Jay
> >
>

[jira] [Created] (KAFKA-1016) Broker should limit purgatory size

[Chris Riccomini (JIRA)]

Chris Riccomini created KAFKA-1016:
--

 Summary: Broker should limit purgatory size
 Key: KAFKA-1016
 URL: https://issues.apache.org/jira/browse/KAFKA-1016
 Project: Kafka
  Issue Type: Bug
  Components: purgatory
Affects Versions: 0.8
Reporter: Chris Riccomini
Assignee: Joel Koshy


I recently ran into a case where a poorly configured Kafka consumer was able to 
trigger out of memory exceptions in multiple Kafka brokers. The consumer was 
configured to have a fetcher.max.wait of Int.MaxInt.

For low volume topics, this configuration causes the consumer to block for 
frequently, and for long periods of time. [~junrao] informs me that the fetch 
request will time out after the socket timeout is reached. In our case, this 
was set to 30s.

With several thousand consumer threads, the fetch request purgatory got into 
the 100,000-400,000 range, which we believe triggered the out of memory 
exception. [~nehanarkhede] claims to have seem similar behavior in other high 
volume clusters.

It kind of seems like a bad thing that a poorly configured consumer can trigger 
out of memory exceptions in the broker. I was thinking maybe it makes sense to 
have the broker try and protect itself from this situation. Here are some 
potential solutions:

1. Have a broker-side max wait config for fetch requests.
2. Threshold the purgatory size, and either drop the oldest connections in 
purgatory, or reject the newest fetch requests when purgatory is full.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

[Tejas Patil (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744186#comment-13744186
 ] 

Tejas Patil edited comment on KAFKA-1012 at 8/19/13 9:20 PM:
-

Attached KAFKA-1012-v2.patch for trunk. An overview of the implementation can 
be found at [0]. Would be happy to hear any suggestions / concerns wrt the 
design and implementation.

Things that are planned but not included in this patch

1. Use a binary format for the offsets topic messages and incorporate 
versioning to support changes over time if required.
2. Cleanup of offsets table when there is a change in ownership of any 
partition of the offsets' topic.
3. Apart from the normal use case of clearing off the unwanted offsets from 
offsets table, support is needed to cleanup all entries specific to a topic or 
group and also based on their age. (This could be even moved to KAFKA-559)
4.  With this patch inbuilt offset management is turned on by default so junits 
run over the same. "kafka.server.OffsetCommitTest" fails with this patch which 
needs to be fixed.

Known limitations / caveats

1. Re-assign partitions: We are assuming that there is no reassignment of the 
offsets' topic partitions
2. Atomicity of offset commit is not currently implemented since we are using a 
concurrent hashmap.

[0] : 
https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management

  was (Author: tejasp):
Attached KAFKA-1012-v1.patch for trunk. An overview of the implementation 
can be found at [0]. Would be happy to hear any suggestions / concerns wrt the 
design and implementation.

Things that are planned but not included in this patch

1. Use a binary format for the offsets topic messages and incorporate 
versioning to support changes over time if required.
2. Cleanup of offsets table when there is a change in ownership of any 
partition of the offsets' topic.
3. Apart from the normal use case of clearing off the unwanted offsets from 
offsets table, support is needed to cleanup all entries specific to a topic or 
group and also based on their age. (This could be even moved to KAFKA-559)

Known limitations / caveats

1. Re-assign partitions: We are assuming that there is no reassignment of the 
offsets' topic partitions
2. Atomicity of offset commit is not currently implemented since we are using a 
concurrent hashmap.

PS: If you plan to run junits, use the v1 patch in KAFKA-1014 on top of this 
one.

[0] : 
https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
  
> Implement an Offset Manager and hook offset requests to it
> --
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
> Attachments: KAFKA-1012.patch, KAFKA-1012-v2.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets 
> from brokers. Currently, consumers are not using this API and directly 
> talking with Zookeeper. 
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing, 
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based 
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for 
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

[Tejas Patil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tejas Patil updated KAFKA-1012:
---

Attachment: KAFKA-1012-v2.patch

> Implement an Offset Manager and hook offset requests to it
> --
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
> Attachments: KAFKA-1012.patch, KAFKA-1012-v2.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets 
> from brokers. Currently, consumers are not using this API and directly 
> talking with Zookeeper. 
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing, 
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based 
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for 
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: review board

[Joe Stein]

Googling a bit there seems to be little difference between them ( seems just a 
matter of preference).  Phabrictor was created at Facebook and suspect the 
additional committers from FB to Hbase and as Hive started there led to them 
switching ( since I bet they don't want to context switch either).  

review board works well


/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
/


On Aug 19, 2013, at 4:46 PM, Jay Kreps  wrote:

> Personally I would prefer review board as we are using it for Samza and
> also at LinkedIn and switching between tools drives me nuts. Does
> phabricator have any particularly good features?
> 
> If someone else is willing to do the leg work, though, I will take whatever
> they set up.
> 
> -Jay
> 
> 
> On Mon, Aug 19, 2013 at 11:58 AM, Neha Narkhede 
> wrote:
> 
>> I personally like the phabricator tool for code reviews and accepting
>> patches - http://phabricator.org/
>> Some big top level Apache projects have adopted phabricator as well -
>> Hiveand
>> HBase 
>> 
>> If we can't do this for some reason, I don't mind review board which is
>> better than what we do today.
>> 
>> Thanks,
>> Neha
>> 
>> 
>> On Mon, Aug 19, 2013 at 10:46 AM, Jay Kreps  wrote:
>> 
>>> I would like to get review board set up for Kafka. We have gone around
>> and
>>> around on different tools, but never actually did anything. I find the
>> JIRA
>>> bullet list style of code review pretty unusable. Any objections?
>>> 
>>> -Jay
>>> 
>> 

[jira] [Created] (KAFKA-1017) High number of open file handles in 0.8 producer

[Swapnil Ghike (JIRA)]

Swapnil Ghike created KAFKA-1017:


 Summary: High number of open file handles in 0.8 producer
 Key: KAFKA-1017
 URL: https://issues.apache.org/jira/browse/KAFKA-1017
 Project: Kafka
  Issue Type: Bug
Reporter: Swapnil Ghike
Assignee: Swapnil Ghike


For over-partitioned topics, each broker could be the leader for at least 1 
partition. In the producer, we randomly select a partition to send the data. 
Pretty soon, each producer will establish a connection to each of the n 
brokers. Effectively, we increased the # of socket connections by a factor of 
n, compared to 0.7.

The increased number of socket connections increases the number of open file 
handles, this could come pretty  close to the OS limit if left unnoticed. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1017) High number of open file handles in 0.8 producer

[Swapnil Ghike (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Ghike updated KAFKA-1017:
-

Description: 
Reported by Jun Rao:

For over-partitioned topics, each broker could be the leader for at least 1 
partition. In the producer, we randomly select a partition to send the data. 
Pretty soon, each producer will establish a connection to each of the n 
brokers. Effectively, we increased the # of socket connections by a factor of 
n, compared to 0.7.

The increased number of socket connections increases the number of open file 
handles, this could come pretty  close to the OS limit if left unnoticed. 

  was:
For over-partitioned topics, each broker could be the leader for at least 1 
partition. In the producer, we randomly select a partition to send the data. 
Pretty soon, each producer will establish a connection to each of the n 
brokers. Effectively, we increased the # of socket connections by a factor of 
n, compared to 0.7.

The increased number of socket connections increases the number of open file 
handles, this could come pretty  close to the OS limit if left unnoticed. 


> High number of open file handles in 0.8 producer
> 
>
> Key: KAFKA-1017
> URL: https://issues.apache.org/jira/browse/KAFKA-1017
> Project: Kafka
>  Issue Type: Bug
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
>
> Reported by Jun Rao:
> For over-partitioned topics, each broker could be the leader for at least 1 
> partition. In the producer, we randomly select a partition to send the data. 
> Pretty soon, each producer will establish a connection to each of the n 
> brokers. Effectively, we increased the # of socket connections by a factor of 
> n, compared to 0.7.
> The increased number of socket connections increases the number of open file 
> handles, this could come pretty  close to the OS limit if left unnoticed. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1017) High number of open file handles in 0.8 producer

[Swapnil Ghike (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Ghike updated KAFKA-1017:
-

Description: 
Reported by Jun Rao:

For over-partitioned topics, each broker could be the leader for at least 1 
partition. In the producer, we randomly select a partition to send the data. 
Pretty soon, each producer will establish a connection to each of the n 
brokers. Effectively, we increased the # of socket connections by a factor of 
n, compared to 0.7.

The increased number of socket connections increases the number of open file 
handles, this could come pretty  close to the OS limit.

  was:
Reported by Jun Rao:

For over-partitioned topics, each broker could be the leader for at least 1 
partition. In the producer, we randomly select a partition to send the data. 
Pretty soon, each producer will establish a connection to each of the n 
brokers. Effectively, we increased the # of socket connections by a factor of 
n, compared to 0.7.

The increased number of socket connections increases the number of open file 
handles, this could come pretty  close to the OS limit if left unnoticed. 


> High number of open file handles in 0.8 producer
> 
>
> Key: KAFKA-1017
> URL: https://issues.apache.org/jira/browse/KAFKA-1017
> Project: Kafka
>  Issue Type: Bug
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
>
> Reported by Jun Rao:
> For over-partitioned topics, each broker could be the leader for at least 1 
> partition. In the producer, we randomly select a partition to send the data. 
> Pretty soon, each producer will establish a connection to each of the n 
> brokers. Effectively, we increased the # of socket connections by a factor of 
> n, compared to 0.7.
> The increased number of socket connections increases the number of open file 
> handles, this could come pretty  close to the OS limit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1017) High number of open file handles in 0.8 producer

[Swapnil Ghike (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Ghike updated KAFKA-1017:
-

Attachment: kafka-1017.patch

This patch fixes two issues:

1. Reduce the number of open file handles on the producer: I think we can clear 
the topic-partition send-cache on the EventHandler once in every refresh 
metadata interval. This will make the producer produce to the same partition 
for say, 15 mins, for a given topic. That should be ok. The topic-partition 
send-cache will be refreshed when the metadata is refreshed.

2. There is a race condition in Producer.send() and Producer.close(). This can 
lead to reopening of a closed ProducerPool and thereby socket leaks.

> High number of open file handles in 0.8 producer
> 
>
> Key: KAFKA-1017
> URL: https://issues.apache.org/jira/browse/KAFKA-1017
> Project: Kafka
>  Issue Type: Bug
>Reporter: Swapnil Ghike
>Assignee: Swapnil Ghike
> Attachments: kafka-1017.patch
>
>
> Reported by Jun Rao:
> For over-partitioned topics, each broker could be the leader for at least 1 
> partition. In the producer, we randomly select a partition to send the data. 
> Pretty soon, each producer will establish a connection to each of the n 
> brokers. Effectively, we increased the # of socket connections by a factor of 
> n, compared to 0.7.
> The increased number of socket connections increases the number of open file 
> handles, this could come pretty  close to the OS limit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

[Tejas Patil (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744718#comment-13744718
 ] 

Tejas Patil commented on KAFKA-1012:


(1) Currently, "KafkaApis" has to figure out the leader of offsets topic and 
perform forwarding of the OffsetFetchRequest. From an implementation 
perspective, "OffsetManager" must abstract out those things. Also, another bad 
thing is that in case of Zk based offset manager, offset fetch request may be 
get redirected even if not required. Both these could be resolved by moving a 
things to "OffsetManager" but as there is strong coupling (leaderCache, 
offsetFetchRequest, aliveBrokers) the entire change might make things look 
hideous. 

(2) Is it necessary to have "offset topic" and save the offsets to logs while 
using Zk based offset manager ? Technically if a user just wants to continue 
using Zk based offset manager, having "offsets topic" would be an overhead. On 
the other hand, having logs for the "offsets topic" may end up giving room for 
smooth transition across the 2 offset manager implementations. 

(3) There is no purgatory maintained for Offset fetch request as fetch requests 
would end up querying Zk or reading from in-memory table of broker (with at max 
one extra hop). This does not involve any disk access and is expected to be 
fast. It could be argued that a busy broker might not be able to respond to the 
request making the request to time out. The request handler threads of the 
broker, which forwards the request, might not process any requests in the mean 
time. If the current implementation seems spooky in this aspect, some 
alternatives would be: add purgatory for offset fetch request (seems overkill 
to me) OR use a thick client which would directly contact the leader directly. 

Any suggestions for the items above would be highly appreciated. 

> Implement an Offset Manager and hook offset requests to it
> --
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
> Attachments: KAFKA-1012.patch, KAFKA-1012-v2.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets 
> from brokers. Currently, consumers are not using this API and directly 
> talking with Zookeeper. 
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing, 
> loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based 
> storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for 
> the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-1014) Junits and system tests for inbuilt offset management

[Tejas Patil (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744719#comment-13744719
 ] 

Tejas Patil commented on KAFKA-1014:


Currently system tests are not on trunk so would wait for that to happen. 
Meanwhile, working on some junits.

> Junits and system tests for inbuilt offset management
> -
>
> Key: KAFKA-1014
> URL: https://issues.apache.org/jira/browse/KAFKA-1014
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Tejas Patil
>Assignee: Tejas Patil
>Priority: Minor
>
> This Jira is for 2 things:
> 1. Fix tests (junits & system tests) which fail due to inbuilt offset 
> management feature.
> 2. Additional tests must be added which includes broker loading logs on 
> leadership change, perform cleanup when a leader becomes follower for offsets 
> topic, and other such possible scenarios.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira