[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[JIRA]

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723259#comment-14723259
 ] 

Håkon Hitland commented on KAFKA-2477:
--

"kafka.controller:name=UncleanLeaderElectionsPerSec,type=ControllerStats" has 
Count = 0 on all brokers.

There is nothing in any controller.log from that day except for one broker 
checking leader imbalance.
([2015-08-24 00:02:32,650] TRACE [Controller 0]: checking need to trigger 
partition rebalance (kafka.controller.KafkaController)
[2015-08-24 00:02:32,684] DEBUG [Controller 0]: preferred replicas by broker 
Map([...]) (kafka.controller.KafkaController)
[2015-08-24 00:02:32,684] DEBUG [Controller 0]: topics not in preferred replica 
Map() (kafka.controller.KafkaController)
[2015-08-24 00:02:32,685] TRACE [Controller 0]: leader imbalance ratio for 
broker 2 is 0.00 (kafka.controller.KafkaController)
, etc.)

> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
> Attachments: kafka_log.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2145) An option to add topic owners.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723515#comment-14723515
 ] 

Jun Rao commented on KAFKA-2145:


[~parth.brahmbhatt], will take a look today.

> An option to add topic owners. 
> ---
>
> Key: KAFKA-2145
> URL: https://issues.apache.org/jira/browse/KAFKA-2145
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>
> We need to expose a way so users can identify users/groups that share 
> ownership of topic. We discussed adding this as part of 
> https://issues.apache.org/jira/browse/KAFKA-2035 and agreed that it will be 
> simpler to add owner as a logconfig. 
> The owner field can be used for auditing and also by authorization layer to 
> grant access without having to explicitly configure acls. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2486) New consumer performance

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723607#comment-14723607
 ] 

Jason Gustafson commented on KAFKA-2486:


The sleep was intended to implement the fetch backoff as Guozhang mentioned, 
but now that I think about it, the better place for it is inside Fetcher, which 
can track when new fetches need to be sent. I think Jay is right that we 
basically don't want to sleep ever since we could be missing IO.

> New consumer performance
> 
>
> Key: KAFKA-2486
> URL: https://issues.apache.org/jira/browse/KAFKA-2486
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
> Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a 
> recent report on the mailing list indicates it's dropped significantly. After 
> evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, 
> compared to 600+MB/s previously. Before release, we should get the 
> performance back on par.
> Some details about where the regression occurred from the mailing list 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E
>  :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good 
> performance -> an error due to broker apparently not accepting the partition 
> assignment strategy. Since this commit seems to add heartbeats and the server 
> side code for partition assignment strategies, I assume we were missing 
> something on the client side and by filling in the server side, things 
> stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or 
> a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the 
> perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, 
> but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new 
> consumer, even if regressions weren't automatically detected. It would at 
> least allow for manually checking for regressions. This should not be 
> difficult since there are already old consumer performance tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2486) New consumer performance

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723615#comment-14723615
 ] 

Jay Kreps commented on KAFKA-2486:
--

I don't think we need to do anything special for the 0 wait time case. I think 
we can eventually add request count quotas to protect against this kind of 
thing, but for now I think maintaining the behavior of most the other consumers 
of "no backoff means no backoff" is totally fine. Presumably if you set that, 
that is what you want.

For the consumer benchmark case I don't think we need to override the max wait 
to 0, not sure why it was that way before. It should be fine with 500ms or 
whatever.

> New consumer performance
> 
>
> Key: KAFKA-2486
> URL: https://issues.apache.org/jira/browse/KAFKA-2486
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
> Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a 
> recent report on the mailing list indicates it's dropped significantly. After 
> evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, 
> compared to 600+MB/s previously. Before release, we should get the 
> performance back on par.
> Some details about where the regression occurred from the mailing list 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E
>  :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good 
> performance -> an error due to broker apparently not accepting the partition 
> assignment strategy. Since this commit seems to add heartbeats and the server 
> side code for partition assignment strategies, I assume we were missing 
> something on the client side and by filling in the server side, things 
> stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or 
> a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the 
> perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, 
> but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new 
> consumer, even if regressions weren't automatically detected. It would at 
> least allow for manually checking for regressions. This should not be 
> difficult since there are already old consumer performance tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Mirror maker SSL support

[Jun Rao]

Currently, MirrorMaker can run the producer with SSL. We will add the
option to run the new consumer in MirrorMaker in KAFKA-2452, which will
enable SSL on the consumer side too.

Thanks,

Jun

On Fri, Aug 28, 2015 at 12:01 PM, Xiang Zhou (Samuel) 
wrote:

> Hi, folks,
>
> I just want to check if anybody know if there are any plan of Kafka
> MirrorMaker support running over SSL. I tried to follow
> https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
> to configure consumer/producer properties with SSL but no luck, looks that
> Kafka MirrorMaker consumer use the difference implementation when consuming
> data compared to new consumer.
>
> So I want to check when SSL MM will be supported. Or is there a JIRA ticket
> to track it so I can follow it?
>
> Thanks!
> Samuel
>

Re: Mirror maker SSL support

[Xiang Zhou (Samuel)]

Thank you for update Jun.
What's the schedule to get new consumer implemented in Mirror Maker?

Samuel

On Mon, Aug 31, 2015 at 10:06 AM, Jun Rao  wrote:

> Currently, MirrorMaker can run the producer with SSL. We will add the
> option to run the new consumer in MirrorMaker in KAFKA-2452, which will
> enable SSL on the consumer side too.
>
> Thanks,
>
> Jun
>
> On Fri, Aug 28, 2015 at 12:01 PM, Xiang Zhou (Samuel) 
> wrote:
>
> > Hi, folks,
> >
> > I just want to check if anybody know if there are any plan of Kafka
> > MirrorMaker support running over SSL. I tried to follow
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
> > to configure consumer/producer properties with SSL but no luck, looks
> that
> > Kafka MirrorMaker consumer use the difference implementation when
> consuming
> > data compared to new consumer.
> >
> > So I want to check when SSL MM will be supported. Or is there a JIRA
> ticket
> > to track it so I can follow it?
> >
> > Thanks!
> > Samuel
> >
>

Re: Review Request 34492: Patch for KAFKA-2210

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34492/#review96909
---


Thanks for the patch. We are getting pretty close. A few more comments below.


core/src/main/scala/kafka/common/ErrorMapping.scala (line 56)


We will need to add the new error code and the exception in 
org.apache.kafka.common.errors in the client as well. Currently, there are a 
few error codes in Errors that are missing in ErrorMapping. This will be fixed 
in a separate jira. In this jira, we can just start with error code 29, which 
is the next unused error code in Errors. The exception in the client should be 
of ApiException.



core/src/main/scala/kafka/security/auth/Acl.scala (lines 48 - 51)


DENY, READ, WRITE should probably be camel case?



core/src/main/scala/kafka/security/auth/Acl.scala (line 53)


principal => principals



core/src/main/scala/kafka/security/auth/Acl.scala (line 99)


I had a comment on this earlier that the current ACL representation makes 
it a bit hard to do dedup. Also, this makes it a bit inconvenient to remove 
ACLs. Basically, you have to specify the exact set of principals, hosts and 
operations to remove an existing ACL. Your reply is that this makes it 
convenient when an admin wants to add the same permission to say 5 principals.

I was thinking that perhaps we can separate the CLI options from the 
underlying ACL representation. For the ACL representation, it seems that it's 
more natural to represent each ACL as either a  or a  tuple. This will allow ACLs 
to be represented uniquely and makes dedupping easy. On the CLI side, we can 
take batch options for convenience, but translate them to a set of unique ACLs.


- Jun Rao


On Aug. 26, 2015, 9:29 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34492/
> ---
> 
> (Updated Aug. 26, 2015, 9:29 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2210
> https://issues.apache.org/jira/browse/KAFKA-2210
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Addressing review comments from Jun.
> 
> 
> Adding CREATE check for offset topic only if the topic does not exist already.
> 
> 
> Addressing some more comments.
> 
> 
> Removing acl.json file
> 
> 
> Moving PermissionType to trait instead of enum. Following the convention for 
> defining constants.
> 
> 
> Adding authorizer.config.path back.
> 
> 
> Addressing more comments from Jun.
> 
> 
> Addressing more comments.
> 
> 
> Now addressing Ismael's comments. Case sensitive checks.
> 
> 
> Addressing Jun's comments.
> 
> 
> Merge remote-tracking branch 'origin/trunk' into az
> 
> Conflicts:
>   core/src/main/scala/kafka/server/KafkaApis.scala
>   core/src/main/scala/kafka/server/KafkaServer.scala
> 
> Deleting KafkaConfigDefTest
> 
> 
> Addressing comments from Ismael.
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into az
> 
> 
> Consolidating KafkaPrincipal.
> 
> 
> Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into az
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/common/network/PlaintextTransportLayer.java
>   
> clients/src/main/java/org/apache/kafka/common/security/auth/KafkaPrincipal.java
>   core/src/main/scala/kafka/server/KafkaApis.scala
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/common/network/PlaintextTransportLayer.java
>  35d41685dd178bbdf77b2476e03ad51fc4adcbb6 
>   
> clients/src/main/java/org/apache/kafka/common/security/auth/KafkaPrincipal.java
>  b640ea0f4bdb694fc5524ef594aa125cc1ba4cf3 
>   
> clients/src/test/java/org/apache/kafka/common/security/auth/KafkaPrincipalTest.java
>  PRE-CREATION 
>   core/src/main/scala/kafka/api/OffsetRequest.scala 
> f418868046f7c99aefdccd9956541a0cb72b1500 
>   core/src/main/scala/kafka/common/AuthorizationException.scala PRE-CREATION 
>   core/src/main/scala/kafka/common/ErrorMapping.scala 
> c75c68589681b2c9d6eba2b440ce5e58cddf6370 
>   core/src/main/scala/kafka/security/auth/Acl.scala PRE-CREATION 
>   core/src/main/scala/kafka/security/auth/Authorizer.scala PRE-CREATION 
>   core/src/main/scala/kafka/security/auth/Operation.scala PRE-CREATION 
>   core/src/main/scala/kafka/security/auth/PermissionType.scala PRE-CREATION 
>   core/src/main/scala/kafka/security/auth/Resource.scala PRE-CREATION 
>   core/src/main/scala/kafka/security/auth/ResourceType.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> a3a8df0545

[jira] [Commented] (KAFKA-2210) KafkaAuthorizer: Add all public entities, config changes and changes to KafkaAPI and kafkaServer to allow pluggable authorizer implementation.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723854#comment-14723854
 ] 

Jun Rao commented on KAFKA-2210:


[~parth.brahmbhatt], thanks for the latest patch. Just reviewed it. In terms of 
sequencing, what's the order that you want ACL related patches to be committed? 
I assume KAFKA-2210, KAFKA-2211, KAFKA-2212 and KAFKA-2145?

> KafkaAuthorizer: Add all public entities, config changes and changes to 
> KafkaAPI and kafkaServer to allow pluggable authorizer implementation.
> --
>
> Key: KAFKA-2210
> URL: https://issues.apache.org/jira/browse/KAFKA-2210
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
> Fix For: 0.8.3
>
> Attachments: KAFKA-2210.patch, KAFKA-2210_2015-06-03_16:36:11.patch, 
> KAFKA-2210_2015-06-04_16:07:39.patch, KAFKA-2210_2015-07-09_18:00:34.patch, 
> KAFKA-2210_2015-07-14_10:02:19.patch, KAFKA-2210_2015-07-14_14:13:19.patch, 
> KAFKA-2210_2015-07-20_16:42:18.patch, KAFKA-2210_2015-07-21_17:08:21.patch, 
> KAFKA-2210_2015-08-10_18:31:54.patch, KAFKA-2210_2015-08-20_11:27:18.patch, 
> KAFKA-2210_2015-08-25_17:59:22.patch, KAFKA-2210_2015-08-26_14:29:02.patch
>
>
> This is the first subtask for Kafka-1688. As Part of this jira we intend to 
> agree on all the public entities, configs and changes to existing kafka 
> classes to allow pluggable authorizer implementation.
> Please see KIP-11 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface
>  for detailed design. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2210) KafkaAuthorizer: Add all public entities, config changes and changes to KafkaAPI and kafkaServer to allow pluggable authorizer implementation.

[Parth Brahmbhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723866#comment-14723866
 ] 

Parth Brahmbhatt commented on KAFKA-2210:
-

[~junrao] yes that is correct sequencing and I haven't updated 2211 and 2212 as 
we were changing 2210. Once 2210 is merged I will update 2211 and 2212 and send 
new pull requests.

> KafkaAuthorizer: Add all public entities, config changes and changes to 
> KafkaAPI and kafkaServer to allow pluggable authorizer implementation.
> --
>
> Key: KAFKA-2210
> URL: https://issues.apache.org/jira/browse/KAFKA-2210
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
> Fix For: 0.8.3
>
> Attachments: KAFKA-2210.patch, KAFKA-2210_2015-06-03_16:36:11.patch, 
> KAFKA-2210_2015-06-04_16:07:39.patch, KAFKA-2210_2015-07-09_18:00:34.patch, 
> KAFKA-2210_2015-07-14_10:02:19.patch, KAFKA-2210_2015-07-14_14:13:19.patch, 
> KAFKA-2210_2015-07-20_16:42:18.patch, KAFKA-2210_2015-07-21_17:08:21.patch, 
> KAFKA-2210_2015-08-10_18:31:54.patch, KAFKA-2210_2015-08-20_11:27:18.patch, 
> KAFKA-2210_2015-08-25_17:59:22.patch, KAFKA-2210_2015-08-26_14:29:02.patch
>
>
> This is the first subtask for Kafka-1688. As Part of this jira we intend to 
> agree on all the public entities, configs and changes to existing kafka 
> classes to allow pluggable authorizer implementation.
> Please see KIP-11 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface
>  for detailed design. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Client-side Assignment for New Consumer

[Jiangjie Qin]

Ewen,

To be more clear about the numbers in LinkedIn including the expected
growth, we have the following table (Assuming avg topic name length is 27
bytes):

#TopicMirrored   #MirrorMakerNodes   #ConsumerPerNode
#BytesPerRebalance #BytesPerMirrorMakerRollingBounce
100   26
4   33MB   33MB
* 26 * 2 = 1.7 GB
300   484
342MB 342MB * 48 * 2 = 32 GB
500   644
1GB 1GB * 48 * 2 = 96 GB

The number is planed based on the current growth rate of our traffic in
future 3 years. The bytes in the table are calculated based on the proposed
protocol. We are concerned about whether the broadcast based protocol would
be able to support the scale at that point. The table above only accounts
for one mirror maker consumer group. Given we have four mirror maker
consumer groups, even more bytes would be transferred.

In general, I think for Kafka as a distributed system that can support
thousands of consumers, the consumer rebalance needs to be cheap. It is
important for a few reasons. e.g. Consumer churn might happen from time to
time when we have so many consumers; Supporting for basic use case like
rolling bounce, etc.

Thanks,

Jiangjie (Becket) Qin


On Sun, Aug 30, 2015 at 12:58 AM, Jiangjie Qin  wrote:

> Ewen,
>
> 1. My concern is for the long subscription set that will be included in
> the protocol metadata. Because we are using selective copy with a long
> subscription set. 100 topics with 27 avg name length will give a 2.7K for
> topic name and 0.4K for the partition number bytes. This gives 3.1K for
> each subscription set. For each mirror maker consumer group, the total size
> of JoinGroupResponse will be 3.1K * 104 * 104 ~ 33 MB. We have several
> other things to think about as well: What happen if the four mirror maker
> consumer group have same broker as coordinator (unlikely for a large
> cluster but still possible)? What if the number of topics increase to 300
> or 500? What if the outbound traffic on the coordinator broker was large?
> What if quota is applied to a client (currently quota only applies to fetch
> and produce, but if metadata is big, maybe we also need to consider take it
> into account)?
>
> 2. For the rolling bounce. Are you saying we can send LeaveGroupRequest if
> it is a shutdown, and don't send LeaveGroupRequest if it is a bounce? I am
> not sure how would that work. But when people do a rolling bounce typically
> it is just a stop-then-start operation. When mirror maker shutdown, how can
> we know whether there will be a follow up start or not? If we send a
> LeaveGroupRequest to trigger a rebalance. In that case, the rebalance
> supposedly would be picked up by all the consumers in the group at next
> heartbeat. The rebalance would likely finish within a few seconds if the
> heartbeat interval is set to 3 seconds. So it seems another round of
> rebalance is unavoidable after the shutdown mirror maker node come back.
>
> 3. For the metadata inconsistency, I am actually not sure how useful it is
> to have all the consumer to fetch from the least used brokers as we are
> doing now. The "least used node" is purely based on the client side view,
> it does not necessarily mean the broker is actually "least used". It is
> totally possible that one of the consume thought a broker is least used
> while everyone else are sending data to that broker. Also, the amount of
> bytes for topic metadata will only be linear to the number of consumers in
> the group. I suppose it will cause big issue. We might get unlucky, for
> example the four mirror maker consumer groups are hashed to the same
> broker. In that case, the total TopicMetadataResponse sent by that that
> broker might be a few MB, but is way less than the total size of
> JoinGroupResponse. So that would be the first issue to solve.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
>
>
>
>
>
>
> On Sat, Aug 29, 2015 at 12:39 AM, Ewen Cheslack-Postava  > wrote:
>
>> @Onur - I think you reviewed the KAFKA-2464 patch in some detail. wrt the
>> specific problem you raised about using largest # of partitions, you're
>> right, with the updated proposal this isn't possible because we only send
>> a
>> hash of the metadata in order to avoid the overhead of per-member metadata
>> being transmitted back to members in the join group response.
>>
>> However, the KAKFA-2464 patch addresses this by avoiding inconsistencies
>> as
>> much as possible by preferring the coordinator for metadata fetches when
>> it
>> detects inconsistencies. If metadata is changing fast enough, this can
>> still result in another round of inconsistency. However, note that this
>> only applies to metadata that would be included in an subscription (i.e.,
>> if quickly created/deleted topic

[GitHub] kafka pull request: KAFKA-2475: Make Copycat only have a Converter...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/172


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2475) Reduce copycat configs to only specify a converter or serializer, not both

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-2475:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 172
[https://github.com/apache/kafka/pull/172]

> Reduce copycat configs to only specify a converter or serializer, not both
> --
>
> Key: KAFKA-2475
> URL: https://issues.apache.org/jira/browse/KAFKA-2475
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
>
> Ultimately, all we care about is getting from byte[] -> Copycat data API. The 
> current split of the interfaces makes it easy to reuse existing serializers, 
> but you still have to implement a new class.
> It will be simpler, both conceptually and by requiring fewer configs, to 
> combine both these steps into a single API. This also allows certain formats 
> to preserve more information across these (e.g. for primitive types in 
> schema, which otherwise could lose certain schema information).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2475) Reduce copycat configs to only specify a converter or serializer, not both

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723901#comment-14723901
 ] 

ASF GitHub Bot commented on KAFKA-2475:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/172


> Reduce copycat configs to only specify a converter or serializer, not both
> --
>
> Key: KAFKA-2475
> URL: https://issues.apache.org/jira/browse/KAFKA-2475
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
>
> Ultimately, all we care about is getting from byte[] -> Copycat data API. The 
> current split of the interfaces makes it easy to reuse existing serializers, 
> but you still have to implement a new class.
> It will be simpler, both conceptually and by requiring fewer configs, to 
> combine both these steps into a single API. This also allows certain formats 
> to preserve more information across these (e.g. for primitive types in 
> schema, which otherwise could lose certain schema information).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2486) New consumer performance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723913#comment-14723913
 ] 

ASF GitHub Bot commented on KAFKA-2486:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/180

KAFKA-2486; fix performance regression in new consumer

The sleep() in KafkaConsumer's poll blocked any pending IO from being 
completed and created a performance bottleneck. It was intended to implement 
the fetch backoff behavior, but that was a misunderstanding of the setting 
"retry.backoff.ms" which should only affect failed fetches. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2486

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/180.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #180


commit 8bec099900fdb511cf9f4b5a31c63d695e6a9c49
Author: Jason Gustafson 
Date:   2015-08-31T19:23:30Z

KAFKA-2486; fix performance regression in new consumer




> New consumer performance
> 
>
> Key: KAFKA-2486
> URL: https://issues.apache.org/jira/browse/KAFKA-2486
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
> Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a 
> recent report on the mailing list indicates it's dropped significantly. After 
> evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, 
> compared to 600+MB/s previously. Before release, we should get the 
> performance back on par.
> Some details about where the regression occurred from the mailing list 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E
>  :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good 
> performance -> an error due to broker apparently not accepting the partition 
> assignment strategy. Since this commit seems to add heartbeats and the server 
> side code for partition assignment strategies, I assume we were missing 
> something on the client side and by filling in the server side, things 
> stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or 
> a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the 
> perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, 
> but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new 
> consumer, even if regressions weren't automatically detected. It would at 
> least allow for manually checking for regressions. This should not be 
> difficult since there are already old consumer performance tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2486; fix performance regression in new ...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/180

KAFKA-2486; fix performance regression in new consumer

The sleep() in KafkaConsumer's poll blocked any pending IO from being 
completed and created a performance bottleneck. It was intended to implement 
the fetch backoff behavior, but that was a misunderstanding of the setting 
"retry.backoff.ms" which should only affect failed fetches. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2486

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/180.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #180


commit 8bec099900fdb511cf9f4b5a31c63d695e6a9c49
Author: Jason Gustafson 
Date:   2015-08-31T19:23:30Z

KAFKA-2486; fix performance regression in new consumer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2485; Allow producer performance to take...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/174


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2485) Allow producer performance to take properties from a file via --producer.config command line parameter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723924#comment-14723924
 ] 

ASF GitHub Bot commented on KAFKA-2485:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/174


> Allow producer performance to take properties from a file via 
> --producer.config command line parameter
> --
>
> Key: KAFKA-2485
> URL: https://issues.apache.org/jira/browse/KAFKA-2485
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> In order to allow ProducerPerformance to produce via ssl connection, we need 
> the following (example) configuration:
> security.protocol=SSL
> ssl.protocol=TLS
> ssl.truststore.type=JKS
> ssl.truststore.location=serverkeystore.jks
> ssl.truststore.password=password
> ssl.keymanager.algorithm=SunX509
> ssl.trustmanager.algorithm=SunX509
> There are two ways to achieve it: 1) extend ProducerPerformance to explicitly 
> accept all 7 ssl-related parameters; 2) allow ProducerPerformance to take 
> properties from a file via --consumer.config command line parameter.
> It seems option 2) is better, because it requires less code, allows new 
> options to be easily added in the future, and doesn't require user to specify 
> password in the command line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2485) Allow producer performance to take properties from a file via --producer.config command line parameter

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-2485:

   Resolution: Fixed
Fix Version/s: 0.8.3
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 174
[https://github.com/apache/kafka/pull/174]

> Allow producer performance to take properties from a file via 
> --producer.config command line parameter
> --
>
> Key: KAFKA-2485
> URL: https://issues.apache.org/jira/browse/KAFKA-2485
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.8.3
>
>
> In order to allow ProducerPerformance to produce via ssl connection, we need 
> the following (example) configuration:
> security.protocol=SSL
> ssl.protocol=TLS
> ssl.truststore.type=JKS
> ssl.truststore.location=serverkeystore.jks
> ssl.truststore.password=password
> ssl.keymanager.algorithm=SunX509
> ssl.trustmanager.algorithm=SunX509
> There are two ways to achieve it: 1) extend ProducerPerformance to explicitly 
> accept all 7 ssl-related parameters; 2) allow ProducerPerformance to take 
> properties from a file via --consumer.config command line parameter.
> It seems option 2) is better, because it requires less code, allows new 
> options to be easily added in the future, and doesn't require user to specify 
> password in the command line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Kafka-trunk #608

[Apache Jenkins Server]

See 

Re: Mirror maker SSL support

[Jun Rao]

Probably within 2-3 weeks.

Thanks,

Jun

On Mon, Aug 31, 2015 at 10:54 AM, Xiang Zhou (Samuel) 
wrote:

> Thank you for update Jun.
> What's the schedule to get new consumer implemented in Mirror Maker?
>
> Samuel
>
> On Mon, Aug 31, 2015 at 10:06 AM, Jun Rao  wrote:
>
> > Currently, MirrorMaker can run the producer with SSL. We will add the
> > option to run the new consumer in MirrorMaker in KAFKA-2452, which will
> > enable SSL on the consumer side too.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Aug 28, 2015 at 12:01 PM, Xiang Zhou (Samuel)  >
> > wrote:
> >
> > > Hi, folks,
> > >
> > > I just want to check if anybody know if there are any plan of Kafka
> > > MirrorMaker support running over SSL. I tried to follow
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
> > > to configure consumer/producer properties with SSL but no luck, looks
> > that
> > > Kafka MirrorMaker consumer use the difference implementation when
> > consuming
> > > data compared to new consumer.
> > >
> > > So I want to check when SSL MM will be supported. Or is there a JIRA
> > ticket
> > > to track it so I can follow it?
> > >
> > > Thanks!
> > > Samuel
> > >
> >
>

[jira] [Resolved] (KAFKA-2486) New consumer performance

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-2486.
-
Resolution: Fixed

Issue resolved by pull request 180
[https://github.com/apache/kafka/pull/180]

> New consumer performance
> 
>
> Key: KAFKA-2486
> URL: https://issues.apache.org/jira/browse/KAFKA-2486
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
> Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a 
> recent report on the mailing list indicates it's dropped significantly. After 
> evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, 
> compared to 600+MB/s previously. Before release, we should get the 
> performance back on par.
> Some details about where the regression occurred from the mailing list 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E
>  :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good 
> performance -> an error due to broker apparently not accepting the partition 
> assignment strategy. Since this commit seems to add heartbeats and the server 
> side code for partition assignment strategies, I assume we were missing 
> something on the client side and by filling in the server side, things 
> stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or 
> a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the 
> perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, 
> but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new 
> consumer, even if regressions weren't automatically detected. It would at 
> least allow for manually checking for regressions. This should not be 
> difficult since there are already old consumer performance tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2486) New consumer performance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724070#comment-14724070
 ] 

ASF GitHub Bot commented on KAFKA-2486:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/180


> New consumer performance
> 
>
> Key: KAFKA-2486
> URL: https://issues.apache.org/jira/browse/KAFKA-2486
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
> Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a 
> recent report on the mailing list indicates it's dropped significantly. After 
> evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, 
> compared to 600+MB/s previously. Before release, we should get the 
> performance back on par.
> Some details about where the regression occurred from the mailing list 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E
>  :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good 
> performance -> an error due to broker apparently not accepting the partition 
> assignment strategy. Since this commit seems to add heartbeats and the server 
> side code for partition assignment strategies, I assume we were missing 
> something on the client side and by filling in the server side, things 
> stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or 
> a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the 
> perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, 
> but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new 
> consumer, even if regressions weren't automatically detected. It would at 
> least allow for manually checking for regressions. This should not be 
> difficult since there are already old consumer performance tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2486; fix performance regression in new ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/180


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2486) New consumer performance

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724079#comment-14724079
 ] 

Jay Kreps commented on KAFKA-2486:
--

Where does this leave perf for the new consumer vs old? Maybe that's getting 
looked into as part of the SSL testing...

> New consumer performance
> 
>
> Key: KAFKA-2486
> URL: https://issues.apache.org/jira/browse/KAFKA-2486
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
> Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a 
> recent report on the mailing list indicates it's dropped significantly. After 
> evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, 
> compared to 600+MB/s previously. Before release, we should get the 
> performance back on par.
> Some details about where the regression occurred from the mailing list 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E
>  :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good 
> performance -> an error due to broker apparently not accepting the partition 
> assignment strategy. Since this commit seems to add heartbeats and the server 
> side code for partition assignment strategies, I assume we were missing 
> something on the client side and by filling in the server side, things 
> stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or 
> a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the 
> perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, 
> but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new 
> consumer, even if regressions weren't automatically detected. It would at 
> least allow for manually checking for regressions. This should not be 
> difficult since there are already old consumer performance tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Kafka-trunk #609

[Apache Jenkins Server]

See 

Changes:

[cshapi] KAFKA-2486; fix performance regression in new consumer

--
[...truncated 1615 lines...]
kafka.admin.AdminTest > testPartitionReassignmentWithLeaderInNewReplicas PASSED

kafka.api.ProducerBounceTest > testBrokerFailure PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.admin.AddPartitionsTest > testManualAssignmentOfReplicas PASSED

kafka.log.LogTest > testCorruptLog PASSED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.api.QuotasTest > testThrottledProducerConsumer PASSED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.admin.DeleteConsumerGroupTest > testGroupWideDeleteInZK PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.api.ConsumerTest > testPauseStateNotPreservedByRebalance PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicWithRecoveredFollower PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.common.ConfigTest > testInvalidGroupIds PASSED

kafka.common.ConfigTest > testInvalidClientIds PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.admin.AdminTest > testShutdownBroker PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.api.ProducerSendTest > testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.admin.AdminTest > testTopicCreationWithCollision PASSED

kafka.admin.AdminTest > testTopicCreationInZK PASSED

kafka.admin.AddPartitionsTest > testReplicaPlacement PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.api.ConsumerTest > testSeek PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckOne PASSED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic PASSED

kafka.api.ConsumerTest > testPositionAndCommit PASSED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.api.ProducerFailureHandlingTest > testWrongBrokerList PASSED

kafka.integration.RollingBounceTest > testRollingBounce PASSED

kafka.common.TopicTest > testInvalidTopicNames PASSED

kafka.common.TopicTest > testTopicHasCollision PASSED

kafka.common.TopicTest > testTopicHasCollisionChars PASSED

kafka.api.ProducerFailureHandlingTest > testNotEnoughReplicas PASSED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion PASSED

kafka.api.ConsumerTest > testUnsubscribeTopic PASSED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures PASSED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.api.ConsumerTest > testListTopics PASSED

kafka.api.ProducerFailureHandlingTest > testNoResponse PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.api.ConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.api.ProducerFailureHandlingTest > testNonExistentTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.api.ProducerFailureHandlingTest > testInvalidPartition PASSED

kafka.api.ConsumerTest > testGroupConsumption PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.api.ProducerFailureHandlingTest > testSendAfterClosed PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckZero PASSED

kafka.integration.PrimitiveApiTest > testFetchRequest

[IMPORTANT] Questions related to AdminUtils thread safety

[Sivananda Reddys Thummala Abbigari]

Hi,

Could you please let me know if calls to "AdminUtils.topicExists" and
"AdminUtils.createTopic" are thread-safe?

Thank you,
Siva.

[GitHub] kafka pull request: KAFKA-2278; JmxTool should support querying al...

[lindong28]

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/181

KAFKA-2278; JmxTool should support querying all objects when object-n…

…ame is omitted

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-2278

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/181.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #181


commit 82cd911cc832a91ecae5bca2a9e2ca8bc7ff69e4
Author: Dong Lin 
Date:   2015-08-31T22:53:22Z

KAFKA-2278; JmxTool should support querying all objects when object-name is 
omitted




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2278) JmxTool should support querying all objects when object-name is omitted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724336#comment-14724336
 ] 

ASF GitHub Bot commented on KAFKA-2278:
---

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/181

KAFKA-2278; JmxTool should support querying all objects when object-n…

…ame is omitted

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-2278

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/181.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #181


commit 82cd911cc832a91ecae5bca2a9e2ca8bc7ff69e4
Author: Dong Lin 
Date:   2015-08-31T22:53:22Z

KAFKA-2278; JmxTool should support querying all objects when object-name is 
omitted




> JmxTool should support querying all objects when object-name is omitted
> ---
>
> Key: KAFKA-2278
> URL: https://issues.apache.org/jira/browse/KAFKA-2278
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dong Lin
>Assignee: Dong Lin
> Attachments: KAFKA-2278.patch
>
>
> One usecase of JmxTool is to allow user to query all existing objects and 
> attributes, such that the developer can quickly lookup the right object name 
> and attribute name in the output. Indeed, existing API of JmxTool allows user 
> to do this when object-name is not specified in argument.
> However, current implementation of JmxTool doesn't support this operation due 
> to the way it is implemented. This is partly due to the fact that some JMX 
> object, such as java.lang:type=MemoryPool, doesn't have value for all its 
> attributes.
> The patch submitted here will fix the problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2490) support new consumer in ConsumerOffsetChecker

[Jun Rao (JIRA)]

Jun Rao created KAFKA-2490:
--

 Summary: support new consumer in ConsumerOffsetChecker
 Key: KAFKA-2490
 URL: https://issues.apache.org/jira/browse/KAFKA-2490
 Project: Kafka
  Issue Type: Sub-task
  Components: core
Affects Versions: 0.8.3
Reporter: Jun Rao


We need to add an option to enable the new consumer in ConsumerOffsetChecker. 
Since the new consumer no longer stores the owner of each partition in ZK, we 
need to patch ConsumerOffsetChecker accordingly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2419:
---
Affects Version/s: 0.8.3

> Allow certain Sensors to be garbage collected after inactivity
> --
>
> Key: KAFKA-2419
> URL: https://issues.apache.org/jira/browse/KAFKA-2419
> Project: Kafka
>  Issue Type: New Feature
>Affects Versions: 0.8.3
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>  Labels: quotas
>
> Currently, metrics cannot be removed once registered. 
> Implement a feature to remove certain sensors after a certain period of 
> inactivity (perhaps configurable).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724422#comment-14724422
 ] 

Jun Rao commented on KAFKA-2419:


[~aauradkar], are you working on this? We probably need to have this fixed in 
the 0.8.3 release.

> Allow certain Sensors to be garbage collected after inactivity
> --
>
> Key: KAFKA-2419
> URL: https://issues.apache.org/jira/browse/KAFKA-2419
> Project: Kafka
>  Issue Type: New Feature
>Affects Versions: 0.8.3
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>  Labels: quotas
>
> Currently, metrics cannot be removed once registered. 
> Implement a feature to remove certain sensors after a certain period of 
> inactivity (perhaps configurable).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2490) support new consumer in ConsumerOffsetChecker

[Ashish K Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh reassigned KAFKA-2490:
-

Assignee: Ashish K Singh

> support new consumer in ConsumerOffsetChecker
> -
>
> Key: KAFKA-2490
> URL: https://issues.apache.org/jira/browse/KAFKA-2490
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 0.8.3
>Reporter: Jun Rao
>Assignee: Ashish K Singh
> Fix For: 0.8.3
>
>
> We need to add an option to enable the new consumer in ConsumerOffsetChecker. 
> Since the new consumer no longer stores the owner of each partition in ZK, we 
> need to patch ConsumerOffsetChecker accordingly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2452) enable new consumer in mirror maker

[Samuel Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724434#comment-14724434
 ] 

Samuel Zhou commented on KAFKA-2452:


It is a very important feature to enable new consumer for mirror maker since 
our project urgently requires it for SSL support. I appreciate early releases 
that I could try out or any updates for this ticket. Thanks!

> enable new consumer in mirror maker
> ---
>
> Key: KAFKA-2452
> URL: https://issues.apache.org/jira/browse/KAFKA-2452
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 0.8.3
>Reporter: Jun Rao
>Assignee: Jiangjie Qin
> Fix For: 0.8.3
>
>
> We need to an an option to enable the new consumer in mirror maker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2486) New consumer performance

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724436#comment-14724436
 ] 

Ewen Cheslack-Postava commented on KAFKA-2486:
--

[~jkreps] System tests are being worked on in KAFKA-2489 
(https://github.com/apache/kafka/pull/179). There's some issue with the new 
consumer timing out when starting up that's causing it to sometimes generate 
bad output, but once that's sorted out we'll start having nightly numbers 
available.

> New consumer performance
> 
>
> Key: KAFKA-2486
> URL: https://issues.apache.org/jira/browse/KAFKA-2486
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
> Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a 
> recent report on the mailing list indicates it's dropped significantly. After 
> evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, 
> compared to 600+MB/s previously. Before release, we should get the 
> performance back on par.
> Some details about where the regression occurred from the mailing list 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E
>  :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good 
> performance -> an error due to broker apparently not accepting the partition 
> assignment strategy. Since this commit seems to add heartbeats and the server 
> side code for partition assignment strategies, I assume we were missing 
> something on the client side and by filling in the server side, things 
> stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or 
> a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the 
> perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, 
> but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new 
> consumer, even if regressions weren't automatically detected. It would at 
> least allow for manually checking for regressions. This should not be 
> difficult since there are already old consumer performance tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2486) New consumer performance

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724448#comment-14724448
 ] 

Jay Kreps commented on KAFKA-2486:
--

[~ewencp] Nightly will be awesome!

I think what I am asking is more basic--did this change actually address the 
perf issue? I.e. is the new consumer and old consumer now back at parity? I ask 
because a lot of code changed since the one off testing so I wouldn't be 
shocked if two or three things all went bad.

> New consumer performance
> 
>
> Key: KAFKA-2486
> URL: https://issues.apache.org/jira/browse/KAFKA-2486
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
> Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a 
> recent report on the mailing list indicates it's dropped significantly. After 
> evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, 
> compared to 600+MB/s previously. Before release, we should get the 
> performance back on par.
> Some details about where the regression occurred from the mailing list 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E
>  :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good 
> performance -> an error due to broker apparently not accepting the partition 
> assignment strategy. Since this commit seems to add heartbeats and the server 
> side code for partition assignment strategies, I assume we were missing 
> something on the client side and by filling in the server side, things 
> stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or 
> a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the 
> perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, 
> but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new 
> consumer, even if regressions weren't automatically detected. It would at 
> least allow for manually checking for regressions. This should not be 
> difficult since there are already old consumer performance tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36858: Patch for KAFKA-2120

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review97213
---



clients/src/main/java/org/apache/kafka/clients/NetworkClient.java (lines 361 - 
366)


We need to think through this logic a bit more. The patch here disconnects 
the connection from the selector, but doesn't mark the connectionState as 
disconnected immediately. Only until the next networkClient.poll(), does the 
connectionState change to disconnected. The issue with this approach is that 
selector.disconnect actually cancels the socket key. So, at that moment, the 
connection is no longer usable. However, the connectionState is still 
connected. A client can check the connection as ready and then makes a send 
call. The send will then get a CancelledKeyException, which is weird.

We are dealing with a similar issue in KAFKA-2411. Our current thinking is 
to have a networkClient.disconnect() that closes the socket key as well as 
removes the client from connectionState. This will make the state in 
networkClient consistent after each poll() call. If we have that, we can just 
call networkClient.disconnect() in handleTimedOutRequests() and handle those 
disconnected connections immediately. Then, we don't need to maintain the 
clientDisconnects state in Selector.


- Jun Rao


On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Aug. 12, 2015, 5:59 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> 7ab2503794ff3aab39df881bd9fbae6547827d3b 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> d35b421a515074d964c7fccb73d260b847ea5f00 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> aa264202f2724907924985a5ecbe74afc4c6c04b 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 0baf16e55046a2f49f6431e01d52c323c95eddf0 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> ce20111ac434eb8c74585e9c63757bb9d60a832f 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> 9133d85342b11ba2c9888d4d2804d181831e7a8e 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 43238ceaad0322e39802b615bb805b895336a009 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 158f9829ff64a969008f699e40c51e918287859e 
>   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
> 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
>   core/src/test/scala/integration/kafka/api/Produc

[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724559#comment-14724559
 ] 

Jun Rao commented on KAFKA-2477:


Interesting, so there is nothing in broker 0's log about truncating the log for 
partition [log.event,3]? 

It seems that this is reproducible. Would it be possible for you to enable 
TRACE level logging for class kafka.log.Log on every broker. This will log the 
new log end offset after each message append. When this issue happens again, we 
can verify if it is indeed that the follower uses a fetch offset that's larger 
than the leader's log end offset.

> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
> Attachments: kafka_log.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

How can I start writing a Copycat connector?

[James Cheng]

Hi,

I'd like to write a Copycat connector. I saw on Github that the Copycat data 
APIs have been committed to trunk. How can I get started? I've never developed 
against trunk before. Do I git clone from trunk, or is there a pre-built set of 
jars from trunk that I can use?

I see that there is a sample connector in copycat/file. For development 
purposes, I could just create a peer to that. Obviously I would not check in my 
connector in that location, but would that be the best way to get started?

Thanks,
-James

[GitHub] kafka pull request: MINOR: Quota's equals() is buggy.

[eribeiro]

GitHub user eribeiro opened a pull request:

https://github.com/apache/kafka/pull/182

MINOR: Quota's equals() is buggy.

It compares upper bound with itself.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eribeiro/kafka equals-bug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/182.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #182


commit 527c978ab92a2216d5700519138c6dc3a5cd3592
Author: Edward Ribeiro 
Date:   2015-09-01T02:26:16Z

MINOR: Quota's equals() is buggy.

It compares upper bound with itself.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: How can I start writing a Copycat connector?

[Gwen Shapira]

Hi James,

This is so exciting :)

While the APIs are still marked as @unstable, we don't have immediate plans
to modify them - so now is a good time to write a connector. Just accept
the possibility that few modifications may be needed before release (I'm
trying to get Ewen to add mandatory connector versions to the API...)

I typically clone trunk and then run ./gradlew install - which builds and
places the jars in local mvn repository.
Then I can use them as dependencies in other projects.

IMO, starting with an existing example and modifying it is a good approach.
You can look at few more examples here:
https://github.com/confluentinc/copycat-hdfs
https://github.com/confluentinc/copycat-jdbc
(Those are still under active development... don't try them in production
yet)

Ewen will probably chime in with more advice :)

Gwen

On Mon, Aug 31, 2015 at 7:14 PM, James Cheng  wrote:

> Hi,
>
> I'd like to write a Copycat connector. I saw on Github that the Copycat
> data APIs have been committed to trunk. How can I get started? I've never
> developed against trunk before. Do I git clone from trunk, or is there a
> pre-built set of jars from trunk that I can use?
>
> I see that there is a sample connector in copycat/file. For development
> purposes, I could just create a peer to that. Obviously I would not check
> in my connector in that location, but would that be the best way to get
> started?
>
> Thanks,
> -James
>
>

[jira] [Created] (KAFKA-2491) Update ErrorMapping with New Consumer Errors

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-2491:
--

 Summary: Update ErrorMapping with New Consumer Errors
 Key: KAFKA-2491
 URL: https://issues.apache.org/jira/browse/KAFKA-2491
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jason Gustafson
Priority: Minor


Some errors used by the new consumer have not been added to ErrorMapping. Until 
this class is removed, it should probably be kept consistent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724636#comment-14724636
 ] 

jin xing commented on KAFKA-2394:
-

I'm new comer for Kafka, I'm directed here by Kafka web page "How to 
contribute";

Can I ask how I can be assigned this task?

> Use RollingFileAppender by default in log4j.properties
> --
>
> Key: KAFKA-2394
> URL: https://issues.apache.org/jira/browse/KAFKA-2394
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Priority: Minor
>  Labels: newbie
>
> The default log4j.properties bundled with Kafka uses ConsoleAppender and 
> DailyRollingFileAppender, which offer no protection to users from spammy 
> logging. In extreme cases (such as when issues like KAFKA-1461 are 
> encountered), the logs can exhaust the local disk space. This could be a 
> problem for Kafka adoption since new users are less likely to adjust the 
> logging properties themselves, and are more likely to have configuration 
> problems which result in log spam. 
> To fix this, we can use RollingFileAppender, which offers two settings for 
> controlling the maximum space that log files will use.
> maxBackupIndex: how many backup files to retain
> maxFileSize: the max size of each log file
> One question is whether this change is a compatibility concern? The backup 
> strategy and filenames used by RollingFileAppender are different from those 
> used by DailyRollingFileAppender, so any tools which depend on the old format 
> will break. If we think this is a serious problem, one solution would be to 
> provide two versions of log4j.properties and add a flag to enable the new 
> one. Another solution would be to include the RollingFileAppender 
> configuration in the default log4j.properties, but commented out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: MockClient's disconnect() method has tw...

[eribeiro]

GitHub user eribeiro opened a pull request:

https://github.com/apache/kafka/pull/183

MINOR: MockClient's disconnect() method has two bugs

 (and a prospetive refactoring)

* First, it compares Strings using `==` instead of `equals()`

* Second, it tries to remove a String from a Set. As disconnect() 
method is only called by a single method on SenderTest then it's safe to 
refactor it to make it both explicit that its argument is a node id and perform 
a Integer.valueOf() before trying to remove from `ready`.

* Third, not a bug, but the Iterator logic can be simplified, shrinking the 
scope of the Iterator without changing the logic.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eribeiro/kafka mockclient-disconnect

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/183.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #183


commit 91189feb46e16b17a86596a011e12df9fefbbb49
Author: Edward Ribeiro 
Date:   2015-09-01T03:24:04Z

MINOR: MockClient's disconnect() method has two bugs (and a prospetive 
refactoring)

* First, it compares Strings using `==` instead of `equals()`

* Second, it tries to remove a String from a Set. As disconnect() 
method is only called by a single method on SenderTest then it's safe to 
refactor it to make it both explicity that its argument is a node id and do a 
Integer.valueOf() before trying to remove from `ready`.

* Third, not a bug, but the Iterator logic can be simplified, shrinking the 
scope of the Iterator without changing the logic.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [IMPORTANT] Questions related to AdminUtils thread safety

[Aditya Auradkar]

topicExists simply reads ZK, so yes. createTopic should also be fine unless
you try to create the same topic concurrently. AdminUtils itself does not
maintain any state, just some static util functions.

On Mon, Aug 31, 2015 at 3:00 PM, Sivananda Reddys Thummala Abbigari <
sthumm...@salesforce.com> wrote:

> Hi,
>
> Could you please let me know if calls to "AdminUtils.topicExists" and
> "AdminUtils.createTopic" are thread-safe?
>
> Thank you,
> Siva.
>

no Kafka KIP meeting tomorrow

[Jun Rao]

Since there are no new KIP issues for discussion, again, there is no
KIP meeting
tomorrow.

Thanks,

Jun