Re: [DISCUSS] KIP-939: Support Participation in 2PC

2023-08-25 Thread Roger Hoover
Other than supporting multiplexing transactional streams on a single producer, I don't see how to improve it. On Thu, Aug 24, 2023 at 12:12 PM Artem Livshits wrote: > Hi Roger, > > Thank you for summarizing the cons. I agree and I'm curious what would be > the alter

Re: [DISCUSS] KIP-939: Support Participation in 2PC

2023-08-23 Thread Roger Hoover
). On Wed, Aug 23, 2023 at 12:53 PM Artem Livshits wrote: > Hi Roger, > > Thank you for the feedback. You make a very good point that we also > discussed internally. Adding support for multiple concurrent > transactions in one producer could be valuable but it seems to be a fairl

Re: [DISCUSS] KIP-939: Support Participation in 2PC

2023-08-22 Thread Roger Hoover
seems you're left with single-threaded model per application process? Thanks, Roger On Tue, Aug 22, 2023 at 5:11 PM Artem Livshits wrote: > Hi Roger, Arjun, > > Thank you for the questions. > > It looks like the application must have stable transactional ids over > ti

Re: [DISCUSS] KIP-939: Support Participation in 2PC

2023-08-21 Thread Roger Hoover
txns happening in the same JVM so it seems like the granularity managing transactional ids and txn state needs to line up with granularity of the DB locking. Does that make sense or am I misunderstanding? Thanks, Roger On Wed, Aug 16, 2023 at 11:40 PM Artem Livshits wrote: > Hello, >

Please Help with Error: org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = ...

2021-08-06 Thread roger
.poll(SocketServer.scala:863) at kafka.network.Processor.run(SocketServer.scala:762) at java.lang.Thread.run(Thread.java:748) Please help to see how can this be resolved. Many Thanks!! Roger

Re: [VOTE] KIP-602: Change default value for client.dns.lookup

2020-05-22 Thread Roger Hoover
+1 (non-binding) Thanks, Badai. On Fri, May 22, 2020 at 10:05 AM Ismael Juma wrote: > Thanks for the KIP, +1 (binding). > > Ismael > > On Fri, May 22, 2020 at 1:40 AM Badai Aqrandista > wrote: > > > Hi All, > > > > I would like to start the vote on KIP-602: Change default value for > > client.

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

2017-09-13 Thread Roger Hoover
s that we feel would be super useful for debugging? > > On Tue, Sep 12, 2017 at 6:08 PM, Roger Hoover > wrote: > > > Thanks, Ewen. > > > > I agree with you on the overhead of measuring time for SMTs and > > converters. I'd still argue for keeping other me

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

2017-09-12 Thread Roger Hoover
ly provide a bit more detail > in some ways (e.g. join & sync vs the entire rebalance). Not sure if we > want to effectively duplicate some info so it can all be located under > Connect names or rely on the existing metrics for some of these. > > -Ewen > > On Tue, Sep 12,

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

2017-09-12 Thread Roger Hoover
> > figure it out. So, e.g., batch size metrics sound like they could be > > > useful, but I'd probably wait until we have a clear use case. It seems > > > likely that it could be useful in diagnosing slow connectors (e.g. the > > > implementation just does something in

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

2017-09-12 Thread Roger Hoover
size metrics would tell you how full your batches are based on your current linger time so you can adjust the config. Cheers, Roger On Mon, Sep 11, 2017 at 7:50 PM, Ewen Cheslack-Postava wrote: > re: questions about additional metrics, I think we'll undoubtedly find more > that people

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

2017-09-11 Thread Roger Hoover
size - Counts of flush trigger method (time vs max batch size) Cheers, Roger On Sun, Sep 10, 2017 at 8:45 AM, Randall Hauch wrote: > Thanks, Gwen. > > That's a great idea, so I've changed the KIP to add those metrics. I've > also made a few other changes: >

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

2017-09-07 Thread Roger Hoover
Edoardo, thanks for the KIP. I think it's a good idea overall. +1 especially for including Session/Principal in the API. (#2 mentioned by Ismael) Also, the AlterPolicy should get the same info as create + delete (#4). Cheers, Roger On Thu, Sep 7, 2017 at 8:43 AM, Ismael Juma wrote:

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-31 Thread Roger Hoover
Makes sense in terms of priorities. Thanks, Apurva. On Thu, Aug 31, 2017 at 11:15 AM, Apurva Mehta wrote: > Thanks for the message, Roger. > > I think having 'acks=all' imply 'acks=minIsr' will probably result in some > improvement in the latency. However, I w

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-31 Thread Roger Hoover
5:18 PM, Roger Hoover wrote: > Sorry if this is a bit out of left field but can't help wondering... > > One way to improve producer performance while still having good guarantees > would be to allow a setting between acks=1 and acks=all. We could > introduce "acks

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-30 Thread Roger Hoover
of 2 followers) whereas * with acks=minIsr, the remote time of each request will be min(lag of 2 followers) Whatever your latency distribution is for replication, for any given remote time (say 100 ms), twice as many requests take longer than that time with acks=all vs acks=minIsr. Thoughts? Rog

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

2017-08-30 Thread Roger Hoover
Great. Thank you, Rajini. On Wed, Aug 30, 2017 at 7:53 AM, Rajini Sivaram wrote: > Hi Roger, > > Thank you for the suggestions. > > I think we should have a separate JIRA to address logging improvements for > authentication. That shouldn't need a KIP. The way the code

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

2017-08-29 Thread Roger Hoover
personation not allowed", or "You account has been locked, please contact cluster admin". Thanks, Roger On Tue, Aug 29, 2017 at 12:41 PM, Roger Hoover wrote: > Hi Rajini, > > The metrics in KIP-188 will provide counts across all users but the log > could potentially

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

2017-08-29 Thread Roger Hoover
g for Kafka separates authorization logs. It seems like a good idea to treat authentication logs the same way whether or not we choose DEBUG or INFO. https://github.com/apache/kafka/blob/trunk/config/log4j.properties#L54-L58 Cheers, Roger On Tue, Aug 29, 2017 at 10:51 AM, Rajini Sivaram wrote:

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

2017-08-29 Thread Roger Hoover
Just re-read the KIP and was wondering if you think INFO would be ok for logging successful authentications? They should be relatively infrequent. On Tue, Aug 29, 2017 at 9:54 AM, Roger Hoover wrote: > +1 (non-binding). Thanks, Rajini > > On Tue, Aug 29, 2017 at 2:10 AM, Ismael Ju

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

2017-08-29 Thread Roger Hoover
r > > > > > > > this, > > > > > > > > > we > > > > > > > > > > could easily add a long expiration period to be > > conservative, > > > > > but I > > > > > > > am > > > >

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

2017-08-29 Thread Roger Hoover
+1 (non-binding). Thanks, Rajini On Tue, Aug 29, 2017 at 2:10 AM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding) from me. > > Ismael > > On Thu, Aug 24, 2017 at 6:29 PM, Rajini Sivaram > wrote: > > > Hi all, > > > > I would like to start vote on KIP-152 to improve diagnostics of > > aut

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

2017-08-17 Thread Roger Hoover
Rajini, The table is super helpful. Thank you. On Thu, Aug 17, 2017 at 2:16 AM, Rajini Sivaram wrote: > Hi Roger, > > Thank you for the review. I have added a table with the scope of errors > counted for each request. > > Regards, > > Rajini > > On Thu, Aug 17,

Re: [DISCUSS] KIP-187 - Add cumulative count metric for all Kafka rate metrics

2017-08-16 Thread Roger Hoover
Rajini, Thank you. This is very useful. Grouping by metric by prefixing the name instead of making them MBeans is not quite as nice but seems like an good compromise for backward compatibility. Cheers, Roger On Wed, Aug 16, 2017 at 5:35 AM, Rajini Sivaram wrote: > Sorry, pressed send

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

2017-08-16 Thread Roger Hoover
t requests? Cheers, Roger On Wed, Aug 16, 2017 at 4:02 PM, Roger Hoover wrote: > Rajini, > > Thank you for the KIP. These are very helpful additions. One question on > the error code metrics: > > Will the total error counting happen at the the level of topic partition

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

2017-08-16 Thread Roger Hoover
successful, the counter for kafka.network:type=RequestMetrics,name=ErrorsPerSec,request=ProduceRequest,error=0 will be incremented by 3? Thanks, Roger On Wed, Aug 16, 2017 at 12:10 PM, Rajini Sivaram wrote: > I have created a KIP to add some additional metrics to support health > checks: >

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

2017-07-05 Thread Roger Hoover
Well deserved, indeed! Congrats, Ismael. On Wed, Jul 5, 2017 at 3:24 PM, Damian Guy wrote: > Congratulations Ismael! Very well deserved. > Cheers, > Damian > On Wed, 5 Jul 2017 at 22:54, Dong Lin wrote: > > > Congratulations Ismael! > > > > On Wed, Jul 5, 2017 at 1:55 PM, Jun Rao wrote: > > >

Re: [VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

2017-05-08 Thread Roger Hoover
+1 Sent from my iPhone > On May 8, 2017, at 5:00 AM, Edoardo Comar wrote: > > +1 > Many thanks Jun > -- > Edoardo Comar > IBM MessageHub > eco...@uk.ibm.com > IBM UK Ltd, Hursley Park, SO21 2JN > > IBM United Kingdom Limited Registered in Englan

Re: [DISCUSS] KIP-153 : Include only client traffic in BytesOutPerSec metric

2017-05-05 Thread Roger Hoover
Very helpful. Thank you, Jun. On Fri, May 5, 2017 at 4:42 PM, Guozhang Wang wrote: > Jun, > > Thanks for the KIP, LGTM. > > Guozhang > > On Fri, May 5, 2017 at 3:38 PM, Ismael Juma wrote: > > > Thanks Jun, looks good to me. > > > > Ismael > > > > On Fri, May 5, 2017 at 11:35 PM, Jun Rao wrote

[jira] [Commented] (KAFKA-3795) Transient system test failure upgrade_test.TestUpgrade

2017-04-03 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953759#comment-15953759 ] Roger Hoover commented on KAFKA-3795: - Happened again: http://confluent-systes

Re: [DISCUSS] KIP-136: Add Listener name and Security Protocol name to SelectorMetrics tags

2017-03-30 Thread Roger Hoover
Edo, Thanks for the proposal. This looks great to me. Cheers, Roger On Thu, Mar 30, 2017 at 8:51 AM, Edoardo Comar wrote: > Hi all, > > We created KIP-136: Add Listener name and Security Protocol name to > SelectorMetrics tags > > https://cwiki.apache.org/confluence/displa

[jira] [Updated] (KAFKA-4755) SimpleBenchmark consume test fails for streams

2017-03-28 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-4755: Priority: Blocker (was: Major) > SimpleBenchmark consume test fails for stre

[jira] [Reopened] (KAFKA-4755) SimpleBenchmark consume test fails for streams

2017-03-28 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover reopened KAFKA-4755: - Happened again. Logs here: http://confluent-kafka-0-10-2-system-test-results.s3-us-west-2

[jira] [Commented] (KAFKA-4689) OffsetValidationTest fails validation with "Current position greater than the total number of consumed records"

2017-03-28 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945543#comment-15945543 ] Roger Hoover commented on KAFKA-4689: - Happened again. Logs here http://confl

Re: [VOTE] KIP-124: Request rate quotas

2017-03-21 Thread Roger Hoover
Rajini, This is great. Thank you. +1 (non-binding) Roger On Tue, Mar 21, 2017 at 8:55 AM, Ismael Juma wrote: > Rajini, > > Thanks for the proposal and for addressing the (sometimes contradictory) > feedback. :) +1 (binding) from me. > > Ismael > > On Mon, Mar 20,

Re: [DISCUSS] KIP-124: Request rate quotas

2017-02-22 Thread Roger Hoover
uest thread resources per request, a given application will generally have a stable access pattern and can figure out empirically how many "request thread units" it needs to meet it's throughput/latency goals. Cheers, Roger On Wed, Feb 22, 2017 at 8:53 AM, Jun Rao wrote: > Hi, Ra

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

2017-02-15 Thread Roger Hoover
Yes. Thank you, Ismael. On Wed, Feb 8, 2017 at 2:30 AM, Ismael Juma wrote: > Hi Roger, > > Sorry for the delay. SCRAM is specified by: > > https://tools.ietf.org/html/rfc5802 > > The following quote is relevant: > > A SCRAM mechanism name is a string "SCRAM-

Re: [VOTE] KIP-48 Support for delegation tokens as an authentication mechanism

2017-02-08 Thread Roger Hoover
Thanks. I found the other discussion thread. Sorry for being behind on this. I'm interested in the future impersonation use cases. This seems to get us closer. +1 (non-binding) On Wed, Feb 8, 2017 at 4:41 AM, Manikumar wrote: > Hi Roger, > > In the current proposal, we only a

Re: [VOTE] KIP-48 Support for delegation tokens as an authentication mechanism

2017-02-07 Thread Roger Hoover
egation token". For impersonation, don't we need to be able to get tokens for other users besides the one making the request? Thanks, Roger On Tue, Feb 7, 2017 at 6:45 PM, Jun Rao wrote: > Hi, Roger, > > Just to clarify. This KIP already allows you to do impersonation at the &g

Re: [VOTE] KIP-48 Support for delegation tokens as an authentication mechanism

2017-02-07 Thread Roger Hoover
Just wondering...how difficult would be it be to later add impersonation ( https://issues.apache.org/jira/browse/KAFKA-3712)? One use case would be a Kafka admin UI that would take action on the cluster on behalf different users.I suppose we could later add an "effectiveUserId" (in Unix termin

Re: [DISCUSS] KIP-118: Drop Support for Java 7 in Kafka 0.11

2017-02-03 Thread Roger Hoover
This is great. Thanks, Ismael. On Fri, Feb 3, 2017 at 7:35 AM, Grant Henke wrote: > Looks good to me. Thanks for handling the KIP. > > On Fri, Feb 3, 2017 at 8:49 AM, Damian Guy wrote: > > > Thanks Ismael. Makes sense to me. > > > > On Fri, 3 Feb 2017 at 10:39 Ismael Juma wrote: > > > > > Hi

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

2017-01-24 Thread Roger Hoover
Thanks, Ismael. Just curious, why does it not make sense to do bcrypt it in the context of SCRAM? On Mon, Jan 23, 2017 at 3:54 PM, Ismael Juma wrote: > Hi Roger, > > SCRAM uses the PBKDF2 mechanism, here's a comparison between PBKDF2 and > bcrypt: > > http://sec

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

2017-01-23 Thread Roger Hoover
Sorry for the late question but is there a reason to choose SHA-1 and SHA-256 instead of bcrypt? https://codahale.com/how-to-safely-store-a-password/ On Fri, Nov 11, 2016 at 5:30 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > I think all the comments and suggestions on this thread h

[jira] [Closed] (KAFKA-4670) Kafka Consumer should validate FetchResponse

2017-01-19 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover closed KAFKA-4670. --- > Kafka Consumer should validate FetchRespo

[jira] [Commented] (KAFKA-4670) Kafka Consumer should validate FetchResponse

2017-01-19 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830670#comment-15830670 ] Roger Hoover commented on KAFKA-4670: - Ah, yes, thanks, [~ijuma]. > Kafka C

[jira] [Created] (KAFKA-4670) Kafka Consumer should validate FetchResponse

2017-01-18 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-4670: --- Summary: Kafka Consumer should validate FetchResponse Key: KAFKA-4670 URL: https://issues.apache.org/jira/browse/KAFKA-4670 Project: Kafka Issue Type: Bug

Re: [VOTE] KIP-108: Create Topic Policy

2017-01-09 Thread Roger Hoover
+1 (non-binding) On Mon, Jan 9, 2017 at 2:15 AM, Edoardo Comar wrote: > Ismael, > thanks for the KIP I see it as quite useful in a managed cloud > environment. > > +1 (non-binding) > -- > Edoardo Comar > IBM MessageHub > eco...@uk.ibm.com > IBM UK

Re: [DISCUSS] KIP-108: Create Topic Policy

2017-01-09 Thread Roger Hoover
Got it. Thanks, Ismael. On Mon, Jan 9, 2017 at 10:42 AM, Ismael Juma wrote: > Hi Roger, > > That's a good question. The server defaults are passed via the `configure` > method of the `Configurable` interface that is implemented by > `CreateTopicPolicy`. I'll mention t

Re: [DISCUSS] KIP-108: Create Topic Policy

2017-01-09 Thread Roger Hoover
This is great. Thanks, Ismael. One question. When TopicDetails are passed to the policy implementation, would the server defaults already have been merged? If not, I think the policy also needs access to the server defaults. Cheers, Roger On Fri, Jan 6, 2017 at 9:26 AM, Ismael Juma wrote

Re: KIP-103: Separation of Internal and External traffic

2017-01-06 Thread Roger Hoover
Thank you, Ismael. Sent from my iPhone > On Jan 6, 2017, at 4:46 PM, Ismael Juma wrote: > > Thanks Roger. I asked around and it seems like `listener name` is what > people found most intuitive in the context of configs. So, I have updated > the KIP to use that. > > Ismae

Re: KIP-103: Separation of Internal and External traffic

2017-01-06 Thread Roger Hoover
fix makes sense. Listeners must have a protocol but that ZooKeeper field is only meant to hold the listener ID. Cheers, Roger Sent from my iPhone > On Jan 6, 2017, at 12:24 PM, Ismael Juma wrote: > > Hi Roger, > > I think `listener_key` makes it clear that it has to be unique pe

Re: KIP-103: Separation of Internal and External traffic

2017-01-06 Thread Roger Hoover
Maybe it's clearer to to say protocol_listener_name? The proposed config allows you to name each listener and refer to their names in various places. On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma wrote: > Hi Colin, > > Thanks for the feedback. It's a good question regarding the name `protocol >

Re: [VOTE] KIP-103: Separation of Internal and External traffic

2017-01-06 Thread Roger Hoover
+1 (non-binding) On Fri, Jan 6, 2017 at 11:16 AM, Tom Crayford wrote: > +1 (non-binding) > > On Fri, Jan 6, 2017 at 6:58 PM, Colin McCabe wrote: > > > Looks good. +1 (non-binding). > > > > What do you think about changing "protocol label" to "listener key"? > > > > best, > > Colin > > > > > >

[jira] [Commented] (KAFKA-4527) Transient failure of ConnectDistributedTest.test_pause_and_resume_sink where paused connector produces messages

2016-12-18 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15759269#comment-15759269 ] Roger Hoover commented on KAFKA-4527: - Happened again: http://confluent-systes

[jira] [Commented] (KAFKA-3808) Transient failure in ReplicaVerificationToolTest

2016-12-18 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15759266#comment-15759266 ] Roger Hoover commented on KAFKA-3808: - Happened again: http://confluent-systes

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

2016-12-18 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15759264#comment-15759264 ] Roger Hoover commented on KAFKA-4166: - Similar failure: http://confluent-systes

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

2016-12-18 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15759260#comment-15759260 ] Roger Hoover commented on KAFKA-4526: - Thanks, [~apurva] > Transient fai

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-12-17 Thread Roger Hoover
ently (possibly just ignoring them). This would need to be added to the specification of the consumer protocol so that all Kafka clients implement it, right? I think it's a good idea but just checking. Cheers, Roger On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax wrote: > Yes and no. I did

[jira] [Commented] (KAFKA-3808) Transient failure in ReplicaVerificationToolTest

2016-12-17 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15758169#comment-15758169 ] Roger Hoover commented on KAFKA-3808: - Happened again: {

[jira] [Resolved] (KAFKA-4554) ReplicaVerificationToolTest.test_replica_lags system test failure

2016-12-17 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover resolved KAFKA-4554. - Resolution: Duplicate Duplicate of https://issues.apache.org/jira/browse/KAFKA-3808

[jira] [Created] (KAFKA-4554) ReplicaVerificationToolTest.test_replica_lags system test failure

2016-12-17 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-4554: --- Summary: ReplicaVerificationToolTest.test_replica_lags system test failure Key: KAFKA-4554 URL: https://issues.apache.org/jira/browse/KAFKA-4554 Project: Kafka

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

2016-12-17 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15758158#comment-15758158 ] Roger Hoover commented on KAFKA-4166: - Failed again: http://confluent-systes

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

2016-12-17 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15758155#comment-15758155 ] Roger Hoover commented on KAFKA-4526: - Failed again {

[jira] [Created] (KAFKA-4551) StreamsSmokeTest.test_streams intermittent failure

2016-12-16 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-4551: --- Summary: StreamsSmokeTest.test_streams intermittent failure Key: KAFKA-4551 URL: https://issues.apache.org/jira/browse/KAFKA-4551 Project: Kafka Issue Type

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

2016-12-16 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754984#comment-15754984 ] Roger Hoover commented on KAFKA-4166: - Happened again here: http://confl

[jira] [Commented] (KAFKA-4527) Transient failure of ConnectDistributedTest.test_pause_and_resume_sink where paused connector produces messages

2016-12-16 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754980#comment-15754980 ] Roger Hoover commented on KAFKA-4527: - Happened again: http://confluent-systes

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

2016-12-16 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754982#comment-15754982 ] Roger Hoover commented on KAFKA-4526: - Happened again here: http://confl

[jira] [Comment Edited] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

2016-12-13 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747454#comment-15747454 ] Roger Hoover edited comment on KAFKA-4166 at 12/14/16 6:3

[jira] [Comment Edited] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

2016-12-13 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747454#comment-15747454 ] Roger Hoover edited comment on KAFKA-4166 at 12/14/16 6:3

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

2016-12-13 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747454#comment-15747454 ] Roger Hoover commented on KAFKA-4166: - It happened twice more on these test: {

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

2016-12-13 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747308#comment-15747308 ] Roger Hoover commented on KAFKA-4526: - This happened again on the Dec 13 nightly

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-09 Thread Roger Hoover
on the higher-level questions though... Cheers, Roger On Wed, Nov 9, 2016 at 11:38 AM, radai wrote: > I think this discussion is getting a bit into the weeds on technical > implementation details. > I'd liek to step back a minute and try and establish where we are in the > la

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-09 Thread Roger Hoover
> I've been doing some bechmarking and it seems like the > > > speedup > > > >> for > > > >> >> >>>> using > > > >> >> >>>>>>> integer keys is about 2-5 depending on the length o

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-08 Thread Roger Hoover
I'll let others debate the performance tradeoffs of strings vs. ints. Roger On Tue, Nov 8, 2016 at 10:54 AM, Nacho Solis wrote: > From Roger's description: > > 5a- separate metadata field, built in serialization > 5c- separate metadata field, custom serialization > 5b- cus

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-07 Thread Roger Hoover
sons are to favor 5a over 5c. For the case of broker plugins, those plugins could also understand the common header format. Cheers, Roger On Mon, Nov 7, 2016 at 3:25 PM, Nacho Solis wrote: > ​Hey Roger. > > The original design involved: > 1- a header set per message (an array of ke

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-07 Thread Roger Hoover
, Roger On Sun, Nov 6, 2016 at 9:25 AM, radai wrote: > making header _key_ serialization configurable potentially undermines the > board usefulness of the feature (any point along the path must be able to > read the header keys. the values may be whatever and require more intimate > know

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-07 Thread Roger Hoover
Magnus, Thanks for jumping in. Do you see a reason that the broker should understand the header structure you've proposed? I'm wondering if metadata should just be bytes from the broker's point of view but clients could implement a common header serde spec on top. Cheers, Roger

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-07 Thread Roger Hoover
Please see comments inline. On Mon, Nov 7, 2016 at 9:32 AM, Michael Pearce wrote: > Hi Roger, > > Thanks for the support. > Thanks for leading the discussion on this. It's an important topic. > > I think the key thing is to have a common key space to make an ecosyste

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-03 Thread Roger Hoover
iendly clients would have a standard header data model to work with. 3. KIP is required both b/c of broker changes and because of client API changes. Cheers, Roger On Wed, Nov 2, 2016 at 4:38 PM, radai wrote: > my biggest issues with a "standard" wrapper format: > > 1. _ALL_

[jira] [Updated] (KAFKA-4361) Streams does not respect user configs for "default" params

2016-10-31 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-4361: Description: For the config params in CONSUMER_DEFAULT_OVERRIDES (https://github.com/apache/kafka

[jira] [Updated] (KAFKA-4361) Streams does not respect user configs for "default" params

2016-10-31 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-4361: Description: For the config params in CONSUMER_DEFAULT_OVERRIDES (https://github.com/apache/kafka

[jira] [Updated] (KAFKA-4361) Streams does not respect user configs for "default" params

2016-10-31 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-4361: Description: For the config params in CONSUMER_DEFAULT_OVERRIDES (https://github.com/apache/kafka

[jira] [Created] (KAFKA-4361) Streams does not respect user configs for "default" params

2016-10-31 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-4361: --- Summary: Streams does not respect user configs for "default" params Key: KAFKA-4361 URL: https://issues.apache.org/jira/browse/KAFKA-4361 Project: Kafka

Re: [VOTE] Add REST Server to Apache Kafka

2016-10-27 Thread Roger Hoover
ed-Flag → 0 / 1 # encoded as 1 byte unsigned integer Message-Length → {length of Message} # encoded as 4 byte unsigned integer Message → *{binary octet} Cheers, Roger On Wed, Oct 26, 2016 at 10:21 PM, Jaikiran Pai wrote: > -1. > > I would personally like Kafka core to be limited to the co

[jira] [Created] (KAFKA-4331) Kafka Streams resetter is slow because it joins the same group for each topic

2016-10-21 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-4331: --- Summary: Kafka Streams resetter is slow because it joins the same group for each topic Key: KAFKA-4331 URL: https://issues.apache.org/jira/browse/KAFKA-4331 Project

[jira] [Commented] (KAFKA-3993) Console producer drops data

2016-08-30 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450054#comment-15450054 ] Roger Hoover commented on KAFKA-3993: - Thanks, [~cotedm]. I tried to set acks

[jira] [Closed] (KAFKA-4063) Add support for infinite endpoints for range queries in Kafka Streams KV stores

2016-08-19 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover closed KAFKA-4063. --- The JIRA UI was unresponsive so I accidentally submitted the form twice. > Add support for infin

[jira] [Created] (KAFKA-4063) Add support for infinite endpoints for range queries in Kafka Streams KV stores

2016-08-18 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-4063: --- Summary: Add support for infinite endpoints for range queries in Kafka Streams KV stores Key: KAFKA-4063 URL: https://issues.apache.org/jira/browse/KAFKA-4063 Project

[jira] [Created] (KAFKA-4064) Add support for infinite endpoints for range queries in Kafka Streams KV stores

2016-08-18 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-4064: --- Summary: Add support for infinite endpoints for range queries in Kafka Streams KV stores Key: KAFKA-4064 URL: https://issues.apache.org/jira/browse/KAFKA-4064 Project

[jira] [Commented] (KAFKA-3993) Console producer drops data

2016-08-10 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415811#comment-15415811 ] Roger Hoover commented on KAFKA-3993: - Thanks, [~vahid]. Yeah, it looks the

[jira] [Updated] (KAFKA-3993) Console producer drops data

2016-08-10 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-3993: Description: The console producer drops data when if the process exits too quickly. I suspect

[jira] [Commented] (KAFKA-3752) Provide a way for KStreams to recover from unclean shutdown

2016-08-02 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404439#comment-15404439 ] Roger Hoover commented on KAFKA-3752: - [~guozhang] You're assessment seem

[jira] [Created] (KAFKA-3993) Console producer drops data

2016-07-26 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-3993: --- Summary: Console producer drops data Key: KAFKA-3993 URL: https://issues.apache.org/jira/browse/KAFKA-3993 Project: Kafka Issue Type: Bug Components

Re: [DISCUSS] Client Side Auto Topic Creation

2016-06-29 Thread Roger Hoover
APIs would only have a verify() method which checks if the spec is satisfied. Cheers, Roger On Wed, Jun 29, 2016 at 8:50 AM, Grant Henke wrote: > Thanks for the discussion, below are some thoughts and responses. > > One of the problems that we currently have with > > the client

[DISCUSS] KAFKA-3761: Controller has RunningAsBroker instead of RunningAsController state

2016-06-23 Thread Roger Hoover
erty, KafkaController.isActive which contains this information. Thanks, Roger

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

2016-06-21 Thread Roger Hoover
Thanks, Eno. On Tue, Jun 21, 2016 at 2:22 AM, Eno Thereska wrote: > Hi Roger, > > I realised I never got back to you on this one, sorry. Some answers inline: > > > On 3 Jun 2016, at 22:48, Roger Hoover wrote: > > > > Hi Eno, > > > > Does this mean t

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

2016-06-16 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334812#comment-15334812 ] Roger Hoover commented on KAFKA-3740: - [~h...@pinterest.com], any update on

[jira] [Created] (KAFKA-3858) Add functions to print stream topologies

2016-06-16 Thread Roger Hoover (JIRA)
Roger Hoover created KAFKA-3858: --- Summary: Add functions to print stream topologies Key: KAFKA-3858 URL: https://issues.apache.org/jira/browse/KAFKA-3858 Project: Kafka Issue Type: New Feature

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

2016-06-03 Thread Roger Hoover
with SIGKILL, will the data in the write buffer be lost (since Kafka Streams disables the transaction log)? That data will be present in the Kafka changelog but will it get applied to the recovered RocksDB database on restart? Thanks, Roger On Fri, Jun 3, 2016 at 2:39 PM, Eno Thereska wrote

[jira] [Work started] (KAFKA-3761) Controller has RunningAsBroker instead of RunningAsController state

2016-05-26 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3761 started by Roger Hoover. --- > Controller has RunningAsBroker instead of RunningAsController st

[jira] [Issue Comment Deleted] (KAFKA-3760) Set broker state as running after publishing to ZooKeeper

2016-05-26 Thread Roger Hoover (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-3760: Comment: was deleted (was: Create PR: https://github.com/apache/kafka/pull/1436) > Set bro

  1   2   >