Re: MirrorMaker improvements

2015-03-25 Thread Jiangjie Qin
hen the link is congested. On WAN >links, however, usually a single connection will cap (due to random losses >and high RTT), long before achieving the capacity of the link. Here is a >reference for this: >http://www.ece.virginia.edu/~mv/edu/715/lectures/TCP/padhye98modeling.pdf > >Rega

Re: [VOTE] KIP-15 add a close with timeout to new producer

2015-03-26 Thread Jiangjie Qin
/03/26/what-is-confusing-about-kafka/> of >confusing >Kafka behaviors, so I'd vote to log an appropriate error message and exit. > >On Wed, Mar 25, 2015 at 10:04 AM, Jiangjie Qin >wrote: > >> Hi Jay, >> >> The reason we keep the blocking behavior if clos

Re: [DISCUSSION] Keep docs updated per jira

2015-03-26 Thread Jiangjie Qin
On 3/26/15, 7:00 PM, "Neha Narkhede" wrote: >> >> Much much easier to do this if the docs are in git and can be reviewed >>and >> committed / reverted with the code (transactions makes synchronization >> easier...). +1 on this, too! > > >Huge +1. > >On Thu, Mar 26, 2015 at 6:54 PM, Joel Koshy

[CANCEL] [VOTE] KIP-15 add a close with timeout to new producer

2015-03-27 Thread Jiangjie Qin
t I do think it >>is a >> > bit >> > > weird. >> > > >> > > I think that likely very few people will call close from within a >> > callback >> > > so it probably isn't a huge issue one way or another. >&g

[VOTE] KIP-15 Add a close method with timeout to new producer

2015-03-27 Thread Jiangjie Qin
https://cwiki.apache.org/confluence/display/KAFKA/KIP-15+-+Add+a+close+method+with+a+timeout+in+the+producer Let us vote again!

[CLOSED] [VOTE] KIP-15 Add a close method with timeout to new producer

2015-03-27 Thread Jiangjie Qin
ntion that we are talking about >callback messages calling close(), and that we are preventing a >deadlock here. >The only reason I could see that "sender thread calls close()" refers >to the callbacks was because I followed the discussion on the list. > >Gwen > >On

Re: Review Request 31850: Patch for KAFKA-1660

2015-03-27 Thread Jiangjie Qin
rnals/RecordAccumulatorTest.java e379ac89c9a2fbfe750d6b0dec693b7eabb76204 core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 3df450784592b894008e7507b2737f9bb07f7bd2 Diff: https://reviews.apache.org/r/31850/diff/ Testing --- Unit tests passed. Thanks, Jiangjie Qin

Re: KIP Hangout - March 31, 2015 - TL;DR;

2015-03-31 Thread Jiangjie Qin
Thanks a lot for the summary, Gwen! About the Quota, does that mean the first quota implementation will be based on YM? I¹m thinking can we pursue a quota solution that has a loose coupling with metrics interfaces? Like something operating system does for FUSE, so we don¹t need to care about which

Re: KIP Hangout - March 31, 2015 - TL;DR;

2015-03-31 Thread Jiangjie Qin
cs. > >On Tue, Mar 31, 2015 at 08:44:44PM +, Jiangjie Qin wrote: >> Thanks a lot for the summary, Gwen! >> About the Quota, does that mean the first quota implementation will be >> based on YM? I¹m thinking can we pursue a quota solution that has a >>loose >> c

Re: Review Request 32781: Patch for KAFKA-2087

2015-04-02 Thread Jiangjie Qin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32781/#review78683 --- Ship it! Ship It! - Jiangjie Qin On April 2, 2015, 5:29 p.m

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-07 Thread Jiangjie Qin
s an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31850/#review79063 --- On March 27, 2015, 11:35 p.m., Jiangjie Qin wrote: > > --- > This is an auto

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-07 Thread Jiangjie Qin
rg/r/31850/diff/ Testing --- Unit tests passed. Thanks, Jiangjie Qin

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-08 Thread Jiangjie Qin
ead lock between sender thread and user thread when memory is full. Will submit another patch to address this. - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3185

Re: Review Request 33017: Fix timing issue in MetadataTest

2015-04-10 Thread Jiangjie Qin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33017/#review79756 --- Ship it! Ship It! - Jiangjie Qin On April 9, 2015, 3:35 p.m

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-10 Thread Jiangjie Qin
oducerSendTest.scala 9811a2b2b1e9bf1beb301138f7626e12d275a8db Diff: https://reviews.apache.org/r/31850/diff/ Testing --- Unit tests passed. Thanks, Jiangjie Qin

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-10 Thread Jiangjie Qin
called from callback. I added a comment to explain the tricky synchronization. - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31850/#review79677 -

Re: Review Request 33017: Fix timing issue in MetadataTest

2015-04-10 Thread Jiangjie Qin
> On April 10, 2015, 9:52 p.m., Jiangjie Qin wrote: > > Ship It! BTW, I think KafkaProducer has the same issue there. Could you update the code there as well? - Jiangjie --- This is an automatically generated e-mail. To rep

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-08 Thread Jiangjie Qin
b Diff: https://reviews.apache.org/r/31850/diff/ Testing --- Unit tests passed. Thanks, Jiangjie Qin

Re: Review Request 32890: Patch for conrolled shutdown rebase on trunk

2015-04-09 Thread Jiangjie Qin
r all the scenarios in this case. core/src/main/scala/kafka/server/ReplicaManager.scala <https://reviews.apache.org/r/32890/#comment128942> This seems not needed anymore after KAFKA-1647 is checked in, because we now always restore the HW on each makeLeader and makeFollower. Can y

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-11 Thread Jiangjie Qin
k this actually fixes things as > > the close could happen after the check. Please see previous reply. - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31850/#review79

Re: Review Request 33125: Fix timing issue in MetadataTest

2015-04-13 Thread Jiangjie Qin
/apache/kafka/clients/MetadataTest.java <https://reviews.apache.org/r/33125/#comment129487> Maybe add a comment explaining we do this to make sure update happen after metadata update is requestd? - Jiangjie Qin On April 13, 2015, 7:36 a.m., Rajini Sivaram

Re: Review Request 33135: Fix KAFKA-2117

2015-04-13 Thread Jiangjie Qin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33135/#review79907 --- Ship it! Ship It! - Jiangjie Qin On April 13, 2015, 3:51 p.m

[DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-04-13 Thread Jiangjie Qin
Hi, I just created a KIP to add a request timeout to NetworkClient for new Kafka clients. https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient Comments and suggestions are welcome! Thanks. Jiangjie (Becket) Qin

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-04-14 Thread Jiangjie Qin
have a server side request timeout exposed as a public configuration. We cannot set the client timeout smaller than that value, so a hard coded value probably won¹t work here. > >-Ewen > >On Mon, Apr 13, 2015 at 2:35 PM, Jiangjie Qin >wrote: > >> Hi, >> >> I just

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-04-15 Thread Jiangjie Qin
ot; wrote: >On Tue, Apr 14, 2015 at 1:57 PM, Jiangjie Qin >wrote: > >> Hi Ewen, thanks for the comments. Very good points! Please see replies >> inline. >> >> >> On 4/13/15, 11:19 PM, "Ewen Cheslack-Postava" wrote: >> >> >Jiang

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-04-16 Thread Jiangjie Qin
e decide to add " >REQUEST_TIMEOUT_CONFIG", I suggest we also make this renaming: admittedly > >it will change the config names but will reduce confusions moving >forward. > > >Guozhang > > >On Wed, Apr 15, 2015 at 6:48 PM, Jiangjie Qin > >w

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-16 Thread Jiangjie Qin
pi/ProducerSendTest.scala 9811a2b2b1e9bf1beb301138f7626e12d275a8db Diff: https://reviews.apache.org/r/31850/diff/ Testing --- Unit tests passed. Thanks, Jiangjie Qin

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-16 Thread Jiangjie Qin
eme is clever but non-obvious, is there a simpler way? > > Jiangjie Qin wrote: > I'm not sure if there is a simpler way. Maybe we can review the current > approach again and see if we can simplify them. > > The goals we want to achieve here are: >

Re: [DISCUSS] New consumer offset commit API

2015-04-16 Thread Jiangjie Qin
Hey Ewen, This makes sense. People usually do not want to stop consuming when committing offsets. One corner case about async commit with retries I am thinking is that it is possible that two offset commits interleave with each other and that might create problem. Like you said maybe we can canc

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-04-16 Thread Jiangjie Qin
also make this renaming: admittedly >it will change the config names but will reduce confusions moving forward. > > >Guozhang > > >On Wed, Apr 15, 2015 at 6:48 PM, Jiangjie Qin >wrote: > >> Checked the code again. It seems that the disconnected channel is not >> de

Re: Review Request 33239: Patch for KAFKA-2126

2015-04-16 Thread Jiangjie Qin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33239/#review80394 --- Ship it! Ship It! - Jiangjie Qin On April 15, 2015, 8:19 p.m

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-20 Thread Jiangjie Qin
Lock When I increase the thread number to 50. It drops to about 0.82M messages/second in all cases. It seems reader lock did not introduce performance issue. - Jiangjie --- This is an automatically generated e-mail.

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-20 Thread Jiangjie Qin
als/RecordAccumulatorTest.java 05e2929c0a9fc8cf2a8ebe0776ca62d5f3962d5c core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 9811a2b2b1e9bf1beb301138f7626e12d275a8db Diff: https://reviews.apache.org/r/31850/diff/ Testing --- Unit tests passed. Thanks, Jiangjie Qin

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-04-20 Thread Jiangjie Qin
; >Thanks, > >Jun > > >On Thu, Apr 16, 2015 at 1:02 PM, Jiangjie Qin >wrote: > >> Hi Harsha, >> >> Took a quick look at the patch. I think it is still a little bit >> different. KAFKA-1788 only handles the case where a batch sitting in >> accum

Review Request 33417: Patch for KAFKA-2138

2015-04-21 Thread Jiangjie Qin
/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java 05e2929c0a9fc8cf2a8ebe0776ca62d5f3962d5c Diff: https://reviews.apache.org/r/33417/diff/ Testing --- Thanks, Jiangjie Qin

Re: Review Request 33417: Patch for KAFKA-2138

2015-04-21 Thread Jiangjie Qin
ready. I'll try to submit a refactored patch and will throw questions if there is anything. Thanks. - Jiangjie ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33417/#review81097 ---

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-22 Thread Jiangjie Qin
d locks are very expensive. I am pretty worried about this. If we > > want to do this we need to do a pretty detailed examination of the perf > > impact. > > Jiangjie Qin wrote: > Hi Jay, I looked into the ReentrantReaderWriterLock implementation and it > seems un

Re: Review Request 33417: Patch for KAFKA-2138

2015-04-22 Thread Jiangjie Qin
c3a94 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java 05e2929c0a9fc8cf2a8ebe0776ca62d5f3962d5c Diff: https://reviews.apache.org/r/33417/diff/ Testing --- Thanks, Jiangjie Qin

Review Request 33552: Patch for KAFKA-2142

2015-04-25 Thread Jiangjie Qin
b1d5b7f2b4091040bdcfb0a60fd5879f45a0 core/src/test/resources/log4j.properties 1b7d5d8f7d5fae7d272849715714781cad05d77b Diff: https://reviews.apache.org/r/33552/diff/ Testing --- Thanks, Jiangjie Qin

Re: Review Request 33552: Patch for KAFKA-2142

2015-04-25 Thread Jiangjie Qin
1b7d5d8f7d5fae7d272849715714781cad05d77b Diff: https://reviews.apache.org/r/33552/diff/ Testing (updated) --- Unit Test passed. Thanks, Jiangjie Qin

Re: Review Request 33552: Patch for KAFKA-2142

2015-04-25 Thread Jiangjie Qin
/log4j.properties 1b7d5d8f7d5fae7d272849715714781cad05d77b Diff: https://reviews.apache.org/r/33552/diff/ Testing --- Unit Test passed. Thanks, Jiangjie Qin

Re: Review Request 33552: Patch for KAFKA-2142

2015-04-25 Thread Jiangjie Qin
, Jiangjie Qin

Re: Review Request 33552: Patch for KAFKA-2142

2015-04-25 Thread Jiangjie Qin
/log4j.properties 1b7d5d8f7d5fae7d272849715714781cad05d77b Diff: https://reviews.apache.org/r/33552/diff/ Testing --- Unit Test passed. Thanks, Jiangjie Qin

Re: Review Request 33557: Patch for KAFKA-1936

2015-04-26 Thread Jiangjie Qin
/test/scala/unit/kafka/server/OffsetCommitTest.scala <https://reviews.apache.org/r/33557/#comment132081> Typo, larger. - Jiangjie Qin On April 25, 2015, 11:36 p.m., Dong Lin wrote: > > --- > This is an automatically gener

Re: Review Request 33557: Patch for KAFKA-1936

2015-04-26 Thread Jiangjie Qin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33557/#review81657 --- Ship it! Ship It! - Jiangjie Qin On April 27, 2015, 4:26 a.m

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-29 Thread Jiangjie Qin
lients/producer/internals/RecordAccumulatorTest.java baa48e7c1b7ac5da8f3aca29f653c3fff88f8009 core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 9811a2b2b1e9bf1beb301138f7626e12d275a8db Diff: https://reviews.apache.org/r/31850/diff/ Testing --- Unit tests passed. Thanks, Jiangjie Qin

Re: Review Request 31850: Patch for KAFKA-1660

2015-04-29 Thread Jiangjie Qin
it tests passed. Thanks, Jiangjie Qin

Re: Review Request 33552: Patch for KAFKA-2142

2015-05-02 Thread Jiangjie Qin
--- Unit Test passed. Thanks, Jiangjie Qin

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-04 Thread Jiangjie Qin
(and maybe change the configuration name of TIMEOUT_CONFIG to REPLICATION_TIMEOUT_CONFIG). Would like to see what people think of above approach? Jiangjie (Becket) Qin On 4/20/15, 6:02 PM, "Jiangjie Qin" wrote: >Jun, > >I thought a little bit differently on this. >Intu

Re: Review Request 33552: Patch for KAFKA-2142

2015-05-04 Thread Jiangjie Qin
sting --- Unit Test passed. Thanks, Jiangjie Qin

Re: Review Request 33552: Patch for KAFKA-2142

2015-05-04 Thread Jiangjie Qin
est was sent to a broker that is down. The be producer will wait for request.timeout without sending any further data. I'll revert this back. - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apa

Re: Review Request 33552: Patch for KAFKA-2142

2015-05-04 Thread Jiangjie Qin
s/src/test/resources/log4j.properties b1d5b7f2b4091040bdcfb0a60fd5879f45a0 core/src/test/resources/log4j.properties 1b7d5d8f7d5fae7d272849715714781cad05d77b Diff: https://reviews.apache.org/r/33552/diff/ Testing --- Unit Test passed. Thanks, Jiangjie Qin

Re: Review Request 33552: Patch for KAFKA-2142

2015-05-04 Thread Jiangjie Qin
https://reviews.apache.org/r/33552/diff/ Testing --- Unit Test passed. Thanks, Jiangjie Qin

Re: Review Request 33552: Patch for KAFKA-2142

2015-05-04 Thread Jiangjie Qin
rTest.java 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 clients/src/test/resources/log4j.properties b1d5b7f2b4091040bdcfb0a60fd5879f45a0 core/src/test/resources/log4j.properties 1b7d5d8f7d5fae7d272849715714781cad05d77b Diff: https://reviews.apache.org/r/33552/diff/ Testing --- Unit Test passed. Thanks, Jiangjie Qin

[DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

2015-05-05 Thread Jiangjie Qin
I am trying to see if we can reuse the NetworkClient class to be used in controller to broker communication. (Also, we can probably use KafkaConsumer which is already using NetworkClient in replica fetchers). Currently NetworkClient does the following things in addition to sending requests. 1

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-06 Thread Jiangjie Qin
Yes, that is the plan. On 5/5/15, 8:23 PM, "Mayuresh Gharat" wrote: >Just a quick question, can we handle REQUEST TIMEOUT as disconnections and >do a fresh MetaDataRequest and retry instead of failing the request? > > >Thanks, > >Mayuresh > > >On Mo

Re: [DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

2015-05-06 Thread Jiangjie Qin
umer. >>However, >> we don't need the metadata and the offset management. So, perhaps it's >> easier to just use NetworkClient. Also, currently, there is one replica >> fetcher thread per source broker. By using NetworkClient, we can change >> that to usi

Re: [DISCUSSION] Reuse o.a.k.clients.NetworkClient in controller.

2015-05-06 Thread Jiangjie Qin
complicated to me. >Instead, we could completely remove the concept of Metadata from >KafkaClient, such that NetworkClient.handleCompletedReceives does not >specifically handle metadata responses, but just call >response.request().callback().onComplete(response) as well, which will

Why we call callback.onCompletion in send()?

2015-05-07 Thread Jiangjie Qin
Hi, I just notice that in new producer we actually call callback.onCompleteion() in the catch clause of send(). This seems breaking the callback execution order guarantee we provided. Any reason why we do this instead of just throw an exception? Jiangjie (Becket) Qin

Re: Review Request 33065: Patch for KAFKA-1928

2015-05-08 Thread Jiangjie Qin
/Sender.java <https://reviews.apache.org/r/33065/#comment133063> Not sure why we are checking if the node is non-negative. But the semantic meaning of the code here has changed. - Jiangjie Qin On May 1, 2015, 10:45 p.m., Gwen Shapira

Review Request 34070: Patch for KAFKA-2186

2015-05-11 Thread Jiangjie Qin
: https://reviews.apache.org/r/34070/diff/ Testing --- Thanks, Jiangjie Qin

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-12 Thread Jiangjie Qin
ain the old behavior >precisely, although maybe for some code it is an issue if they start >seeing >timeout exceptions that they wouldn't have seen before? > >-Ewen > >On Wed, May 6, 2015 at 6:06 PM, Jun Rao wrote: > >> Jiangjie, >> >> Yes, I think using

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

2015-05-12 Thread Jiangjie Qin
Hey Harsha, It looks you created the KIP page at wrong place. . . Can you move the page to a child page of https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposa ls Thanks. Jiangjie (Becket) Qin On 5/6/15, 6:12 PM, "Harsha" wrote: >Thanks for the review Joel. I agree don'

Re: Review Request 31850: Patch for KAFKA-1660

2015-05-12 Thread Jiangjie Qin
ore/src/test/scala/integration/kafka/api/ProducerSendTest.scala 9811a2b2b1e9bf1beb301138f7626e12d275a8db Diff: https://reviews.apache.org/r/31850/diff/ Testing --- Unit tests passed. Thanks, Jiangjie Qin

Add missing API to old high level consumer

2015-05-12 Thread Jiangjie Qin
Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(Map offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala clas

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

2015-05-12 Thread Jiangjie Qin
in+the+new+producer >I checked other KIPS they are under /KAFKA as well. > >Thanks, >Harsha >On May 12, 2015 at 2:12:30 PM, Jiangjie Qin (j...@linkedin.com.invalid) >wrote: > >Hey Harsha, > >It looks you created the KIP page at wrong place. . . Can you move the >page

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

2015-05-12 Thread Jiangjie Qin
other KIPS they are under /KAFKA as well. > >Thanks, >Harsha >On May 12, 2015 at 2:12:30 PM, Jiangjie Qin (j...@linkedin.com.invalid) >wrote: > >Hey Harsha, > >It looks you created the KIP page at wrong place. . . Can you move the >page to a child page of >https://c

Re: Message corruption with new Java client + snappy + broker restart

2015-05-12 Thread Jiangjie Qin
Does this related to KAFKA-2189? Jiangjie (Becket) Qin On 5/12/15, 9:41 PM, "Roger Hoover" wrote: >Hi, > >When using Samza 0.9.0 which uses the new Java producer client and snappy >enabled, I see messages getting corrupted on the client side. It never >happens with the old producer and it neve

[DISCUSS] Add missing API to old high level consumer

2015-05-13 Thread Jiangjie Qin
Add the DISCUSS prefix to the email title : ) From: Jiangjie Qin mailto:j...@linkedin.com>> Date: Tuesday, May 12, 2015 at 4:51 PM To: "dev@kafka.apache.org<mailto:dev@kafka.apache.org>" mailto:dev@kafka.apache.org>> Subject: Add missing API to old high level consum

Review Request 34241: Patch for KAFKA-2190

2015-05-14 Thread Jiangjie Qin
/diff/ Testing --- Thanks, Jiangjie Qin

Re: Can we throw away "ResponsesBeingSent" Gauge?

2015-05-14 Thread Jiangjie Qin
I think we probably can drop this completely. The new selector should have provided enough metrics to describe the IO thread status. And it is async anyway, so ³being sent² does not look really helpful. Jiangjie (Becket) Qin On 5/14/15, 1:06 PM, "Todd Palino" wrote: >I don't believe we're using

Re: [DISCUSS] Add missing API to old high level consumer

2015-05-14 Thread Jiangjie Qin
g to phase out the >scala clients then we need to strive to not be making changes to them on >trunk. > >~ Joe Stein >- - - - - - - - - - - - - - - - - > > http://www.stealth.ly >- - - - - - - - - - - - - - - - - > >On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin >wr

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-15 Thread Jiangjie Qin
(Becket) Qin On 5/12/15, 2:59 PM, "Mayuresh Gharat" wrote: >+1 Becket. That would give enough time for clients to move. We should make >this change very clear. > >Thanks, > >Mayuresh > >On Tue, May 12, 2015 at 1:45 PM, Jiangjie Qin >wrote: > >&g

Re: [DISCUSS] Add missing API to old high level consumer

2015-05-15 Thread Jiangjie Qin
we agreed that it will include at least the >new java consumer. If other features (e.g., security, admin, etc) can be >done at or before that, they will be part of the 0.8.3 release too. >Updated >the release plan wiki. > >Thanks, > >Jun > > >On Wed, May 13, 2015

Re: Review Request 34241: Patch for KAFKA-2190

2015-05-15 Thread Jiangjie Qin
954852170d67cbb8ff4f113301816d2a2daf5e91 Diff: https://reviews.apache.org/r/34241/diff/ Testing --- Thanks, Jiangjie Qin

[DISCUSSION] Two behaviors in new producer

2015-05-18 Thread Jiangjie Qin
Hi, I have two questions when writing patch for KAFKA-2142 and needs some help: 1. In send(), we actually might call callback.onCompletion(), this might break the guarantee that callbacks are fired in sending order of a particular partition. 2. In RecordAccumulator, current logic is that

Re: [DISCUSSION] Two behaviors in new producer

2015-05-18 Thread Jiangjie Qin
ortunity is large here we can discuss if it's worth trying to fix and >what gets worse. > >-Jay > > >On Mon, May 18, 2015 at 3:44 PM, Jiangjie Qin >wrote: > >> Hi, >> >> I have two questions when writing patch for KAFKA-2142 and needs some >>h

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-19 Thread Jiangjie Qin
o that end I suggest we do two things: >> 1. Include KAKFA-1788. I know that technically these two things are >> different but from the user's point of view they aren't. >> 2. Include in the KIP the explanation to the user of the full set of >> timeouts, what they

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-19 Thread Jiangjie Qin
because connections are established when we >>want >> to >> > send a request and presumably we will begin the timer then? >> > >> > To that end I suggest we do two things: >> > 1. Include KAKFA-1788. I know that technically these two things are >> &

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-19 Thread Jiangjie Qin
e user and it >requires the client to understand the internals of kafka. We should have a >single timeout from the users perspective and handle other timeouts >internally like a batch timeout. > >Mayuresh > >On Tue, May 19, 2015 at 12:42 PM, Jiangjie Qin >wrote: > >>

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-19 Thread Jiangjie Qin
>something like base the request time off the oldest batch. Let me think >about the implications of that, it's definitely a problem. > >-Jay > >On Tue, May 19, 2015 at 12:42 PM, Jiangjie Qin >wrote: > >> Hey Jay, >> >> That is also a viable solution.

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-20 Thread Jiangjie Qin
ime the >request spends in the accumulator vs. some other component (assuming they >have the overall timeout)? Same for requests in flight, as long as I have >that client side timeout? And if they care about what component is the >bottleneck, could that be better exposed by the exceptions

Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

2015-05-20 Thread Jiangjie Qin
I actually feel many [VOTE] threads eventually become [DISCUSS] as people just put tons of comments there :) On 5/20/15, 11:52 AM, "Jay Kreps" wrote: >Makes sense. Honghai, want to do a [VOTE] thread just so everything is >official? > >-Jay > >On Wed, May 20, 2015 at 11:22 AM, Gwen Shapira >wr

Review Request 34569: Patch for KAFKA-2186

2015-05-21 Thread Jiangjie Qin
/browse/KAFKA-2186 Repository: kafka Description --- rebased on trunk Diffs - core/src/main/scala/kafka/javaapi/consumer/ConsumerConnector.java cc3400ff81fc0db69b5129ad7b440f20a211a79d Diff: https://reviews.apache.org/r/34569/diff/ Testing --- Thanks, Jiangjie Qin

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-05-21 Thread Jiangjie Qin
r use cases) before adding such >configs. We should also consider how we can simplify or even eliminate >existing configs. > >Re: requests in flight may be a good example: Becket had given a valid >use-case i.e., support strict ordering. Maybe we can replace it with a >"ena

Re: [VOTE] KIP-21 Dynamic Configuration

2015-05-22 Thread Jiangjie Qin
+1 to the updated wiki (non-binding) On 5/19/15, 2:26 PM, "Andrii Biletskyi" wrote: >Hi, > >Sorry I wasn't able to participate. I don't have objections about removing >config changes from AlterTopic (as I understand both AddedConfig and >DeletedConfig) - you are welcome to update the KIP page. >

[DISCUSSION] Can we move the ack timeout in ProducerRequest to broker?

2015-05-26 Thread Jiangjie Qin
Hi, I am updating the wiki for KIP-19 and wondering why we have a replication timeout on producer side and in producer request? >From what I understand this is a server side setting and the reasons we need >this replication timeout is because we want to control the purgatory size. If >that is

Re: [DISCUSSION] Can we move the ack timeout in ProducerRequest to broker?

2015-05-28 Thread Jiangjie Qin
f the zone. >>So, >> I am not sure if it's better to just reason about the replication time >>as a >> single timeout on the broker side. In theory, different producers may >>want >> to pick different replication time depending on the topics being sent. >&

Re: KIP Wiki

2015-06-01 Thread Jiangjie Qin
+1 On 6/1/15, 11:53 AM, "Ashish Singh" wrote: >I like the idea! > > >On Mon, Jun 1, 2015 at 9:51 AM, Aditya Auradkar < >aaurad...@linkedin.com.invalid> wrote: > >> Hey everyone, >> >> We have enough KIP's now (25) that it's a bit hard to tell which ones >>are >> adopted or under discussion by gl

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-06-01 Thread Jiangjie Qin
(Becket) Qin On 5/21/15, 5:54 PM, "Jiangjie Qin" wrote: >Based on the discussion we have, I just updated the KIP with the following >proposal and want to see if there is further comments. > >The proposal is to have the following four timeout as end state. > >1. max.buffer.

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

2015-06-02 Thread Jiangjie Qin
t;partitionsFor. >> Are we converting that to use MAX_ENQUEUE_TIMEOUT_MS_CONFIG? The naming >>is >> a bit awkward, but we need to use something there. >> >> Finally, just a nit, but the naming conventions for variables are >>getting >> inconsistent. Some h

[VOTE] KIP-19 Add a request timeout to NetworkClient

2015-06-02 Thread Jiangjie Qin
Hi folks, Thanks a lot for all the input and help. It looks we do not have further concerns on KIP-19. I’ve updated the wiki page, let’s vote! https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient Thanks, Jiangjie (Becket) Qin

Re: [VOTE] KIP-19 Add a request timeout to NetworkClient

2015-06-02 Thread Jiangjie Qin
Ah, yes. Just changed it to ³request.timeout.ms². Thanks. Jiangjie (Becket) Qin On 6/2/15, 2:31 PM, "Jun Rao" wrote: >The wiki references "network.request.timeout.ms" in ProducerConfig. It >should be "request.timeout.ms", right? > >Thanks, > >

Re: [VOTE] KIP-19 Add a request timeout to NetworkClient

2015-06-03 Thread Jiangjie Qin
;On Tue, Jun 2, 2015 at 3:13 PM, Jiangjie Qin >wrote: > >> Ah, yes. Just changed it to ³request.timeout.ms². >> >> Thanks. >> >> Jiangjie (Becket) Qin >> >> On 6/2/15, 2:31 PM, "Jun Rao" wrote: >> >> >The wiki references "

Review Request 35023: Patch for KAFKA-2246

2015-06-03 Thread Jiangjie Qin
: https://reviews.apache.org/r/35023/diff/ Testing --- Thanks, Jiangjie Qin

Re: [DISCUSS] KIP-25 System test improvements

2015-06-04 Thread Jiangjie Qin
Hey Gwen, Currently the test and code are downloaded at the same time. Supposedly the tests in the same repository should cover match the code. Are you saying people downloaded a release from some random place and want to verify it? If that is the case, does that mean people still need to find the

Review Request 30063: Patch for KAFKA-1840

2015-01-19 Thread Jiangjie Qin
rg/r/30063/diff/ Testing --- Thanks, Jiangjie Qin

Review Request 30083: Patch for KAFKA-1839

2015-01-20 Thread Jiangjie Qin
24954de66ccc5158696166b7e2aabad0f1b1f287 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala a17e8532c44aadf84b8da3a57bcc797a848b5020 Diff: https://reviews.apache.org/r/30083/diff/ Testing --- Thanks, Jiangjie Qin

Re: Review Request 30083: Patch for KAFKA-1839

2015-01-20 Thread Jiangjie Qin
://reviews.apache.org/r/30083/diff/ Testing --- Thanks, Jiangjie Qin

Re: Review Request 30063: Patch for KAFKA-1840

2015-01-20 Thread Jiangjie Qin
rrorMaker.scala 5cbc8103e33a0a234d158c048e5314e841da6249 Diff: https://reviews.apache.org/r/30063/diff/ Testing --- Thanks, Jiangjie Qin

  1   2   3   4   5   6   7   8   9   10   >