Re: [DISCUSSION] KIP-15 close(timeout) for producer

t is not a public interface. I mentioned it in Proposed Changes section which I thought was supposed to provide implementation details. Thanks again for all the comments and suggestions! Jiangjie (Becket) Qin On 3/10/15, 8:57 PM, "Jiangjie Qin" wrote: >The KIP page has been

Re: [DISCUSSION] KIP-15 close(timeout) for producer

>just use close(-1)? > >5. Proposed Changes => it seems you only talked about the close(-1) case, >how about close(>=0)? > >Guozhang > >On Thu, Mar 12, 2015 at 1:49 AM, Jiangjie Qin >wrote: > >> Hey Joe & Jay, >> >> Thanks for the comments on

Re: [DISCUSSION] KIP-15 close(timeout) for producer

-1660, my understanding is it is. Any idea? Jiangjie (Becket) Qin On 3/12/15, 11:50 AM, "Jiangjie Qin" wrote: >Hey Guozhang, > >Thanks for the comments. I updated the page as suggested. >For 3), that’s right, I put this in java doc. Do you think we need to >reject value

Re: [DISCUSSION] KIP-15 close(timeout) for producer

he question is exactly whether there would be a motivational use case we >can brainstorm for those additional use cases? > >-Jay > >On Thu, Mar 12, 2015 at 4:24 PM, Jiangjie Qin >wrote: > >> Hi folks, >> >> Since there are actually a long interleaved discussion

Re: Review Request 31706: Patch for KAFKA-1997

3e core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 543070f4fd3e96f3183cae9ee2ccbe843409ee58 core/src/test/scala/unit/kafka/consumer/ZookeeperConsumerConnectorTest.scala 19640cc55b5baa0a26a808d708b7f4caf491c9f0 Diff: https://reviews.apache.org/r/31706/diff/ Testing --

Re: Review Request 31735: Fix KAFKA-1863

c/main/java/org/apache/kafka/common/protocol/types/Schema.java <https://reviews.apache.org/r/31735/#comment123971> Constrcut the schema with a given list of its field values. - Jiangjie Qin On March 4, 2015, 6:39 p.m., Guozhan

Re: Review Request 32063: Patch for KAFKA-2019

tps://reviews.apache.org/r/32063/#comment124013> Can you put this logic only in PartitionAssignor? zkUtil is used cross the board. - Jiangjie Qin On March 14, 2015, 12:13 a.m., Joseph Holsten wrote: > > --- > This is an automatically g

Re: [DISCUSSION] KIP-15 close(timeout) for producer

e logs with >repeating errors until someone gets notified. How about >error-message-and-close(-1), i.e. record the error and force shutdown? > >On Thu, Mar 12, 2015 at 11:50 AM, Jiangjie Qin >wrote: > >> Hey Guozhang, >> >> Thanks for the comments. I updated the page

Re: [DISCUSSION] KIP-15 close(timeout) for producer

rom them which is the opposite of what we >want. > >-Jay > >On Thu, Mar 12, 2015 at 1:49 AM, Jiangjie Qin >wrote: > >> Hey Joe & Jay, >> >> Thanks for the comments on the voting thread. Since it seems we probably >> will have more discussion on this

Re: [DISCUSSION] KIP-15 close(timeout) for producer

blocking is good or alternately throwing an exception directly >from >close(). Basically I would just worry about subtly doing something >slightly >different from what the user asked for as it will be hard to notice that >behavior difference. > >-Jay > >On Sat, Mar 14, 2015 at 5:48

Re: [DISCUSSION] KIP-15 close(timeout) for producer

Basically I would just worry about subtly doing something >> slightly >> > different from what the user asked for as it will be hard to notice >>that >> > behavior difference. >> > >> > -Jay >> > >> > On Sat, Mar 14, 2015 at 5:48 PM,

Re: [DISCUSSION] KIP-15 close(timeout) for producer

Guozhang > >On Mon, Mar 16, 2015 at 9:52 AM, Jiangjie Qin >wrote: > >> It seems there are two options we can choose from when close() is called >> from sender thread (callback): >> 1. Log an error and close the producer using close(-1) >> 2. Log an error and bl

Re: [DISCUSSION] KIP-15 close(timeout) for producer

locking. The immediate shutdown approach does have some >unsafeness in some cases. > >Guozhang > >On Mon, Mar 16, 2015 at 11:50 AM, Jiangjie Qin >wrote: > >> It looks that the problem we want to solve and the purpose we want to >> achieve is: >> If user use

Re: [DISCUSSION] KIP-15 close(timeout) for producer

;s called from the sender thread? If close(0) >needs to wait for the sender thread to join, wouldn't this cause a >deadlock? > >Thanks, > >Jun > >On Mon, Mar 16, 2015 at 2:26 PM, Jiangjie Qin >wrote: > >> Thanks Guozhang. It wouldn’t be as thoroughly consider

Re: [DISCUSSION] KIP-15 close(timeout) for producer

un Rao wrote: > >> Hmm, does that mean that after close(0), the sender thread is not >>necessary >> gone? Normally, after closing an entity, we expect all internal threads >> associated with the entity are shut down completely. >> >> Thanks, >> >> J

Re: Review Request 31706: Patch for KAFKA-1997

) - core/src/main/scala/kafka/tools/MirrorMaker.scala 87b925c7332470de59a5b098d51c4bf0dfab8b8f Diff: https://reviews.apache.org/r/31706/diff/ Testing --- Thanks, Jiangjie Qin

Re: [Discussion] Using Client Requests and Responses in Server

I think those two requests are only used by controller to broker communication. Not sure if client side will need them in KIP-4, unlikely I guess. Jiangjie (Becket) Qin On 3/17/15, 6:08 PM, "Gwen Shapira" wrote: >Hi, > >I'm starting this thread for the various questions I run into while >refact

Review Request 32193: Patch for KAFKA-1997

, Jiangjie Qin

Re: Review Request 32199: Patch for KAFKA-1858

does it make sense to add a non deamon thread check in each test? So we can make sure there is no deamon thread left from some tests. - Jiangjie Qin On March 18, 2015, 11:12 a.m., Adrian Preston wrote: > > --- > This is an auto

Re: Review Request 32193: Patch for KAFKA-1997

://reviews.apache.org/r/32193/diff/ Testing --- Thanks, Jiangjie Qin

Re: [DISCUSSION] KIP-15 close(timeout) for producer

code, assuming users are always calling producer.close() as a clean up step, personally I prefer approach 2) as it follows convention. Any thoughts? Jiangjie (Becket) Qin On 3/17/15, 10:25 AM, "Jiangjie Qin" wrote: >Hi Jun, > >Yes, as Guozhang said, the main reason we se

Re: [DISCUSSION] KIP-15 close(timeout) for producer

>> >> On Wed, Mar 18, 2015 at 2:52 PM, Jay Kreps wrote: >> >> > Personally I'm in favor of (1) just to reduce the number of different >>APIs. >> > People will find the difference between abort and close subtle and >> > confusing and

Re: [DISCUSSION] KIP-15 close(timeout) for producer

ds, that is confusing enough. Adding a >third won't help. I understand that having a few microseconds before the >thread shuts down could be unexpected but I think there is nothing >inherently wrong with that. > >-Jay > >On Thu, Mar 19, 2015 at 3:09 PM, Jiangjie Qin >w

Re: [VOTE] KIP-15 add a close method with timeout to KafkaProducer

this is different than >> flush(time); close() and why close has its own timeout also? >> >> Lastly, what is the forceClose flag? This isn't documented in the public >> interface so it isn't clear how to completely use the feature just by >> reading the KIP.

Re: [DISCUSSION] KIP-15 close(timeout) for producer

ther a close() call is made from the sender thread or not? >> >> Thanks, >> >> Jun >> >> On Wed, Mar 18, 2015 at 2:13 PM, Jiangjie Qin >> >> wrote: >> >> > It looks we have another option and are now deciding between the >> following >&

Re: [DISCUSSION] KIP-15 close(timeout) for producer

5, Jun Rao wrote: >> > >> >> So in (1), if a close() or close(timeout) is called from a callback, >>we >> >> will just turn that into a close(0)? Implementation wise, how do we >>know >> >> whether a close() call is made from the sender thre

Review Request 32434: Patch for KAFKA-2042

://reviews.apache.org/r/32434/diff/ Testing --- Thanks, Jiangjie Qin

Re: [VOTE] KIP-15 add a close method with timeout to KafkaProducer

Push up the thread for voting after discussion on the KIP hangout. On 3/19/15, 9:03 PM, "Jiangjie Qin" wrote: >We had some additional discussions on the discussion thread. Pushing up >this thread to resume voting. > >On 3/11/15, 8:47 PM, "Jay Kreps" wrote: &

Re: Review Request 32434: Patch for KAFKA-2042

feda9c922d7dab17e424f8e6f0aa0a3f968e3729 Diff: https://reviews.apache.org/r/32434/diff/ Testing --- Thanks, Jiangjie Qin

Re: Review Request 32434: Patch for KAFKA-2042

c8bde8b732defa20819730d87303a9a80d01116f clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java feda9c922d7dab17e424f8e6f0aa0a3f968e3729 Diff: https://reviews.apache.org/r/32434/diff/ Testing --- Thanks, Jiangjie Qin

[VOTE] KIP-15 add a close with timeout to new producer

https://cwiki.apache.org/confluence/display/KAFKA/KIP-15+-+Add+a+close+method+with+a+timeout+in+the+producer As a short summary, the new interface will be as following: Close(Long Timeout, TimeUnit timeUnit) 1. When timeout > 0, it will try to wait up to timeout for the sender thread to compl

Re: Review Request 32434: Patch for KAFKA-2042

y, visit: https://reviews.apache.org/r/32434/#review77653 --- On March 24, 2015, 8:57 p.m., Jiangjie Qin wrote: > > --- > This is an automatically generated e-mail.

[CANCEL] [VOTE] KIP-15 add a close method with timeout to KafkaProducer

d start a new thread. >> >> On Tue, Mar 24, 2015 at 12:19 PM, Jiangjie Qin >> wrote: >> > Push up the thread for voting after discussion on the KIP hangout. >> > >> > On 3/19/15, 9:03 PM, "Jiangjie Qin" wrote: >> > >>

Re: KAFKA-2042

Hi Jun, This issue does not only affect flush(). It is just with flush() the probability is much higher. It will affect the following scenario: 1. Producer started and refreshed metadata. 2. User call producer.send() to send message 1 to topic A, but message A is in accumulator. 3. User call produ

Review Request 32465: Patch for KAFKA-2047

/diff/ Testing --- Thanks, Jiangjie Qin

Re: [VOTE] KIP-15 add a close with timeout to new producer

gt; >-Jay > >On Wed, Mar 25, 2015 at 9:17 AM, Jay Kreps wrote: > >> +1 >> >> -Jay >> >> On Tue, Mar 24, 2015 at 2:43 PM, Guozhang Wang >>wrote: >> >>> +1. >>> >>> On Tue, Mar 24, 2015 at 2:15 PM, Jiangjie Qin >>

Re: MirrorMaker improvements

Hi Jun, I think bumping up socket.receive.buffer.bytes will help but might not be sufficient. There are actually two related factors here: 1. Pipelining TCP packets when send a single request/response. 2. Pipelining multiple requests/responses Bumping up socket.receive.buffer.bytes help with the 1)

Re: Review Request 31850: Patch for KAFKA-1660

.java e379ac89c9a2fbfe750d6b0dec693b7eabb76204 core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 3df450784592b894008e7507b2737f9bb07f7bd2 Diff: https://reviews.apache.org/r/31850/diff/ Testing --- Unit tests passed. Thanks, Jiangjie Qin

Re: Review Request 32465: Patch for KAFKA-2047

32465/diff/ Testing --- Thanks, Jiangjie Qin

Re: Review Request 32465: Patch for KAFKA-2047

ally generated e-mail. To reply, visit: https://reviews.apache.org/r/32465/#review77789 ------- On March 25, 2015, 8:48 p.m., Jiangjie Qin wrote: > > --- > This is an au

Re: MirrorMaker improvements

om a single consumer to a >broker. > >For identical mirroring, I thought that when asking for data between a >pair >of offsets the result should always be the same. Would it be possible to >produce also indicating the offsets where the data should go? > >Regards, >Vlad > &

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

I updated KIP-32 wiki with Anna's suggestion on ProduceResponse format change. Because it looks everyone also has a +1 on ProduceResponse format change, I take this update passed as well. KIP-32 passed with +6(binding). Thanks a lot for everyone's participation and all the comments. BTW, the pa

Re: Review Request 35187: Fix KAFKA-2253

e part into tryCompleteWatched/purgeComplete? This code block can be simplified to: inWriteLock(removeWatchersLock) { if (watchersForKey.get(key) == this) watchersForKey.remove(key) } - Jiangjie Qin On June 7, 2015, 4:49 a.m., Guozhang Wang

Re: Review Request 35201: Fix KAFKA-2253

watchers. In tryCompleteWatched, we synchronize on the watcher first then grab the removeWatchersLock. Will there be a deadlock? - Jiangjie Qin On June 8, 2015, 4:46 a.m., Guozhang Wang wrote: > > --- > This is an automatically generated e-

Re: Review Request 35201: Fix KAFKA-2253

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35201/#review87065 --- Ship it! Ship It! - Jiangjie Qin On June 8, 2015, 6:47 p.m

Re: [Discussion] New Consumer API / Protocol

Thanks for kicking off this discussion, Guozhang. We might also want to discuss the API to expose the high watermark. Some discussion has been there in KAFKA-2076. Thanks, Jiangjie (Becket) Qin On 6/9/15, 1:12 PM, "Guozhang Wang" wrote: >This email is to kick-off some discussion around the cha

Re: Review Request 35201: Fix KAFKA-2253

> On June 11, 2015, 1:07 a.m., Jun Rao wrote: > > core/src/main/scala/kafka/server/DelayedOperation.scala, lines 264-266 > > > > > > Not sure if we need this check. Since all writes to watchersForKey are > > sync-ed,

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

Very useful KIP. I have no clear opinion over where to put the framework will be better yet. I agree with Gwen on the benefits we can get from have a separate project for Copycat. But still have a few questions: 1. As far as code is concerned, Copycat would be some datasource adapters + Kafka clie

Re: Review Request 33620: Patch for KAFKA-1690

operation. clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java (lines 127 - 139) <https://reviews.apache.org/r/33620/#comment141794> Not sure if it matters, but JSSE reference guide suggested to have a loop checking handshake state when shutdown. - Jiangj

Re: EOL JDK 1.6 for Kafka

+1 On 7/1/15, 1:00 PM, "Gwen Shapira" wrote: >Huge +1. > >I don't think there is any other project that still supports 1.6. > >On Wed, Jul 1, 2015 at 8:05 AM, Harsha wrote: >> Hi, >> During our SSL Patch KAFKA-1690. Some of the reviewers/users >> asked for support this config >>

Re: [ANNOUNCE] New Committer

Congrats, Gwen! On 7/6/15, 7:17 PM, "Jaikiran Pai" wrote: >Congratulations Gwen. > >Gwen has been very helpful in various places (blogs, user mailing lists) >which has helped me (and I'm sure many others) in using Kafka and even >contributing patches. Very well deserved promotion. > >-Jaikiran >

Re: Review Request 36345: Patch for KAFKA-2322

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36345/#review91321 --- Ship it! LGTM. - Jiangjie Qin On July 9, 2015, 9:05 a.m

Re: How "low-level" management of threads is justified in Mirror Maker?

Hi Kostya, I actually did not think about this too much when write the code. My gut feeling is that the benefit of using a thread pool is that you have jobs come and go and want to safe your own resource management overhead. In Mirror maker, there are only fixed number of mirror maker thread and a

Re: Inquiry regarding unreviewed patch (KAFKA-1614)

Hi Kim, I took a quick look at the patch. It seems all the information you need are available in Kafka now. There are several mBeans you can take a look. They are in kafka.log. Including log start/end offsets, size and number of log segments. Thanks, Jiangjie (Becket) Qin On 7/13/15, 5:01 PM,

Re: Review Request 36548: Patch for KAFKA-2336

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36548/#review92020 --- Ship it! Looks good to me. - Jiangjie Qin On July 16, 2015, 6

Re: Review Request 36548: Patch for KAFKA-2336

> On July 17, 2015, 4:26 a.m., Jiangjie Qin wrote: > > Looks good to me. Actually do we need to talk to Zookeeper every time? Can we read the data from topic metadata cache directly? - Jiangjie --- This is an automatically ge

Re: consumer memory footprint

I think the rough calculation of max memory footprint for each high level consumer would be: (Number Of Partitions For All Topics) * fetch.message.max.bytes * queued.max.message.chunks + (some decompression memory cost for a message) In your case, it would be 10 times above. Thanks, Jiangjie (B

Re: Review Request 36548: Patch for KAFKA-2336

> On July 17, 2015, 4:26 a.m., Jiangjie Qin wrote: > > Looks good to me. > > Jiangjie Qin wrote: > Actually do we need to talk to Zookeeper every time? Can we read the data > from topic metadata cache directly? > > Gwen Shapira wrote: > Good point, Jian

Re: Review Request 36548: Patch for KAFKA-2336

> On July 17, 2015, 4:26 a.m., Jiangjie Qin wrote: > > Looks good to me. > > Jiangjie Qin wrote: > Actually do we need to talk to Zookeeper every time? Can we read the data > from topic metadata cache directly? > > Gwen Shapira wrote: > Good point, Jian

Re: Review Request 36593: Patch for KAFKA-2347

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36593/#review92178 --- Ship it! LGTM. Thanks. - Jiangjie Qin On July 18, 2015, 5:02

Re: Review Request 36652: Patch for KAFKA-2351

Open source Kafka convention is to put the bracket on the same line. T - Jiangjie Qin On July 21, 2015, 8:11 p.m., Mayuresh Gharat wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://re

Re: Review Request 36652: Patch for KAFKA-2351

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36652/#review92488 --- Ship it! Latest patch looks good to me. - Jiangjie Qin On July

Review Request 36664: Patch for KAFKA-2353

/36664/diff/ Testing --- Thanks, Jiangjie Qin

Re: Review Request 36664: Patch for KAFKA-2353

s? - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36664/#review92496 --- On July 21, 2015, 11:03 p.m., Jiangjie Qin wrote

Re: Review Request 36664: Patch for KAFKA-2353

36664/diff/ Testing --- Thanks, Jiangjie Qin

Re: Review Request 36664: Patch for KAFKA-2353

eviews.apache.org/r/36664/#review92496 --- On July 22, 2015, 5:02 a.m., Jiangjie Qin wrote: > > --- > This is an automatically generated e-mail. To rep

Re: [DISCUSS] KIP-27 - Conditional Publish

I missed yesterday's KIP hangout. I'm currently working on another KIP for enriched metadata of messages. Guozhang has already created a wiki page before ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Enriched+Message+Metadata). We plan to fill the relative offset to the offset field in

Re: [VOTE] Switch to GitHub pull requests for new contributions

+1 (non binding) Can we have a wiki for procedure and let people verify the steps? After that we can update the Apache project page. On Tue, Jul 21, 2015 at 5:38 PM, Edward Ribeiro wrote: > +1 (non binding) > > On Tue, Jul 21, 2015 at 7:36 PM, Jay Kreps wrote: > > > +1 > > > > -Jay > > > > On

Re: Should 0.8.3 consumers correctly function with 0.8.2 brokers?

I don't think we have consumer coordinator in 0.8.2 brokers. So KafkaConsumer in 0.8.3 will only be able to subscribe to partitions explicitly. Subscribing to a topic won't work with 0.8.2 brokers. Jiangjie (Becket) Qin On Wed, Jul 22, 2015 at 4:26 AM, Stevo Slavić wrote: > I'm getting "Unknown

Re: Review Request 36664: Patch for KAFKA-2353

is a really bad idea: > > https://www.sumologic.com/2014/05/05/why-you-should-never-catch-throwable-in-scala/ > > Jiangjie Qin wrote: > Ah... Didn't know that before. I explicitly listed the exceptions. > > Guozhang Wang wrote: > Searching ": Throwable" gives me 180+

Re: Review Request 36664: Patch for KAFKA-2353

iff: https://reviews.apache.org/r/36664/diff/ Testing --- Thanks, Jiangjie Qin

Re: Review Request 36664: Patch for KAFKA-2353

--- On July 23, 2015, 12:51 a.m., Jiangjie Qin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/36664/ > --- > &

Re: Review Request 36664: Patch for KAFKA-2353

we log an error and keep running? > > > > Isn't it better to kill the processor, since we don't know what's the > > state of the system? If the acceptor keeps placing messages in the queue > > for a dead processor, isn't it a separate issue? > > Jiang

Re: Review Request 36664: Patch for KAFKA-2353

is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36664/#review92607 --- On July 23, 2015, 12:51 a.m., Jiangjie Qin wrote: > > --- > This is

Re: Round Robin Among Consumers

J A, It looks to me that in your case you actually want to scale the topic, right? Otherwise wouldn't a single consumer be enough? Jiangjie (Becket) Qin On Wed, Jul 22, 2015 at 7:39 PM, J A wrote: > Why have partition at all, if I don't need to scale topic. Coupling topic > scalability with co

Re: Review Request 36722: Patch for KAFKA-2332

x27;s hold back and wait for that to be checked in first and see if we need any change. - Jiangjie Qin On July 23, 2015, 7:12 a.m., Dong Lin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https:

Re: Review Request 36664: Patch for KAFKA-2353

> On July 22, 2015, 10:40 a.m., Ismael Juma wrote: > > core/src/main/scala/kafka/network/SocketServer.scala, line 401 > > <https://reviews.apache.org/r/36664/diff/2/?file=1018375#file1018375line401> > > > > Is it intentional to ignore `java.lang.Error` too?

Re: Round Robin Among Consumers

lve this problem. Not exactly a round-robin > but same idea. > > I think thats what J A was referring to in "decoupling consumers" - > different consumers have slightly different requirements. > > Gwen > > > > > > On Thu, Jul 23, 2015 at 9:44 AM, Jiangjie

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Hey Guozhang, I just took a quick look at the KIP, is it very similar to mirror maker with message handler? Thanks, Jiangjie (Becket) Qin On Thu, Jul 23, 2015 at 10:25 PM, Ewen Cheslack-Postava wrote: > Just some notes on the KIP doc itself: > > * It'd be useful to clarify at what point the p

Re: Review Request 36652: Patch for KAFKA-2351

> On July 24, 2015, 4:13 p.m., Jun Rao wrote: > > core/src/main/scala/kafka/network/SocketServer.scala, line 264 > > > > > > Not sure if it's better to keep the thread alive on any throwable. For > > unexpected exce

Re: error while high level consumer

This is due to the zookeeper path storing the previous owner info hasn't been deleted at the moment. If the rebalance completes after retry, it should be fine. Jiangjie (Becket) Qin On Fri, Jul 24, 2015 at 6:54 PM, Kris K wrote: > Hi, > > I started seeing these errors in the logs continuously w

Re: [DISCUSS] KIP-27 - Conditional Publish

tives (though some may not be mutually exclusive). Many people > don't > > have all the context from the past couple of years of the project. Are > > there any other relevant wikis or docs besides the following? > > > > https://cwiki.apache.org/confluence/display/KAFKA/I

Re: New Producer and "acks" configuration

I seems to me that performance wide, acks=1 and acks=0 will be pretty much the same if max.in.flight.request.per.connection is set to very high and does not cause a request sending to block. In new producer, if there are send failure from time to time, I would guess asks=1 would even have better p

Re: Kafka Consumer thoughts

I agree with Ewen that a single threaded model will be tricky to implement the same conventional semantic of async or Future. We just drafted the following wiki which explains our thoughts in LinkedIn on the new consumer API and threading model. https://cwiki.apache.org/confluence/display/KAFKA/Ne

Re: Kafka Consumer thoughts

the > wiki refers to, but none loaded for me. Just making sure if its just me or > can everyone not see the pictures? > > On Wed, Jul 29, 2015 at 12:00 PM, Jiangjie Qin wrote: > >> I agree with Ewen that a single threaded model will be tricky to >> implement the same con

Re: KAfka Mirror Maker

Mirror Maker does not have specific restrictions on cluster size. The error you saw was because consumer was not able to talk to the broker. Can you try to use kafka-console-consumer to consume some data from your source cluster and see if it works? It should be under KAFKA_HOME/bin/ Jiangjie (Be

Re: Kafka Consumer thoughts

uot; I would suggest that that is really the worst > outcome. These type of "modes" that are functionally totally different are > just awful from a documentation, testing, usability, etc pov. I would > strongly prefer we pick either of these, document it, and make it work well >

Re: Review Request 35421: Patch for KAFKA-2026

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35421/#review93543 --- Ship it! Ship It! - Jiangjie Qin On June 13, 2015, 10:07 a.m

Re: [DISCUSS] Partitioning in Kafka

Just my two cents. I think it might be OK to put this into Kafka if we agree that this might be a good use case for people who wants to use Kafka as temporary store for stream processing. At very least I don't see down side on this. Thanks, Jiangjie (Becket) Qin On Tue, Jul 28, 2015 at 3:41 AM,

Re: Gauging Interest in adding Encryption to Kafka

I think the goal here is to make the actual message stored on broker to be encrypted, because after we have SSL, the transmission would be encrypted. In general there might be tow approaches: 1. Broker do the encryption/decryption 2. Client do the encryption/decryption >From performance point of

Re: Kafka Consumer thoughts

is client isn't low-level >>> enough and we need to introduce another layer underneath. This would happen >>> either because we dictate too much to make it usable for various >>> applications, frameworks, or use cases. This is the concern with dictating >>> t

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

I think the abstraction of processor would be useful. It is not quite clear to me yet though which grid in the following API analysis chart this processor is trying to satisfy. https://cwiki.apache.org/confluence/display/KAFKA/New+consumer+API+change+proposal For example, in current proposal. It

Re: Gauging Interest in adding Encryption to Kafka

Jul 31, 2015 at 1:20 PM, eugene miretsky > > >> wrote: > > >> > I think that Hadoop and Cassandra do [1] (Transparent Encryption) > > >> > > > >> > We're doing [2] (on a side note, for [2] you still need > > authentication on > > >> &g

Re: [DISCUSS] KIP-27 - Conditional Publish

ill allow > Kafka > > to act as an event store in an Event Sourcing application. If the > event(s) > > are rejected by the store, the original command may need to be > re-validated > > against the new state. > > > > On Tue, Jul 28, 2015 at 1:27 AM Jiangjie Q

Re: Modified the contribution pages in the wiki

Thanks Gwen, it is much cleaner now :) Just wondering do we have any plan of deprecating the patch review tool? Jiangjie (Becket) Qin On Tue, Aug 4, 2015 at 2:14 PM, Gwen Shapira wrote: > Just FYI: > > I heavily refactored the Patch Review wiki page ( > https://cwiki.apache.org/confluence/displ

Re: Modified the contribution pages in the wiki

'll remove all trace of the old process from the wiki and > remove the tool from our trunk. > > Sounds good? > > > On Tue, Aug 4, 2015 at 3:39 PM, Jiangjie Qin > wrote: > > > Thanks Gwen, it is much cleaner now :) > > Just wondering do we have any plan of d

Re: JIRA KAFKA-1716

Looking at the stacktrace and I am wondering if it has been solved in KAFKA-2241. Thanks, Jiangjie (Becket) Qin On Wed, Aug 5, 2015 at 8:00 AM, Chris Barlock wrote: > While I don't have a patch to contribute today, per the "Nag us" note > here: > > http://kafka.apache.org/contributing.html > >

Re: Modified the contribution pages in the wiki

Hi Gwen, I just added one sentence saying we would prefer PR for new patches. Let me know if you think it is too verbose. Thanks, Jiangjie (Becket) Qin On Wed, Aug 5, 2015 at 8:57 AM, Gwen Shapira wrote: > Completely agree! Thank you for reviewing and improving. > > > > > > On Wed, Aug 5, 201

Re: Modified the contribution pages in the wiki

> Can you check if this page makes it clear that we take PRs and PRs only? > > Gwen > > On Wed, Aug 5, 2015 at 10:36 AM, Jiangjie Qin > wrote: > > > Hi Gwen, > > > > I just added one sentence saying we would prefer PR for new patches. Let > me >

[DISCUSSION] KIP-29 - Add an IsrPropagateIntervalMs config to KafkaConfig

Hi, I just created KIP-29 to add a new configuration of IsrPropagateIntervalMs to KafkaConfig. The motivation is to batch and throttle the ISR propagation in cases where large amount of ISR change occurs such as a broker gets bounced or failed. Otherwise it might overwhelm the controller. https:/

<    1   2   3   4   5   6   7   8   9   10   >