Re: [DISCUSS] KIP-388 Add observer interface to record request and response

bump. I think the proposed API (Observer) is useful for any sort of multi-tenant environment for chargeback and reporting purposes. if no one wants to comment, can we initiate a vote? On Mon, Nov 5, 2018 at 6:31 PM Lincong Li wrote: > > Hi everyone. Here >

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

ce area of clients, central observation of client activity is in > my opinion an essential feature. > > Peter > > On Thu, Nov 8, 2018 at 12:13 PM radai wrote: > > > bump. > > > > I think the proposed API (Observer) is useful for any sort of > > multi-tenant en

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

a few questions: 1. how do you handle possible duplications caused by the "special" producer timing-out/retrying? are you explicitely relying on the "exactly once" sequencing? 2. what about the combination of log compacted topics + replicator downtime? by the time the replicator comes back up ther

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

do you think ? > > > > > > > > > > > > Jason Gustafson wrote on 27/11/2018 00:09:56: > > > > > > > > > Another wrinkle to consider is KIP-320. If you are planning to > > replicate > > > > > __consumer_offsets direc

[DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

on to the current bound on the number of messages. This comes after several incidents at Linkedin where a sudden "spike" of large message batches caused an out of memory exception. Thank you, Radai

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

to that point, making immediate impact tiny (conf upgrade is trivial, behavior remains backwards compatible, and new feature is "opt-in"). I will extend the Unit tests (and fix the bugs) as part of the implementation. As for the performance implications: 1. my initial runs (https://github

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

, 2016 at 2:23 PM, Mayuresh Gharat wrote: > Nice write up Radai. > I think what Jun said is a valid concern. > If I am not wrong as per the proposal, we are depending on the entire > pipeline to flow smoothly from accepting requests to handling it, calling > KafkaApis and h

Re: why cant SslTransportLayer be muted before handshake completion?

the root issue is i may not have enough available memory to read from the socket. I'll do the simple thing and just avoid muting handshaking sockets, for now. On Wed, Nov 2, 2016 at 6:59 PM, Harsha Chintalapani wrote: > HI Radai, > One main reason is to keep the handsh

Re: [DISCUSS] KIP-72 Allow Sizing Incoming Request Queue in Bytes

here are my proposed changes: https://github.com/radai-rosenblatt/kafka/commit/8d7744ab8a6c660c4749b495b033b948a68efd3c at this point i've run this code on a test cluster under load that OOMs "vanilla" 0.10.1.0 and verified that my code deployed under the same condition remain

Re: [DISCUSS] KIP-82 - Add Record Headers

are friendly clients would have a standard header data model to > work with. > 3. KIP is required both b/c of broker changes and because of client API > changes. > > Cheers, > > Roger > > > On Wed, Nov 2, 2016 at 4:38 PM, radai wrote: > > > my biggest issue

[VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

he new proposed behavior, but as for implementation, i have the code up here: https://github.com/radai-rosenblatt/kafka/tree/broker-memory-pool-with-muting and I've stress-tested it to work properly (meaning it chugs along and throttles under loads that would DOS 10.0.1.0 code). I also believe tha

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

2 PM, Gwen Shapira wrote: > Hey Radai, > > Looking at the proposal, it looks like a major question is still > unresolved? > "This configuration parameter can either replace queued.max.requests > completely, or co-exist with it (by way of either-or or respecting > both bo

Re: [DISCUSS] KIP-82 - Add Record Headers

alizable, this > would > > make it fairly tricky (though it would not be impossible) to have made > work > > nicely. Having the value customizable we thought is a reasonable tradeoff > > here of flexibility over contract of interaction between different > parties.

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

I've updated the KIP page to specify the new config would co-exist with "queued.max.request" to minimize the impact on compatibility. On Tue, Nov 8, 2016 at 7:02 AM, radai wrote: > My personal opinion on this is that control of memory was always the > intent behind queue

Re: [DISCUSS] KIP-82 - Add Record Headers

econds which > > means > > > you only have another 0.75 microseconds to do everything else you want > to > > > do with a message (1M messages/s means 1 micro second per message). > With > > > string header keys there is still 0.5 micro seconds to process a &

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

congratulations! ) ( ) ( (^)(^)(^)(^) _i__i__i__i_ () ||>o<|###| () On Mon, Nov 7, 2016 at 6:03 AM, Michael Noll wrote: > Congratulations, Becket! > > Best wishes, > Michael > > On Thu, Nov 3, 2016 at 5:13 PM, Efe Gencer wrote: > > > Congr

Re: [DISCUSS] KIP-82 - Add Record Headers

gt; So io.confluent.schema-registry can be namespace 0x01 on my deployment > and 0x57 on yours, and the poor guys developing the app don't need to > worry about that. > > Gwen > > On Tue, Nov 8, 2016 at 4:23 PM, radai wrote: > > +1 for sean's document. it covers pre

Re: [DISCUSS] KIP-82 - Add Record Headers

code from stack overflow, and they only come to me when said copy-pasted code doesnt work :-) i would like to write a plugin that just drops messages without an audit header on the floor (think of this as the equivalent of a content-inspecting firewall). On Tue, Nov 8, 2016 at 6:41 PM, radai wr

Re: [DISCUSS] KIP-82 - Add Record Headers

ared (agreed upon) > understanding of what the namespaces mean. Which I think makes sense, > because the alternate (sharing *nothing* at all) would mean that there > would be no way to understand each other. > > -James > > > Gwen > > > > On Tue, Nov 8, 20

Re: [DISCUSS] KIP-82 - Add Record Headers

ll look normal. > > This means you only need to sync usage inside your own organization. > Still hard, but somewhat easier than syncing with the entire world. > > On Tue, Nov 8, 2016 at 10:07 PM, radai wrote: > > and we can start with {namespace, id} and no re-mapping support an

Re: [DISCUSS] KIP-82 - Add Record Headers

thinking about it some more, the best way to transmit the header remapping data to consumers would be to put it in the MD response payload, so maybe it should be discussed now. On Wed, Nov 9, 2016 at 12:09 AM, radai wrote: > im not opposed to the idea of namespace mapping. all im saying

Re: [DISCUSS] KIP-82 - Add Record Headers

n as it is being configured should be tolerable. > > > /Magnus > > 2016-11-09 13:06 GMT+01:00 Michael Pearce : > > > Just following/catching up on what seems to be an active night :) > > > > @Radai sorry if it may seem obvious but what does MD stand for? > > >

Re: Use Android App as a “Producing client” for Kafka?

also, for large-enough kafka clusters (#topics, #partitions) just downloading the metadata (required by the client to know where to publish stuff to) by be a big hit on your bandwidth consumption. I would go with something like a rest-proxy (there are a few open source ones) or roll your own server

Re: [DISCUSS] KIP-82 - Add Record Headers

com/protocol-buffers/docs/encoding#varints) for > the header length field (int32 in the proposal) and for header ids? If you > don't use headers, the overhead would be a single byte and for each header > id < 128 would also need only a single byte? > > > > On Wed, Nov 9, 20

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

cause of the log msg printed by the pool. 1.2 I have a benchmark suite showing the performance cost of the gc pool is absolutely negligible - https://github.com/radai-rosenblatt/kafka-benchmarks/tree/master/memorypool-benchmarks 1.3 as for the complexity of the impl - its just ~150 lines and

Re: [DISCUSS] KIP-81: Max in-flight fetches

selectively reading from sockets achieves memory control (up to and not including talk of (de)compression) this is exactly what i (also, even mostly) did for kip-72 - which i hope in itself should be a reason to think about both KIPs at the same time because the changes will be similar (at least i

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

my personal opinion - a log compacted topic is basically a kv-store, so a map API. map.put(key, null) is not the same as map.remove(key), which to me means a null value should not represent a delete. a delete should be explicit (meaning flag). On Wed, Nov 9, 2016 at 11:01 AM, Mayuresh Gharat wro

any plans to switch to java 8?

with java 7 being EOL'ed for more than a year and a half now (apr 2015, see http://www.oracle.com/technetwork/java/eol-135779.html) i was wondering if there's an official plan/timetable for transitioning the kafka codebase over to java 8?

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

On Fri, Nov 11, 2016 at 5:02 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > Radai, > > 11. The KIP talks about a new server configuration parameter > *memory.pool.class.name > <http://memory.pool.class.name> *which is not in the implementation. Is it > s

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

memory - for each channel, at most one failed read > attempt would be made while the pool is out of memory. I think this would > also delay muting of SSL channels since they can continue to read into > their (already allocated) network buffers and unwrap the data and block > only whe

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

o read). > But since this and open point 2 (optimization) are implementation details, > they can be looked at during PR review. > > It will be good to add SSL testing to the test plan as well, since there is > additional code to test for SSL. > > > On Fri, Nov 11, 2016 at

Re: [DISCUSS] KIP-81: Max in-flight fetches

is to selectively read from sockets instead of > throttling FetchRequests sends. I also mentioned it will be reusing > the MemoryPool implementation created in KIP-72 instead of adding > another memory tracking method. > > Please have another look. As always, comments are welcome ! > &

Re: [VOTE] KIP-84: Support SASL SCRAM mechanisms

small nitpick - given that s,t,k and i are used as part of a rather large CSV format, what is the gain in having them be single letter aliases? in other words - why not salt=... , serverKey=... , storedKey=... , iterations=... ? On Tue, Nov 15, 2016 at 7:26 AM, Mickael Maison wrote: > +1 > > On

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

termark(timeout) or something, and make Selector.poll() wait for low watermark at the end of poll() if no work has been done (so as not to wait on memory needlessly for requests that done require it, as rajini suggested earlier) On Wed, Nov 16, 2016 at 9:04 AM, Jun Rao wrote: > Hi, Radai,

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

;BuffersOutstanding" ? On Thu, Nov 17, 2016 at 7:01 PM, Jun Rao wrote: > Hi, Radai, > > 2. Yes, on the server side, the timeout is hardcoded at 300ms. That's not > too bad. We can just leave it as it is. > > 3. Another thing. Do we plan to expose some JMX metrics so th

Re: should HeartbeatThread be a daemon thread?

a similar issue exists with the old client stack as well - https://github.com/apache/kafka/pull/1930 On Mon, Nov 21, 2016 at 9:10 PM, Jason Gustafson wrote: > Hey David, > > It probably should be a daemon thread. Perhaps open a JIRA? > > Thanks, > Jason > > On Mon, Nov 21, 2016 at 2:03 PM, David

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

Hi Jun, I've added the sensor you requested (or at least I think I did ) On Fri, Nov 18, 2016 at 12:37 PM, Jun Rao wrote: > KafkaRequestHandlerPool

Re: [DISCUSS] 0.10.1.1 Plan

"compatibility guarantees that are expected by people who subclass these classes" sorry if this is not the best thread for this discussion, but I just wanted to pop in and say that since any subclassing of these will obviously not be done within the kafka codebase - what guarantees are needed? On

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

will do (only added a single one so far, the rest TBD) On Mon, Nov 28, 2016 at 10:04 AM, Jun Rao wrote: > Hi, Radai, > > Could you add a high level description of the newly added metrics to the > KIP wiki? > > Thanks, > > Jun > > On Wed, Nov 23, 2016 at 3:45

Re: any plans to switch to java 8?

7 AM, Sean McCauliff < > sean.mccaul...@gmail.com > > > > > wrote: > > > > > Wait for JDK 9 which is supposed to be 4-5 months from now? > > > > > > Sean > > > > > > On Thu, Nov 10, 2016 at 10:23 AM, radai > > > wrote: &

Re: [DISCUSS] 0.10.1.1 Plan

n Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma wrote: > On Sat, Nov 26, 2016 at 11:08 PM, radai > wrote: > > > "compatibility guarantees that are expected by people who subclass these > > classes" > > > > sorry if this is not the best thread for this discu

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

+1 (non-binding) On Tue, Nov 29, 2016 at 8:08 AM, wrote: > +1 (non-binding) > > Thanks, > > Mayuresh > > > > On Nov 29, 2016, at 3:18 AM, Michael Pearce > wrote: > > > > Hi All, > > > > We have been discussing in the below thread and final changes have been > made to the KIP wiki based on these

Re: [DISCUSS] KIP-82 - Add Record Headers

14.11.2016, 21:15, "Ignacio Solis" : > > > > 1) Yes - Headers are worthwhile > > > > 2) Yes - Headers should be a top level option > > > > > > > > On Mon, Nov 14, 2016 at 9:16 AM, Michael Pearce < > > michael.pea.

Re: [DISCUSS] KIP-82 - Add Record Headers

n Thu, Dec 1, 2016 at 11:20 AM, Gwen Shapira wrote: > On Thu, Dec 1, 2016 at 10:24 AM, radai wrote: > > "For use cases within an organization, one could always use other > > approaches such as company-wise containers" > > this is what linkedin has traditionally don

Re: [DISCUSS] KIP-82 - Add Record Headers

:) > > >> > > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd Palino > wrote: > > >> > Well I guess my question for you, then, is what is holding you back > > from > > >> > full support for headers? What’s the bit that you’re missing that > ha

Re: [DISCUSS] KIP-82 - Add Record Headers

" requests that ask for more than 40 partitions, or > require exactly 3 replicas, or no more than 50GB partition size, etc. > > ACLs were added a bit ad-hoc, if we are planning to apply more rules > to requests (and I think we should), we may want a bit more generic > design ar

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

+1 (non-binding). small nit pick - just because you returned a response to user doesnt mean the memory id no longer used. for some cases the actual "point of termination" may be the deserializer (really impl-dependant), but generally, wouldnt it be "nice" to have an explicit dispose() call on resp

Re: [DISCUSS] KIP-82 - Add Record Headers

ded features) so we know what version of the wrapper > is used > > > Producer IP address (in case of clients being in our vm/open stack > infra > > > where they can move around, producer id will stay the same but > this would > > > change) > > > > &g

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

I can see several issues with the current proposal. messages, even if sent under a TX, are delivered directly to their destination partitions, downstream consumers need to be TX-aware. they can either: 1. be oblivious to TXs. that means they will deliver "garbage" - msgs sent during eventually-

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

kiness around how not to expose these offsets to any consumers until everything's been replicated to followers, but we believe its possible. On Thu, Dec 15, 2016 at 2:31 PM, radai wrote: > I can see several issues with the current proposal. > > messages, even if sent under a T

Re: [DISCUSS] KIP-82 - Add Record Headers

we know what version of the wrapper > is > > used > > > Producer IP address (in case of clients being in our vm/open stack > > infra > > > where they can move around, producer id will stay the same but this > > would > > > change

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

cross the whole datacenter, not just the broker) and in terms of eco system adoption (less burden on client writers, as we want to make it easier to have lots of client impls). On Thu, Dec 15, 2016 at 11:59 PM, Apurva Mehta wrote: > Hi Radai, > > Thanks for your email. You raise s

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

already suffers it. On Fri, Dec 16, 2016 at 9:24 AM, radai wrote: > Hi Apurva, > > here's an outline of what I had in mind: > > 1. TXs in progress are written "sideways" on the partition leader. this > can be (in order of increasing reliability and decreasing perfor

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

I've added the 3 new metrics/sensors i've implemented to the KIP. at this point I would need to re-validate the functionality (which i expect to do early january). code reviews welcome ;-) On Mon, Nov 28, 2016 at 10:37 AM, radai wrote: > will do (only added a single one so far,

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

regarding efficiency: I'd like to distinguish between server efficiency (resource utilization of the broker machine alone) and overall network efficiency (resource utilization on brokers, producers and consumers, including network traffic). my proposal is not as resource-efficient on the broker (a

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

tion was committed. For some scenarios, we > believe that offset ordering would still be preferred than transaction > ordering and that is why in KIP-98 proposal we default to the former while > leave the door open if users want to switch to the latter case. > > > Guozhang > >

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

this kip fixes a "bug" (quirk?) that arises when people implement headers "in V" (in the payload part of a message). if you have proper headers you obviously dont need to to stick them in V and so wont run into this, but its still a valid issue. On Mon, Dec 19, 2016 at 3:06 PM, Jay Kreps wrote:

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

i didnt mean to sound as insisting. what i actually mean is it would still be a valid issue but of much less concern. On Mon, Dec 19, 2016 at 7:50 PM, radai wrote: > this kip fixes a "bug" (quirk?) that arises when people implement headers > "in V" (in the payload pa

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

of that is > obvious from the proposal. > > -Jay > > > > > > On Tue, Dec 20, 2016 at 9:11 AM, Joel Koshy wrote: > > > Just got some time to go through most of this thread and KIP - great to > see > > this materialize and discussed!! > > I will

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

when the leader decides to commit a TX (of X msgs, known at this point), it writes an "intent to append X msgs" msg (control?) followed by the X msgs (at this point it is the leader and therefor point of sync, so this can be done with no "foreign" msgs in between). if there's a crash/change of lead

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

+1 On Wed, Dec 21, 2016 at 9:51 AM, Dong Lin wrote: > +1 (non-binding) > > On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin wrote: > > > Hi, > > > > I want to start a voting thread on KIP-92 which proposes to add per > > partition lag metrics to KafkaConsumer. The KIP wiki page is below: > > > > htt

Re: Different key with the same digest in log compaction

i think the code assumes that with a "good enough" hash function (and maybe few enough keys) the chance of such a collision is acceptably small to justify the savings of not keeping the keys in memory. On Wed, Dec 21, 2016 at 11:50 PM, Renkai wrote: > Hi, all: > > I am just learning the Kafka

Re: custom offsets in ProduceRequest

IIUC if you replicate from a single source cluster to a single target cluster, the topic has the same number of partitions on both, and no one writes directly to the target cluster (so master --> slave) the offsets would be preserved. but in the general case - how would you handle the case where m

Re: custom offsets in ProduceRequest

e of these options look good to me. On Thu, Dec 29, 2016 at 3:22 AM, Andrey L. Neporada < anepor...@yandex-team.ru> wrote: > Hi! > > > On 27 Dec 2016, at 19:35, radai wrote: > > > > IIUC if you replicate from a single source cluster to a single target >

Re: custom offsets in ProduceRequest

uld be made part of the replicator component (mirror maker, or whatever else you want to use) - if topic X does not exist in destination, create it, reset initial offsets to match source, start replication On Thu, Dec 29, 2016 at 12:41 PM, Andrey L. Neporada < anepor...@yandex-team.ru> wrote

Re: custom offsets in ProduceRequest

or even better - if topic creation is done dynamically by the replicator, setting the initial offsets for partitions could be made part of topic creation ? even less API changes this way On Thu, Dec 29, 2016 at 10:49 PM, radai wrote: > ah, I didnt realize we are limiting the discussion

Re: [DISCUSS] KIP-105: Addition of Record Level for Sensors

link leads to 104. i think this is the correct one - https://cwiki.apache.org/confluence/display/KAFKA/KIP-105%3A+Addition+of+Record+Level+for+Sensors ? On Fri, Dec 30, 2016 at 8:31 PM, Aarti Gupta wrote: > Hi all, > > I would like to start the discussion on KIP-105: Addition of Record Level > f

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

I've just re-validated the functionality works - broker throttles under stress instead of OOMs. at this point my branch ( https://github.com/radai-rosenblatt/kafka/tree/broker-memory-pool-with-muting) is "code complete" and somewhat tested and im waiting on the voting proce

Re: Different key with the same digest in log compaction

looking at the code (SkimpyOffsetMap.get/put) they both start with hashInto(key, hash1) and then ignore key from that point on - so we're not using the key unless im missing something? as for the probability of collision - it depends on the hash algo and the number of keys. if you configure it to

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

the issue with tracking committed offsets is whos offsets do you track? 1. some topics have multiple groups 2. some "groups" are really one-offs like developers spinning up console consumer "just to see if there's data" 3. there are use cases where you want to deliberately "wipe" data EVEN IF its

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

also 4. some apps may do their own offset bookkeeping On Tue, Jan 3, 2017 at 5:29 PM, radai wrote: > the issue with tracking committed offsets is whos offsets do you track? > > 1. some topics have multiple groups > 2. some "groups" are really one-offs like develope

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

@jun - good proposal. i was willing to concede that read-uncommitted was impossible under my proposal but if LSO/NSO is introduced is becomes possible. On Tue, Jan 3, 2017 at 7:50 AM, Jun Rao wrote: > Just to follow up on Radai's idea of pushing the buffering logic to the > broker. It may be po

Re: [DISCUSS] KIP-124: Request rate quotas

i dont think time/cpu% are easy to reason about. most user-facing quota systems i know (especially the commercial ones) focus on things users understand better - iops and bytes. as for quotas and "overhead" requests like heartbeats - on the one hand subjecting them to the quota may cause clients t

Re: [DISCUSS] KIP-124: Request rate quotas

@jun: i wasnt concerned about tying up a request processing thread, but IIUC the code does still read the entire request out, which might add-up to a non-negligible amount of memory. On Thu, Feb 23, 2017 at 11:55 AM, Dong Lin wrote: > Hey Rajini, > > The current KIP says that the maximum delay w

Re: [DISCUSS] KIP-82 - Add Record Headers

gt; > headers, I think we want to ensure some least common features are > supported > > and not limited. > > > > > > Some examples we have already. > > > > On consume interceptors a security interceptor may need to take the > > current

Re: [DISCUSS] KIP-82 - Add Record Headers

I will settle for any API really, but just wanted to point out that as it stands right now the API targets the most "advanced" (hence obscure and rare) use cases, at the expense of the simple and common ones. i'd suggest (as the minimal set): Header header(String key) - returns JUST ONE (the very

Re: [DISCUSS] KIP-82 - Add Record Headers

@michael: i used void because im used to java beans. thinking about it, i dont see much use for returning false from adding a header: if the headers are in read-only you should probably thrown an IllegalStateException because lets face it, 99% of users dont check return values. returning "this" is

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

quick comment on the request objects: i see "abstract class NewTopic" and "class NewTopicWithReplication" and " NewTopicWithReplicaAssignments" 1. since the result object is called CreateTopicResults should these be called *Request? 2. this seems like a suboptimal approach to me. imagine we add a

Re: [DISCUSS] KIP-82 - Add Record Headers

where do you see insert-in-the-middle/replace being commonly used? lineage tracing, as you call it, would probably be implemented by way of: 1. every "stop" along the way appending itself (at the end) 2. some replication technologies, instead of just doing #1, may clear out everything when they re

Re: [DISCUSS] KIP-82 - Add Record Headers

what i think is the most common case - a for each loop: for (Header stop : headers("lineage")) { //examine stop } On Tue, Mar 7, 2017 at 12:31 PM, radai wrote: > ing, as you call it, would probably be implemente

Re: [DISCUSS] KIP-82 - Add Record Headers

the common "stack" we envision at linkedin would consist of (at least) the following components that add headers to every outgoing request: 1. auditing/"lineage" - appends a header containing "node" (hostname etc), time (UTC time) and destination (cluster/topic). these accumulate as requests get m

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

> > the scope is already big enough here. Also, in practice, users have > > > > workarounds for cases where there are timeouts or failures to > > > > communicate. > > > > > > > > best, > > > > Colin > > > > > > >

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

> > > > >> > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > Mayuresh > > > >> > > > >

Re: KafkaProducer may send duplicated message sometimes

kafka (at least out of the box as it is now) is not an exactly-once system. its an at-least-once system, meaning the scenario you described (and similar ones involving socket disconnections, for example) exist by design. there is a KIP for adding exactly once guarantees (among other things) that y

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

gt; >> >> >> >> >> To alleviate this issue, only messages larger than 1Kb > > will > > > > be > > > >> >> >> >> allocated in > > > >> >> >> >> >> the MemoryPool. Smaller messages will be allocate

Re: KafkaProducer may send duplicated message sometimes

e needs to be written to account for this - either "dedup" things you read or make sure read side-effects are idempotent in some other way. longer-term when KIP-98 is implemented you could consider switching to it On Fri, Mar 31, 2017 at 9:28 AM, Yang Cui wrote: > Hi Radai, >

Re: [VOTE] KIP-112 - Handle disk failure for JBOD

+1, LGTM On Mon, Apr 3, 2017 at 9:49 AM, Dong Lin wrote: > Hi all, > > It seems that there is no further concern with the KIP-112. We would like > to start the voting process. The KIP can be found at > *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 112%3A+Handle+disk+failure+for+JBOD

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

its better to expose a raw, simple API and let such applications manage their own clean-up logic (this again ties into 1.4 and no "one size fits all" solution) On Tue, Jan 3, 2017 at 11:49 PM, Dong Lin wrote: > On Tue, Jan 3, 2017 at 11:01 PM, Ewen Cheslack-Postava > wrote: >

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

n Wed, Jan 4, 2017 at 7:54 AM, radai wrote: > a major motivation for this KIP is cost savings. > > lots of internal systems at LI use kafka as an intermediate pipe, and set > the topic retention period to a "safe enough" amount of time to be able to > recover from cr

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

-up for all eternity, unless its cleaned-up itself. ... complexity :-) On Wed, Jan 4, 2017 at 8:04 AM, radai wrote: > in summary - i'm not opposed to the idea of a per-topic clean up config > that tracks some set of consumer groups' offsets (which would probably work > for

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

im all for (working towards) getting rid of old code, but there's still no solid migration path - you'll be "stranding" users on deprecated, no longer maintained code with no "safe" way out that does not involve downtime (specifically old and new consumers cannot correctly divide up partitions betw

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

I cant speak for anyone else, but a rolling upgrade is definitely how we (LinkedIn) will do the migration. On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira wrote: > it sounds good to have > it, but that's probably not how people will end up migrati >

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

Will do (sorry for the delay). and thank you. On Fri, Jan 6, 2017 at 7:56 AM, Ismael Juma wrote: > Radai, you have more than enough votes to declare the vote successful. > Maybe it's time to do so. :) Also, once you have done that, it would be > good to move this KIP to the adopte

Re: Different key with the same digest in log compaction

Colin McCabe wrote: > That's a fair point. The calls to Arrays.equals are comparing just the > hashes, not the keys. > > Colin > > On Tue, Jan 3, 2017, at 17:15, radai wrote: > > looking at the code (SkimpyOffsetMap.get/put) they both start with > > hashInto(ke

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

JIRA up - https://issues.apache.org/jira/browse/KAFKA-4602 PR up - https://github.com/apache/kafka/pull/2330 KIP wiki has been updated. On Fri, Jan 6, 2017 at 8:16 AM, radai wrote: > Will do (sorry for the delay). > and thank you. > > On Fri, Jan 6, 2017 at 7:56 AM, Ismael

out of the box security?

in light of things like this - https://www.bleepingcomputer.com/news/security/mongodb-apocalypse-is-here-as-ransom-attacks-hit-10-000-servers/ and given that plenty of people/orgs have public facing kafka installations that are wide open - https://www.shodan.io/search?query=kafka (yes, i realize t

Re: [DISCUSS] KIP-82 - Add Record Headers

while HTTP-style (string, string) are the most common and most familiar, there is a very significant impact on msg size, especially given that some payloads are literally a few integers (think stock quotes) and would be dwarfed by an http-like header segment. I think we're ok with not allowing for

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

LGTM, +1 On Wed, Jan 11, 2017 at 1:01 PM, Dong Lin wrote: > Hi all, > > It seems that there is no further concern with the KIP-107. At this point > we would like to start the voting process. The KIP can be found at > https://cwiki.apache.org/confluence/display/KAFKA/KIP-107 > %3A+Add+purgeDataBe

Re: [DISCUSS] KIP-81: Max in-flight fetches

t; > >> On Wed, Dec 14, 2016 at 12:29 PM, Rajini Sivaram > >> wrote: > >> > Edo, > >> > > >> > I wouldn't introduce a new config entry, especially since you don't > need > >> it > >> > after KAFKA-4137. As a tempor

Re: [VOTE] KIP-74: Add FetchResponse size limit in bytes

+1 On Fri, Jan 20, 2017 at 9:51 PM, Apurva Mehta wrote: > +1 > > On Fri, Jan 20, 2017 at 5:19 PM, Jason Gustafson > wrote: > > > +1 > > > > On Fri, Jan 20, 2017 at 4:51 PM, Ismael Juma wrote: > > > > > Good catch, Colin. +1 to editing the wiki to match the desired > behaviour > > > and what wa

  1   2   3   >