Re: Producer request latency increase after client 0.10 upgrade

Hi Yifan, Just to make sure I understand, the comparison is between 0.8.2.1 and 0.10.0.0 clients (both) connecting to a 0.10.0.0 cluster? We haven't received reports of increased average producer latency and I'm not aware of a change that could cause such a dramatic difference. Is compression bein

Re: Using kafka as a "message bus" for an event store

inline On Mon, Sep 5, 2016 at 12:00 AM, F21 wrote: > Hi all, > > I am currently looking at using Kafka as a "message bus" for an event > store. I plan to have all my events written into HBase for permanent > storage and then have a reader/writer that reads from HBase to push them > into kafka.

Re: Producer request latency increase after client 0.10 upgrade

One thing that changed in 0.9.0.0 is the introduction of `max.block.ms`, which is 60 seconds by default. Since you have "block.on.buffer.full=false", you may want to tweak `max.block.ms`. Ismael On Mon, Sep 5, 2016 at 1:54 PM, Ismael Juma wrote: > Hi Yifan, > > Just to make sure I understand, t

Re: Issue adding server (0.10.0.0)

Hi all We keep having this issue. After increasing the fetch threads, we cleared the entire cluster, upgraded to 0.10.0.1, started all nodes, and all was well. We cannot reduce the fetch size, as it is equal to our max.message.size. Increasing the number of replica threads to a higher count increa

Kafka related queries

Hi All, I am very new to Kafka. I am still on the verge of deciding the platform to choose between Kafka and Kinesis for our requirement. Also it may take me more time to come up with a full proof answer, hence thought of parallelizing my search. Following are a few queries around Kafka :

After restart Kafka does not see Topics

Hello, I am running kafka 0.9.0.1, there I have 1 broker witth around 5 topics each has 1 partition, I manage kafka via KafkaManager webui tool, periodically something happens with Kafka and it stops seeing topics, despite that process runs (withou errors) and topics available on the disk. In order

How to set the offset of a topic:partition for a specific consumer group to repay / reconsume messages ?

Hi, This seems like a basic question but I can't find the answer : I'm trying to find the right tool (in kafka/bin ) to set an offset value of a topic:partition for a specific consumer-group in order to replay consumed messages this link tells how to get the offset of the topic:partition of a con

Kafka Streams: joins without windowing (KStream) and without being KTables

Hi, I've been thinking how to solve with Kafka Streams one of my business process without success for the moment. Hope someone can help me. I am reading from two topics events like that (I'll simplify the problem at this point): ObjectX Key: String Value: String ObjectY Key: String Value: Strin

Re: Kafka related queries

Hi Gauri, I'll take a stab at your questions (others will undoubtedly correct me if I'm wrong). Kafka does work over TCP (what else :) No format expectations (binary). End to end latency depends on various parameters, like network, memory, but it is (if you consume straight away; i.e. no consumer

Re: Kafka related queries

BTW regarding latency: https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines Marko Bonaći Monitoring | Alerting | Anomaly Detection | Centralized Log Management Solr & Elasticsearch Support Sematext | Contact

Re: Kafka Streams: joins without windowing (KStream) and without being KTables

Hey, are you sure, you want to join everything? This will result in a huge memory footprint of your application. You are right, that you cannot use KTable, however, windowed KStream joins would work -- you only need to specify a huge window (ie, use Long.MAX_VALUE; this will effectively be "infini

Authorization with Topic Wildcards

Hi all, Although the documentation mentions that one can use wildcards with topic ACLs, I couldn't get that to work. Essentially, I want to set an Allow Read/Write ACL on topics com.domain.xyz.* to a certain user. This would give this user Read/Write access to topics com.domain.xyz.abc and com.dom

Re: Authorization with Topic Wildcards

if you're running that at a bash or similar shell, you need to quote the "*" so that bash doesn't expand it as a glob: ./kafka-acls.sh --authorizer-properties zookeeper.connect= --add --allow-principal User:"user01" --topic 'com.domain.xyz.*' --group group01 --operation read It may be instructi

micro-batching in kafka streams

Hi, What’s the best way to do micro-batching in Kafka Streams? Any plans for a built-in mechanism? Perhaps StateStore could act as the buffer? What exactly are ProcessorContext.schedule()/punctuate() for? They don’t seem to be used anywhere? http://hortonworks.com/blog/apache-storm-design-patt

Re: Authorization with Topic Wildcards

Yes, I am running it from the command line. Zookeeper has *com.domain.xyz.** under /kafka-acl node. So it looks like it's being added correctly. I actually allowed some time for ACL propagation to the Kafka brokers. On Mon, Sep 5, 2016 at 11:42 AM, Tom Crayford wrote: > if you're running that

Re: Authorization with Topic Wildcards

Hi Derar, The support for wildcards is limited to `*` at this point. Sorry for the confusion. If you're interested to submit a PR to clarify the documentation, that would be great. :) Ismael On Mon, Sep 5, 2016 at 7:38 PM, Derar Alassi wrote: > Hi all, > > Although the documentation mentions t

Re: Kafka Streams: joins without windowing (KStream) and without being KTables

Hi Matthias, Good question... the main problem is related with the kind of my data. The first source of data is time based and the second one not but both have a field with the same value (I don't know how to use it in the join without being key. It can't, let me explain why): ObjectX (sameValue,

Re: Using kafka as a "message bus" for an event store

Hi Tom, Thank you so much for your response. I had a feeling that approach would run into scalability problems, so thank you for confirming that. Another approach would be to have each service request a subscription from the event store. The event store then creates a unique kafka topic for

Re: Question: Data Loss and Data Duplication in Kafka

Thanks Radha Krishna! So from what I understand, data loss can happen at producer due to BufferExhaustedException, failure to close/terminate producer and due to communication errors (first figure below). And at the broker during unclean leader election (i.e. electing a leader that was not in IS

Re: Monitoring Kafka client

Hello Otis, Thank you for your reply. Sorry for not being very explicit. For this particular case, the failed application was on the consumer side, however, monitoring the producer in the same way would be desired as well. I had a look into SMP. I looks good however I'm up to finding a way to chec

Producer/Consumer config for length Kafka broker maintenance

Hi all, there are so many timeouts to tweak mentioned in the documentation that I wonder what the correct configuration for producer and consumer is to survive a, say, 1 hour, broker shutdown. With "survive" I mean that the processes are idle or blocked and keep trying to send their data, an

Re: Kafka bootup exception while recovering log file

I'm not from the Kafka dev team so I won't be able to comment whether this is an expected way to fail or if this needs to be handled in a more cleaner/robust manner (at least very least probably a better exception message). Since you have put in efforts to write a test case and narrow it down t

Re: Producer request latency increase after client 0.10 upgrade

Hi Ismael, Thanks for replying. Yes. It's the comparison between 0.8.2.1 and 0.10.0.0 clients connecting to 0.10.0.0 brokers(log.message.format.version=0.8.2.1). No compression is used and that's all the producer config. >From the Kafka code, I think the request-latency-avg metric seem not inclu

Re: kafka broker is dropping the messages after acknowledging librdkafka

Hi Jun / Kafka Team, Do we have any solution for this issue ? "During zookeeper re-init kafka broker truncates messages and ends up in loosing records" I'm ok with with duplicate messages being stored instead of dropping. Is there any configuration in kafka where follower broker replicates thes

Re: KIP-33 Opt out from Time Based indexing

Hi Jun, sorry for the late reply. Regarding B, my main concern was just complexity of understanding what's going on. As you can see it took me probably some 2 days or so, to fully grab all the details in the implementation and what the impacts are. Usually I prefer to turn things I don't use of