Re: Single Key Aggregation

Hi Matthias, I am working on 10.2.1. and I do see same outputs on same keys albeit not always and a bit random. I shall try to code a simpler example free from domain level code and share it with you in a while. I am willing to debug this further as well, if you could tell me the classes that I

Re: ticketing system Design

need some more input on this. Kafka is a queue it doesn't take any action. sender(producer) sends data to kafka and consumer pulls data from kafka queue. so there is no assignment of data to any consumer. if a process/human cann't take any action then kafka cann't help in this case. hope it

Re: Kafka Streams: Problems implementing rate limiting processing step

Andre, Back to your original question I think KIP-138 could help, as it allows you to punctuate based on processing time as well. https://cwiki.apache.org/confluence/display/KAFKA/KIP-138%3A+Change+punctuate+semantics Guozhang On Mon, Jun 19, 2017 at 2:02 PM, Andre Eriksson wrote: > So I'm tr

Re: facing serialization issues with Kafka Streams application ..

Damian, This type of questions seem to be common, could you add an entry into FAQ? https://cwiki.apache.org/confluence/display/KAFKA/FAQ Guozhang On Tue, Jun 20, 2017 at 8:23 AM, Debasish Ghosh wrote: > On Tue, Jun 20, 2017 at 7:03 PM, Damian Guy wrote: > > > m.groupBy(mapper, Serdes.String(

Re: Kafka Streams vs Spark Streaming : reduce by window

Although we have seen this request commonly being asked in the community, to me people are actually requesting slightly different things when they mention "one final output", or "trigger", or "window closure" terms, etc. So I'd like to try summarizing them here based on my own understanding before

Re: KStream Usage spikes memory consumption and breaks Kafka

Could you provide some configuration information and more context? What application are you running, when is it running out of memory? Otherwise it's hard to tell. Eno > On 20 Jun 2017, at 22:15, IT Consultant <0binarybudd...@gmail.com> wrote: > > Hi All , > > Kafka instance is breaking down w

KStream Usage spikes memory consumption and breaks Kafka

Hi All , Kafka instance is breaking down when used Kstream . It runs out of memory frequently resulting into service unavailabilty , Is it a good practice to use Kstream ? What other option must be tried to avoid such breakage ? If it's best pratice , how do we fine tune kafka to withhold load ?

Kafka MirrorMaker - errors/warning

Hi All - i've setup Kafka MirrorMaker using link - Kafka Mirror Maker best Practices , and am getting the following warnings -> [2017-06-20 12:29:31,381] WARN The configuration queue.time = 100 was suppl

Re: [VOTE] 0.11.0.0 RC1

Hi Ismael, Thanks for running the release. Running tests ('gradlew.bat test') on my Windows 64-bit VM results in these checkstyle errors: :clients:checkstyleMain [ant:checkstyle] [ERROR] C:\Users\User\Downloads\kafka-0.11.0.0-src\clients\src\main\java\org\apache\kafka\common\protocol\Errors.ja

Re: Kafka Streams: Problems implementing rate limiting processing step

Two comments: (1) This approach is basically "within" the processor, as it does use ProcessorContext -- "outside" would mean to use IQ API. (2) As reported, this does not work and fails with an exception and this is by design and thus we won't change it. -Matthias On 6/20/17 10:10 AM, Michal

JMX security options resulting in log4j issue

Hi, I want to enable security for JMX but I'm hitting a broker startup issue which I can't quite work out and wonder if anyone has any ideas. $ export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \ -Dcom.sun.management.jmxremote.port=12346 \ -Dcom.sun.management.jmxremote.rmi.port=12346 \ -Dcom

Re: Kafka Streams: Problems implementing rate limiting processing step

Thanks, maybe I miss-read it: I then tried implementing my own scheduling that periodically sends/clears out messages using the ProcessorContext provided to the aforementioned transform ste I understood it to say it inspects a state store, sends the messages that should be sent and removes the

Re: Kafka Streams: Problems implementing rate limiting processing step

>> I didn't know you could write to state stores from outside a >> processor/transformer. You can't. And as far as I understand this thread, nobody said you can. Did I miss something? -Matthias On 6/20/17 1:02 AM, Michal Borowiecki wrote: > I didn't know you could write to state stores from ou

kafka version 0.10.2.1 consumer can not get the message

Hello experts I write the kafka demo with java . The prouct can send the message but the consumer can not get the message My kafka configuration is ok ./kafka-console-producer.sh --broker-list localhost:9080 --topic testkun ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic testkun --

Re: facing serialization issues with Kafka Streams application ..

On Tue, Jun 20, 2017 at 7:03 PM, Damian Guy wrote: > m.groupBy(mapper, Serdes.String(), Serdes.String()).count(" > HostAggregateCount") > Thanks! It works .. -- Debasish Ghosh http://manning.com/ghosh2 http://manning.com/ghosh Twttr: @debasishg Blog: http://debasishg.blogspot.com Code: http:

Re: Kafka Monitoring

Similar to other approaches, our service uses JMX via Jolokia and then we save the time-series data in Redis. Then we expose this in a number of ways including our dashboard, etc. We have found Redis to be quite good for a time-series backend for this purpose. This all gets setup automatically a

Re: Kafka Monitoring

Not for monitoring Kafka. We pull the JMX metrics two ways - one is a container that wraps around the Kafka application and annotates the beans to be emitted to Kafka as metrics, which gets pulled into our autometrics/InGraphs system for graphing. But for alerting, we use an agent that polls the cr

Re: Kafka Monitoring

Using Elasticsearch, Logstash, and Kibana is a pretty popular pattern at LinkedIn. Also giving honorable mentions to Kafka Monitor and Kafka Manager since they hadn't been mentioned yet https://github.com/yahoo/kafka-manager https://github.com/linkedin/kafka-monitor Thanks, Andrew Hoblitzell Sr.

Re: Kafka Monitoring

You can look at enabling JMX on kafka ( https://stackoverflow.com/questions/36708384/enable-jmx-on-kafka-brokers) using JMXTrans (https://github.com/jmxtrans/jmxtrans) and a config ( https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md) to gather stats, and insert them into

Re: facing serialization issues with Kafka Streams application ..

Hi, You should provide the serdes in the `groupByKey()` operation. The `map` will trigger a re-partition in the `groupByKey` as you have changed the key. In fact you could replace the `map` and `groupByKey` with: m.groupBy(mapper, Serdes.String(), Serdes.String()).count("HostAggregateCount") Tha

Re: Kafka Monitoring

If you’re using confluent, you can use the control center. It’s not free however. From: Muhammad Arshad Reply-To: "users@kafka.apache.org" Date: Monday, June 19, 2017 at 5:52 PM To: "users@kafka.apache.org" Subject: Kafka Monitoring Hi, wanted to see if there is Kafka monitoring which is av

facing serialization issues with Kafka Streams application ..

Hi - I have the following Scala snippet in a Kafka streams application .. val builder = new KStreamBuilder() val logRecords: KStream[Array[Byte], LogRecord] = builder.stream(Serdes.ByteArray(), logRecordSerde, config.toTopic) val m: KStream[Array[Byte], String] = logRecords.mapValues(hostExtract

Re: ticketing system Design

It can be implemented on top of Kafka with some effort, but Kafka doesn't solve this problem for you directly. Amazon SQS does provide something like what you want, perhaps looking for other projects similar to SQS will yield some results. From: Abhimanyu Nag

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

+1 On 19/06/17 21:31, Vahid S Hashemian wrote: Thanks everyone. Great discussion. Because these Read or Write actions are interpreted in conjunction with particular resources (Topic, Group, ...) it would also make more sense to me that for committing offsets the ACL should be (Group, Write). S

Re: Kafka Streams: Problems implementing rate limiting processing step

I didn't know you could write to state stores from outside a processor/transformer. Interesting to hear that it is working although I'd be careful as KIP-67 warns it can introduce undefined behaviour: https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams

Mirroring Schema registry using mirror maker

Hi, I am trying to mirror _schemas topic (essentially schema registry) using mirror maker, which is not working. It only replicates newly created schemas even though auto.offset.reset is set to earliest. Any ideas? Producer Config: bootstrap.servers=:9092 group.id=mirrormaker exclude.interna