Re: Use Kafka Streams for windowing data and processing each window at once

Hello Guozhang, Managing to have a stream with just one record per each key per window is definitely something we want to achieve. But, it is not all. We also want to process the whole window at once so our callback would receive just one collection of aggregates per window once it is ready.  We

Documentation/Article/Case Study on Scala as the Kafka Backbone Language

Hello, Is anyone aware of any links or website where I can find information/case study etc. to why Scala was the best choice for kafka design? I hope this is not too much of a "Naive" question since I have had a very humble introduction to Scala. I understand that Scala is considered where distri

Re: UNKNOWN_PRODUCER_ID error when running Streams WordCount demowith processing.guarantee set to EXACTLY_ONCE

Hello Guozhang, Bill Thanks much for your explanation. I assigned this JIRA issue to myself and will try to fix it.   Regards, Lambdaliu(Shaobo Liu)   On 2018-07-21 at 02:24 Bill Bejeck   wrote:   >Created JIRA https://issues.apache.org/jira/browse/KAFKA-7190 > >-Bill > >On Fri, Jul 20, 2018 at 1:

Debugging message timestamps in Sarama

Hi! I'm working on debugging a problem with how message timestamps are handled in the sarama client. In some cases, the sarama client won't associate a timestamp with a message while the kafka console consumer does. I've found the documentation on the message format here: https://kafka.apache.o

Re: Documentation/Article/Case Study on Scala as the Kafka Backbone Language

Not necessarily for Kafka, but you can definitely google “Java vs. Scala” and find a variety of reasons . I did a study for a client and ultimately here are the major reasons I found : 1. Functional programming language which leads itself to stateless systems 2. Better / easier to use stream pro

Re: Debugging message timestamps in Sarama

Hey Craig, what exact problem you have with Sarama client? On Mon, Jul 23, 2018 at 5:11 PM Craig Ching wrote: > Hi! > > I'm working on debugging a problem with how message timestamps are handled > in the sarama client. In some cases, the sarama client won't associate a > timestamp with a messa

Graph Processing with Kafka

Hi, If anyone is interested in using Kafka for graph processing, I started a project to try to use Kafka Streams for that purpose. Feedback is most welcome! https://yokota.blog/2018/07/23/kafka-graphs-graph-analytics-with-apache-kafka/ https://github.com/rayokota/kafka-graphs

Re: Use Kafka Streams for windowing data and processing each window at once

I see. In that case, one workaround would be to query the state store directly after you know that no more updates would be applied to that store in a `punctuation` function: note that punctuation is a feature that's only available in the Processor API, but you can always add such a lower-level im

Re: Debugging message timestamps in Sarama

Hi Dmitry, Are you associated with the Sarama project? If so, understand that part of what I want is to learn about Sarama and the Kafka message format ;) The problem I'm having is that if I turn on: log.message.timestamp.type=LogAppendTime in the broker, then produce on topic1 with console pr

Kafka Streams throughput performance overtime

Hi, I have a stream application that performs rollup from 15mins to Hourly, then Hourly to Daily. The process needs to be continuously run 24 hours and each 15 mins approx 12 million records gets posted (a JSON record per message) into the input topic. There are 3 separate processors correspondi

Kafka Broker : Socket.request.max.bytes & Socket.receive.buffer.bytes

Hi All, I am trying to increase my Kafka producer client throughput and for same, i was exploring server.properties file parameters. I would like to better understand the relation between "Socket.request.max.bytes" and "Socket.receive.buffer.bytes" parameter sets in broker's server.properties fil