Re: PR Review Request

2019-06-14 Thread Matthias J. Sax
Request like this should go to the dev list. -Matthias On 6/14/19 2:16 PM, M. Manna wrote: > Hello, > > Could this be reviewed? Apologies for spamming the DL with such minor > things. > > https://github.com/apache/kafka/pull/6771 > > Thanks, > > On Thu, 30 May 2019 at 16:43, M. Manna wrote:

Issue facing, Aerospike Connect for Kafka - Inbound

2019-06-14 Thread Mich Talebzadeh
I generate a simple pricing as follows: echo "${UUID}:${TICKER}" >> ${IN_FILE} UUID is the key and ticker is the name of security like “IBM” This is what I have in my shell script cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh \ --broker-list rhes75:909

Re: PR Review Request

2019-06-14 Thread M. Manna
Hello, Could this be reviewed? Apologies for spamming the DL with such minor things. https://github.com/apache/kafka/pull/6771 Thanks, On Thu, 30 May 2019 at 16:43, M. Manna wrote: > Hello, > > Could someone review/merge the PR ? > > https://github.com/apache/kafka/pull/6771 > > It had some t

Re: How to process KStream windowed message parallelly

2019-06-14 Thread Matthias J. Sax
There is no built-in support for this atm. Async processing support as suggested via KIP-408 might help in the future. But there in not much activity on this KIP atm. (https://cwiki.apache.org/confluence/display/KAFKA/KIP-408%3A+Add+Asynchronous+Processing+To+Kafka+Streams) > If I add submit Thre

Re: Kafka streams dropping events in join when reading from earliest message on topic

2019-06-14 Thread Matthias J. Sax
How do you know that the result should be 900,000 messages? Did you verify that the keys match and that the timestamps are correct? Did you try to remove grace-period or set a higher value? Maybe there is an issue with ouf-of-order data? -Matthias On 6/14/19 5:05 AM, giselle.vandon...@klarrio.co

Re: Kafka Streams (DSL API) - Any side effects of having more sub topologies?

2019-06-14 Thread Matthias J. Sax
>> We are using Kafka streams 2.1.1 and leveraging DSL APIs for doing stateful >> processing (aggregation). As per our use case we ended up creating ~400 sub >> topologies. >> Our observation is that based on the number of partitions in the >> input/source topic, corresponding stream tasks get

Re: Kafka-streams: Mapping between number of stream threads and CPU cores?

2019-06-14 Thread Matthias J. Sax
There is no mapping. As a rule of thumb I would say 4 to 8 thread should be ok on a 4-core machine. But you might want to monitor CPU utilization to get more insight. -Matthias On 6/13/19 10:03 AM, emailtokir...@gmail.com wrote: > Hi, > > Is there any mapping between number of stream threads an

Re: All topic are assigned to same consumer in a consumer group

2019-06-14 Thread Matthias J. Sax
What `PartitionAssignor` did you configure? Are you by any change using Kafka Streams? For this case, it's expected, because Kafka Streams co-partitions during the assignment. -Matthias On 6/12/19 11:27 PM, sunilv...@gmail.com wrote: > I have 100 topics with one partition each. > I have 10 inst

Re: Retrieve open windows

2019-06-14 Thread Matthias J. Sax
You can scan the whole window-store via IQ. -Matthias On 6/12/19 10:02 PM, Parthasarathy, Mohan wrote: > Hi, > > Is there a way to know about the currently open windows that is currently not > expired yet ? We are trying to track down a problem where some messages are > read by the application

Kafka streams dropping events in join when reading from earliest message on topic

2019-06-14 Thread giselle . vandongen
I have two streams of data flowing into a Kafka cluster. I want to process this data with Kafka streams. The stream producers are started at some time t. I start up the Kafka Streams job 5 minutes later and start reading from earliest from both topics (about 900 000 messages already on each topi