Re: Streams 'External source topic not found' while using groupBy/aggregate

Hi Guozhang, I tested your backport branch, and it looks like it works fine. The same code that was producing the "Invalid topology building" error is working correctly against your branch, just like it is with trunk. Mathieu On Sun, Aug 14, 2016 at 11:25 PM, Guozhang Wang wrote: > Hi Mathie

Automated Testing w/ Kafka Streams

Hey all, At my workplace, we have a real focus on software automated testing. I'd love to be able to test the composition of a TopologyBuilder with org.apache.kafka.test.ProcessorTopologyTestDriver

Regarding properties in Apache Kafka

Hello, This is Vijju. I have some questions related to properties of Kafka. 1. Can we change the topic name that a producer is writing to 2. Can we change the bootstrap server list 3. Can we change the consumer group name 4. Can we change the number of consumers etc. and be able to verify and mea

kafka 0.9 running on AWS environment caching zookeeper IP instead of using the DNS

Hi All, We are using kafka version 0.9 and we are facing issues, in kafka connecting to zookeeper. Here are mode details: Zookeeper is in front AWS Elastic Load Balancer and the ELB has been associated with a DNS name, say zookerper1.com. Our broker configuration looks the below zookeeper.

Re: kafka 0.9 running on AWS environment caching zookeeper IP instead of using the DNS

This recently came up on the list, check out: http://search-hadoop.com/m/uyzND12ZzwunNJzS&subj=Kafka+DNS+Caching+in+AWS On Mon, Aug 15, 2016 at 10:46 AM, Digumarthi, Prabhakar Venkata Surya < prabhakarvenkatasurya.digumar...@capitalone.com> wrote: > Hi All, > > > We are using kafka version 0.9 an

Re: kafka 0.9 running on AWS environment caching zookeeper IP instead of using the DNS

Also, short answer is you have to roll your own system with EIPs and use the EIPs directly to ensure cache issues don't occur (ie resolve the EIP DNS to an internal IP and then that is cycled). Otherwise you should be in a VPC with known IPs so no problem then. On Mon, Aug 15, 2016 at 11:11 AM, Jo

Re: Getting maximum message size

The maximum message size is all dependent on the message max bytes you set on the topic (or global broker). You want to make sure the consumer max is in sync with the broker max. Remember that pumping huge messages though can saturate your network quickly. Try to trim down message structure to onl

Change Apache Kafka properties dynamically without restarting servers using Archaius

Hello, I have some questions related to properties of Apache Kafka. We have a four node Kafka Cluster which is on Amazon EMR. Currently, in order to change any properties we do restart the servers to pick the new changes made. For the following: 1. Can we change the topic name that a producer is

Re: Automated Testing w/ Kafka Streams

Mathieu, follow-up question: Are you also doing or considering integration testing by spawning a local Kafka cluster and then reading/writing to that cluster (often called embedded or in-memory cluster)? This approach would be in the middle between ProcessorTopologyTestDriver (that does not spaw

DLL Hell

kafka-trunk\streams>gradle buildCaused by: java.lang.RuntimeException: librocksdbjni-win64.dll was not found inside JAR.at org.rocksdb.NativeLibraryLoader.loadLibraryFromJarToTemp(NativeLibraryLoader.java:106) at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoad

Re: Automated Testing w/ Kafka Streams

Hi Michael, It would definitely be an option. I am not currently doing any testing like that; it could replace the ProcessorTopologyTestDriver-style testing that I'd like to do, but there are some trade-offs to consider: - I can't do an isolated test of just the TopologyBuilder; I'd be bri

Re: DLL Hell

Hi Martin, rocksdb does not currently distribute a Windows-compatible build of their rocksdbjni library. I recently wrote up some instructions on how to produce a local build, which you can find here: http://mail-archives.apache.org/mod_mbox/kafka-users/201608.mbox/%3CCAHoiPjweo-xSj3TiodcDVf4wNnn

Re: Automated Testing w/ Kafka Streams

Out of curiosity, are you aware of kafka.util.TestUtils and Apache Curator TestingServer? I’m using this successfully to test publis / consume scenarios with things like Flink, Spark and custom apps. What would stop you from taking the same approach? – Best regards, Radek Gruchalski ra...@gruchals

Re: Automated Testing w/ Kafka Streams

Mathieu, Your composition of Per-module Unit Tests + ProcessorTopologyTestDriver + System Tests looks good to me, and I agree with you that since this is part of your pre-commit process, which could be triggered concurrently from different developers / teams, EmbeddedSingleNodeKafkaCluster + Embed