Re: Kafka Connect CPU spikes bring down Kafka Connect workers

On Mon, May 15, 2017 at 2:06 PM, Phillip Mann wrote: > Currently, Kafka Connect experiences a spike in CPU usage which causes > Kafka Connect to crash. What kind of crash? Can you provide an error or stacktrace? > There is really no useful information from the logs to help me > understand w

Why cannot kakfka copy data to a new replica automatically when one of replica is down?

I have a kafka clusters with 3 brokers. 192.168.80.82,192.168.80.146 and 192.168.80.110 This is the topic description named DruidAppColdStartMsg: [appuser@hz-192-168-80-146 kafka]$ bin/kafka-topics.sh --describe --topic DruidAppColdStartMsg --zookeeper 10.120.241.50:2181 Topic:DruidAppColdStartM

processing JSON

Should you serialize JSON as part of the producer or consumer? Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.linkedin.com/in/bobwakefieldmba Twitter: @BobLovesData

Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

Hi All, Given the simple and non-controversial nature of the KIP, I would like to start the voting process for KIP-157: Add consumer config options to streams reset tool *https://cwiki.apache.org/confluence/display/KAFKA/KIP+157+-+Add+consumer+config+options+to+streams+reset+tool

Re: Debugging Kafka Streams Windowing

Thanks for the reply Guozhang! But I think we are talking of 2 different issues here. KAFKA-5167 is for LockException. We face this issue intermittently, but not a lot. There is also another issue where a particular broker is marked as dead for a group id and Streams process never recovers from th

Kafka Connect CPU spikes bring down Kafka Connect workers

Currently, Kafka Connect experiences a spike in CPU usage which causes Kafka Connect to crash. There is really no useful information from the logs to help me understand what is causing this to happen. Is this a known issue? If it matters, my configuration settings are as follows: format.clas

Re: Kafka-streams process stopped processing messages

I do run the clients with 0.10.2.1 and it takes hours What I don't understand is why it takes hours to boot on a server that has all the data in RocksDB already. Is that related to the amount of data in RocksDB (changelog topics) or the data in the source topic the processes reads from? On Mon, 15

Re: kafka_2.11-0.10.0.1 - windows - line 84

This issue seems already resolved in latest 0.10.2 release, could you double check? Also could you describe a bit more details on the observed error messages etc? On Mon, May 15, 2017 at 4:40 AM, Regradj Hakim wrote: > Hello, > > I wanted to report a comment error that generates a message on th

Re: Debugging Kafka Streams Windowing

I'm wondering if it is possibly due to KAFKA-5167? In that case, the "other thread" will keep retrying on grabbing the lock. Guozhang On Sat, May 13, 2017 at 7:30 PM, Mahendra Kariya wrote: > Hi, > > There is no missing data. But the INFO level logs are infinite and the > streams practically s

Re: Kafka-streams process stopped processing messages

Hello Shimi, Could you try upgrading your clients to 0.10.2.1 (note you do not need to upgrade your servers if it is already on 0.10.1, since newer Streams clients can directly talk to older versioned brokers since 0.10.1+) and try it out again? I have a few optimizations to reduce rebalance laten

Re: Strict ordering of messages in Kafka

You can do it, but the performance trade off will be very costly. There is no ordering guarantee between partitions, so you would have to work with topics with only one partition. I believe you would also need to set the batch.size in the producer to 1 to ensure the messages aren't re-ordered withi

Re: Strict ordering of messages in Kafka

Afaik that is not possible out of the box as that would require synchronization across multiple threads/instances. The throughput of such an approach would be terrible as the parallelism of KStreams is tightly coupled with the number of partitions. I'd say if you need such ordering you should reco

Strict ordering of messages in Kafka

Hi, Is it that there is no way to guarantee strict ordering of messages within Kafka across partitions? Within a single partition, the only way for strict ordering is to set max.in.flight.requests.per.connection=1 and acks=all ? Thanks, Janette Janette Wong, P.Eng. | janette.w...@rbc.com | Sen

kafka_2.11-0.10.0.1 - windows - line 84

Hello, I wanted to report a comment error that generates a message on the CLI ('#' ...) as I started Kafka from the windows script. kafka_2.11-0.10.0.1\bin\windows\kafka-run-class.bat I did not find that case on Jira, nor how to contribute directly. https://issues.apache.org/jira/browse/KAFKA-52

Re: Brokers is down by “java.io.IOException: Too many open files”

If you're using a systemd based OS you'll actually need to set it in the unit file. LimitNOFILE=10 https://kafka.apache.org/documentation/#upgrade_10_1_breaking contains some changes re file handles as well. __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486

Re: Increase speed of replica fetch in kafka?

Have you tried using more partitions for the topic? Am 05.05.2017 um 23:39 schrieb Girish Aher: Have you tried increasing num.replica.fetchers on your brokers? On Fri, May 5, 2017 at 12:59 PM, Archie wrote: Is there anyway I can speed up the rate at which the slave replicas will fetch data

Re: Repartition topics in Kafka Streams when there is only one partition

Hi Frank, It doesn't make a lot of sense to create repartition topics for single partition input topics. However the way it currently works we don't have the partition information at hand when making the decision that we need to repartition. It is purely based on the key potentially changing. The

Repartition topics in Kafka Streams when there is only one partition

Hi, I'm using quite a lot of topics, and the speed on each topic isn't extreme, so I generally use a single partition. In this situation, the high level API of Kafka Streams still makes repartition topics, but I don't see there use. Wouldn't it make sense to skip repartition topics in this case,

Re: Kafka Streams reports: "The timestamp of the message is out of acceptable range"

Ah. I updated the Streams app, the brokers still ran a slightly older version. Updating the brokers seems to fix this. Thanks. On Mon, May 15, 2017 at 8:50 AM, Eno Thereska wrote: > Hi Frank, > > Could you confirm that you're using 0.10.2.1? This error was fixed ad part > of this JIRA I believe