Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released

great work from the community. Thanks for driving this huge release! On Thu, Jun 29, 2017 at 12:17 PM, Paolo Patierno wrote: > Thank you all for great contributions and Ismael for driving this release ! > > > Paolo Patierno > Senior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows Em

Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released

Thank you all for great contributions and Ismael for driving this release ! Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoT Microsoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno

Re: enabling 2 security protocols and ACL

We need to give Describe/Write privileges for producer user and Describe/Read privileges for the consumer user. While using kafka-acls.sh script, you can use "--producer/--consumer" options to give a user producer or consumer privileges. You can find examples here: https://kafka.apache.org/docume

Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released

+1. Thank you Ismael for your hard work on this release. --Vahid From: Guozhang Wang To: "d...@kafka.apache.org" Cc: "users@kafka.apache.org" , annou...@apache.org, kafka-clients Date: 06/28/2017 07:22 PM Subject:Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released Ismael,

mirroring Kafka while preserving the order

Is it possible to configure mirror maker using message handler to preserve the order of messages in each topic partition. In the particular use case we're looking at both source and destination topics have same number of partitions. Sunil

Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released

Ismael, Thanks for running this release! Guozhang On Wed, Jun 28, 2017 at 5:57 PM, Jun Rao wrote: > Hi, Ismael, > > Thanks a lot for running this release! > > Jun > > On Wed, Jun 28, 2017 at 5:52 PM, Ismael Juma wrote: > > > The Apache Kafka community is pleased to announce the release for Ap

Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released

Hi, Ismael, Thanks a lot for running this release! Jun On Wed, Jun 28, 2017 at 5:52 PM, Ismael Juma wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.11.0.0. This is a feature release which includes the completion > of 32 KIPs, over 400 bug fixes and i

[ANNOUNCE] Apache Kafka 0.11.0.0 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.11.0.0. This is a feature release which includes the completion of 32 KIPs, over 400 bug fixes and improvements, and more than 700 pull requests merged. All of the changes in this release can be found in the release n

Kafka logs - server.log & controller.log

Hi All - in Kafka, i see two different log files - server.log & controller.log files. What specifically gets logged in these log files ? i.e. what are these 2 files meant to log ?

Re: Kafka Sometimes Fails to Start on Boot

Hello, Unfortunately Kafka does indeed startup and run for a little bit before crashing with the above exception, so doing one simple check wouldn't work. I could theoretically keep this script running forever, and constantly checking for it being up. However that's really a hacky solution, and I

Re: Kafka Sometimes Fails to Start on Boot

Can you not put a service wrapper for startup? It will attempt a restart if the executable isn't up and running successfully. I am not familiar with Unix side, but in Windows you can use a powershell to utilise such thing. It's a better approach. Let me know what you think. On 28 Jun 2017 8:34 p

Re: Kafka Sometimes Fails to Start on Boot

I am using the same configuration for all brokers. However, each broker is running on a completely separate host (I'm not running all three brokers on the same host). I can get all three running if I manually start kafka again, however it's just occasionally on boot one fails to start with this err

Re: Kafka Sometimes Fails to Start on Boot

Aren't u using the same JMX port for all brokers? I dont think it will work for more than 1 broker. On 28 Jun 2017 8:22 pm, "Eric Coan" wrote: > Hey, > > No worries. I'm starting the brokers with a script yes (that ends up > generating the command I pasted: > > ``` > > KAFKA_JMX_OPTS="-Dc

Re: Kafka Sometimes Fails to Start on Boot

Hey, No worries. I'm starting the brokers with a script yes (that ends up generating the command I pasted: ``` KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=$FQDN -Djava.n

Re: Kafka Sometimes Fails to Start on Boot

Please forgive my autocorrect options :( On 28 Jun 2017 8:06 pm, "M. Manna" wrote: Hi, OS is not an issue, I have a 3 broker setup and I have experienced this too. How are toy atarting the brokers? Is this a concurrent start or have you got some startup scriptto bring up all the brokers? KR,

Re: Kafka Sometimes Fails to Start on Boot

Hi, OS is not an issue, I have a 3 broker setup and I have experienced this too. How are toy atarting the brokers? Is this a concurrent start or have you got some startup scriptto bring up all the brokers? KR, On 28 Jun 2017 6:47 pm, "Eric Coan" wrote: > Hello, > > I've recently been doing re

Kafka Sometimes Fails to Start on Boot

Hello, I've recently been doing research into getting our Kafka cluster running outside of Mesos (for a couple of reasons). However I'm noticing about 10% of the time Kafka fails to start on boot (or more accurately starts, and immediately exits). I find it weird since all brokers are using the ex

Re: [ERICSSON] - Trade Compliance: ECCN code for Apache Items

@Martin Gainty - see my link: "The Apache Software Foundation (ASF) is a 501(c)3 nonprofit charity based in the United States of America. All of our products are developed via online collaboration in public forums and distributed from a central server within the U.S. Therefore, U.S. export laws an

Re: kafka-streams app(s) stopped consuming new events

Nothing for stat-change.log for giving time window. Last line logged 4 hours before app stopped. Any ideas so far? Personally i don't see anything relevant in logs. On Wed, Jun 28, 2017 at 6:33 PM, Bill Bejeck wrote: > Sure, couldn't hurt. > > Thanks, > Bill > > On Wed, Jun 28, 2017 at 9:51 AM,

Re: Weird broker lock-up causing an almost global downtime

Thanks for the info Vincent. -Bill On Wed, Jun 28, 2017 at 12:19 PM, Vincent Rischmann wrote: > I'm not sure what exactly to search for, but here are a couple of > things. > > First, looking at the broker 1, there's this: > https://pastebin.com/raw/XMLpjj7J which I find weird, because I'm sure

Re: Weird broker lock-up causing an almost global downtime

I'm not sure what exactly to search for, but here are a couple of things. First, looking at the broker 1, there's this: https://pastebin.com/raw/XMLpjj7J which I find weird, because I'm sure I didn't create those topics manually...which means to me that the streams application was still up (despit

Re: kafka-streams app(s) stopped consuming new events

Sure, couldn't hurt. Thanks, Bill On Wed, Jun 28, 2017 at 9:51 AM, Dmitriy Vsekhvalnov wrote: > Here are logs: > > app: > https://gist.github.com/dvsekhvalnov/f98afc3463f0c63b1722417e3710a8 > e7#file-kafka-streams-log > brokers: > https://gist.github.com/dvsekhvalnov/8e870f7347394e8d004c282880e

enabling 2 security protocols and ACL

Hi, I am working on enabling both PLAINTEXT and SASL_PLAINTEXT together on 2 ports so that clients have time to transition to SASL_PLAINTEXT without impact. The listeners work well, but I had some queries regarding ACL. If I enable the SimpleAuthorizer and add Read and Write ACL rules for the clie

Re: kafka-streams app(s) stopped consuming new events

Here are logs: app: https://gist.github.com/dvsekhvalnov/f98afc3463f0c63b1722417e3710a8e7#file-kafka-streams-log brokers: https://gist.github.com/dvsekhvalnov/8e870f7347394e8d004c282880ef385a#file-kafka-broker-1-2-3-log All broker logs are same, so single gist. There are also state-change.log fi

Re: [ERICSSON] - Trade Compliance: ECCN code for Apache Items

MG>am requesting clarification below From: Axelle Margot Sent: Tuesday, June 27, 2017 8:48 AM To: users@kafka.apache.org Cc: Celine Heerdt; Wassila BenJelloul.ext; Florent Poulain Subject: [ERICSSON] - Trade Compliance: ECCN code for Apache Items Hello, You wer

Re: kafka-streams app(s) stopped consuming new events

Hi Dmitry, At the moment I don't have anything specific to look for, just trying to get more context around the issue. As for the logs maybe broker and streams logs for the last 30 minutes up to the time the application stopped processing records. Thanks, Bill On Wed, Jun 28, 2017 at 9:04 AM, D

Re: kafka-streams app(s) stopped consuming new events

Hi Bill, 1. sure, can extract some logs, what exactly to look for? There are 11 hours of logs and most of them looks like: [2017-06-27 03:30:50,553] [] [INFO ] [StreamThread-1] [org.apache.kafka.streams.processor.internals.StreamThread] [stream-thread [StreamThread-1] Committing all tasks because

Re: kafka producer latency

I have now moved my brokers to local disks (though not dedicated disks). Running the tests again, although there is a significant improvement in max latency during steady state operation, still when I take down the kafka node that has the zookeeper leadership, I get 10-15 second max latencies.

Re: kafka-streams app(s) stopped consuming new events

Hi Dimitry, I'm happy to help, but I could use more information. Can you share the streams logs and broker logs? Have you confirmed messages are still being delivered to topics (via console consumer)? Thanks, Bill On Wed, Jun 28, 2017 at 8:24 AM, Dmitriy Vsekhvalnov wrote: > Hi all, > > look

kafka-streams app(s) stopped consuming new events

Hi all, looking for some assistance in debugging kafka-streams application. Kafka broker 0.10.2.1 - x3 Node cluster kafka-streams 0.10.2.1 - x2 application nodes x 1 stream thread each. In streams configuration only: - SSL transport - kafka.streams.commitIntervalMs set to 5000 (instead of d

Re: 0.11.0.0 RC2 Vote

Thanks Guozhang. I fixed the quickstart references to use a version variable. Ismael On Tue, Jun 27, 2017 at 11:57 PM, Guozhang Wang wrote: > My vote on the original VOTE thread somehow cannot show up, I have tried > several ways and nothing works. So I am sending it as a new email which > shou