Re: Unable to start kafka server, 0.0.0.0:9092: Address already in use. Please help

Hi, Could you Post your zookeeper.properties and server.properties file details? Thanks, On 9 August 2017 at 07:08, shyla deshpande wrote: > I have even tried deleting the contents of log.dirs and dataDir before > starting the zookeeper and kafka server, still no success. I appreciate any > he

Kafka Streams Job | DirectoryNotEmptyException

Hi All, I wrote a Kafka Streams job that went running for 3-4 hrs, after which it started throwing these errors.Not sure why we got these errors. I am using Kafka 11.0 both on broker side as well as on consumer side. *Machine1* 2017-08-08 17:25:35 INFO StreamThread:193 - stream-thread [LICSp-7

Kafka Replication modes

Kafka supports two replication modes: Asynchronous and Synchronous replication. I couldnt find a property that allows you to switch between the two. Could someone please enlighten me on this. -Sameer.

Re: Kafka Replication modes

Not sure about, what you mean by Asynchronous and Synchronous replication. details about replication are here: http://kafka.apache.org/documentation/#replication Kafka producers can choose whether they wait for the message to be acknowledged by 0,1 or all (-1) replicas by using "acks" config prope

Re: Kafka Streams Job | DirectoryNotEmptyException

Hi, This is a bug in 0.11. You can work around it by setting StreamsConfig.STATE_DIR_CLEANUP_DELAY_MS_CONFIG to Long.MAX_VALUE Also, if you have logs it would be easier to either attach them or put them in a gist. It is a bit hard to read in an email. Thanks, Damian On Wed, 9 Aug 2017 at 10:10

Re: [kafka streams] 'null' values in state stores

Hi Guy, Indeed, I referenced the wrong source-code, sorry about that ;-) I created KAFKA-5717 for this. Thanks Bart On Tue, Aug 8, 2017 at 8:08 PM, Damian Guy wrote: > The change logger is not used during restoration of the in-memory-store. > Restoration is handled > https://github.com/apach

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

Thanks Ewen, I just edited the KIP to reflect the changes. Regards, Umesh On Wed, 9 Aug 2017 at 11:00 Ewen Cheslack-Postava wrote: > Great, looking good. I'd probably be a bit more concrete about the > Proposed Changes (e.g., "will log an warning if the config is specified" > and "since the Jso

ERROR Error when sending message to topic test01 with key: null, value: 4 bytes with error

Hi all, I have setup Kafka 0.10.2.1 with SSL enabled When trying console producer, I got the following error: My Command: ./bin/kafka-console-producer.sh --broker-list localhost:9093 --producer.config /hme/kafka/config/producer.properties --sync --topic test01 Error from console: ERROR Error w

Create Topic Error: Create Topic Error and cannot write to console producer

Hi, I have setup Kafka 0.10.2.1 with SSL. Check Status: openssl s_client -debug -connect n1:9093 -tls1 New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA ... SSL-Session: Protocol : TLSv1 PSK identity hint: None Start Time: 1502285690 Timeout : 7200 (sec) Verify ret

Streams: Fetch Offset 0 is out of range for partition foo-0, resetting offset

I have a small test setup with a local zk/kafka server and a streams app that loads sample data. The test setup is usually up for a day or two before a new build goes out and its blown away and loaded from scratch. Lately I've seen that after a few hours the stream app will stop processing and st

Re: Create Topic Error: Create Topic Error and cannot write to console producer

Hi, What's the status of your SSL? Have you verified that the setup is working? You can enable rough logins using log4j.properties file supplier with kafka and set the root logging level to DEBUG. This prints out more info to trace things. Also, you can enable security logging by adding -Djavax.s

Re: Random consumer offset rewinds/resets

Hi Ismael, thanks a lot for your for your answer, it was indeed exactly the issue we had! We did see the ticket for the issue before, but the steps to reproduce included a consumer group restart, which we didn't do, so that we thought we our problem is different. However, since patching with the

Kafka Producer/Consumer APIs vs Apache Camel

Hello All, Are their any advantages of using Kafka Producer/Consumer APIs over Apache Camel? Thanks!! -- NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution

auto.offset.reset defaulting to earliest

Hi, I am running standalone connector and explicitly setting auto.offset.reset parameter to latest in the properties file (which is default value). However, when I look at the logs, here's what I see: 2017-08-09 15:34:24,731] INFO Instantiated task hdfs-sink-cdpos_cdhdr_view-0 with version 3.2.0

Hostname verification details for TLS implementation

Hello, i have my test/development certficates created for X509 request extensions and SAN names cover: DNS.1 localhost > DNS.2 *.testsystem.net To make things more practical, I have used the advertised.listeners and listeners to ONLY SSL://localhost:9093. I have verified the certificates and c

Re: Unable to start kafka server, 0.0.0.0:9092: Address already in use. Please help

Hi Manna , Thanks for the response. All the properties in both the files are the defaults provided when we download kafka. I have listed them below. zookeeper.properties - dataDir=/tmp/zookeeper clientPort=2181 maxClientCnxns=0 server.properties ---

Re: Unable to start kafka server, 0.0.0.0:9092: Address already in use. Please help

Could you also provide the following outputs (assuming you have only 1 broker and zookeeper): 1) zookeeper-shell.sh localhost:2181 get /brokers/ids/0 from the output, can you see if your broker info is correct Also, in your server.properties where is advertised.listeners and listeners properties

Re: Unable to start kafka server, 0.0.0.0:9092: Address already in use. Please help

Hi Mannna. Thanks again. I see the problem, but not sure how to fix.. server.properties -- > listener settings -- listeners=PLAINTEXT://localhost:9092 advertised.listeners=PLAINTEXT://localhost:9092 $ bin/zookeeper-shell.sh localhost:2181 get /brok

Re: auto.offset.reset defaulting to earliest

In fact, bunch of parameters are not taking effect, no matter what I set it in config: auto.commit.interval.ms = 5000 enable.auto.commit = false Is it because the connector is setting consumer configs to default somewhere in the code and ignoring values in config file? Any help is appreciated.

Re: Unable to start kafka server, 0.0.0.0:9092: Address already in use. Please help

Shyla, Lets break it down into two sequential checks 1) I didn't see these ZK configs. Please gracefully shutdown everything, and add these to your zookeeper.properties before restarting ZK and broker: tickTime=1000 initLimit=3 syncLimit=7 Now run the previously mentioned commands for zookeeper

Re: Kafka Streams Job | DirectoryNotEmptyException

The issue was fixed by this: https://issues.apache.org/jira/browse/KAFKA-5562 it is on trunk, but will likely be back ported to 0.11 On Wed, 9 Aug 2017 at 10:57 Damian Guy wrote: > Hi, > > This is a bug in 0.11. You can work around it by setting > StreamsConfig.STATE_DIR_CLEANUP_DELAY_MS_CONFIG

Re: Create Topic Error: Create Topic Error and cannot write to console producer

Dear Manna, What's the status of your SSL? Have you verified that the setup is working? Yes, I used " openssl s_client -debug -connect n1.test.com:9092 -tls1 Output: CONNECTED(0003) write to 0x853e70 [0x89fd43] (155 bytes => 155 (0x9B)) - 16 03 01 00 96 01 00 00-92 03 01 59 8b 6d 0d

Re: Create Topic Error: Create Topic Error and cannot write to console producer

And, server.properties ## broker.id=11 port=9092 host.name=n1 advertised.host.name=192.168.0.11 allow.everyone.if.no.acl.found=true super.users=User:CN=n1.test.com,OU=TEST,O=TEST,L=TEST,ST=TEST,C=TEST listeners=SSL://n1.test.com:9092 advertised.listeners=SSL://n1.test.com:9092 ssl.cli

Re: Create Topic Error: Create Topic Error and cannot write to console producer

Your openssl test is showing connected with port 9092. but your previous messages show 9093 - is there some typo issues? Where is SSL running Please share the following and don't leave any details out. This will only create more assumptions. 1) server.properties 2) Zookeeper.properties Also, run

Re: Create Topic Error: Create Topic Error and cannot write to console producer

server.properties ## broker.id=11 port=9093 host.name=n1 advertised.host.name=192.168.0.11 allow.everyone.if.no.acl.found=true super.users=User:CN=n1.test.com,OU=TEST,O=TEST,L=TEST,ST=TEST,C=TEST listeners=SSL://n1.test.com:9093 advertised.listeners=SSL://n1.

Re: Create Topic Error: Create Topic Error and cannot write to console producer

FYI, about zookeeper, I used my existing zookeeper (as I have existing zookeeper up and running, which is also used for hbase) zookeeper versoom: 3.4.10 zoo.cfg ## tickTime=2000 initLimit=10 syncLimit=5 dataDir=/usr/local/zookeeper/data dataLogDir=/usr/local/zookeeper/datalog clientPort

Re: Create Topic Error: Create Topic Error and cannot write to console producer

About: zookeeper-shell.sh localhost:2181 get /brokers/ids/11 The result: zookeeper-shell.sh n1.test.com:2181 Connecting to n1.test.com:2181 Welcome to ZooKeeper! JLine support is disabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null WATCHER:: WatchedEvent state:SaslAuthe

Re: Create Topic Error: Create Topic Error and cannot write to console producer

About: zookeeper-shell.sh localhost:2181 get /brokers/ids/11 The result: zookeeper-shell.sh n1.test.com:2181 Connecting to n1.test.com:2181 Welcome to ZooKeeper! JLine support is disabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null WATCHER:: get /brokers/ids/11 Watc

Re: Create Topic Error: Create Topic Error and cannot write to console producer

( I have 3 test nodes) get /brokers/ids/11 {"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL:// n1.test.com:9093 "],"jmx_port":-1,"host":null,"timestamp":"1502310695312","port":-1,"version":4} cZxid = 0x40002787d ctime = Thu Aug 10 04:31:37 HKT 2017 mZxid = 0x40002787d mtime =

Re: Create Topic Error: Create Topic Error and cannot write to console producer

Dear Manna, Where can I set "-Djavax.security.debug=all" for Kafka? Regards On Thu, Aug 10, 2017 at 5:08 AM, Ascot Moss wrote: > ( I have 3 test nodes) > > get /brokers/ids/11 > > {"listener_security_protocol_map":{"SSL":"SSL"},"endpoints":["SSL:// > n1.test.com:9093"],"jmx_port":-1,"host":nu

Re: Create Topic Error: Create Topic Error and cannot write to console producer

if you remove host.name, advertised.host.name and port from server.properties, does it work for you? I am using SSL without ACL. it seems to be working fine. On 9 August 2017 at 22:03, Ascot Moss wrote: > About: > zookeeper-shell.sh localhost:2181 > get /brokers/ids/11 > > > The result: > > zoo

Re: Create Topic Error: Create Topic Error and cannot write to console producer

I commented out both #host.name, #advertised.host.nam (new server.properties) broker.id=11 port=9093 #host.name=n1.test.com #advertised.host.name=192.168.0.11 allow.everyone.if.no.acl.found=true super.users=User:CN=n1.test.com,OU=TEST,O=TEST,L=TEST,ST=TEST,C=TEST listeners=SSL://n1.test.com:9093 a

How to debug - NETWORK_EXCEPTION

Hi I am sending very small 32 byte message to Kafka broker in a tight loop with 250ms sleep. I have one broker, 1 partition, and replication factor = 1. After about 4200 messages, I get *following *error pasted below. How can I debug this error ? Can you please throw some ideas for me to debug ?

Continuous latency in acknowledgment when leader broker gets down

HI, I am running Kafka locally in my MAC machine with single node and 3 brokers, each running in different port. I was trying the Kafka PoC with having multiple brokers up and running and sending messages to Kafka and in between failing some nodes (leader nodes). There are some observations as b

Re: Kafka Streams Job | DirectoryNotEmptyException

Sure, I will attach next time. Thanks for your help. -Sameer. On Wed, Aug 9, 2017 at 3:27 PM, Damian Guy wrote: > Hi, > > This is a bug in 0.11. You can work around it by setting > StreamsConfig.STATE_DIR_CLEANUP_DELAY_MS_CONFIG to Long.MAX_VALUE > > Also, if you have logs it would be easier to

How performant is the confluent REST proxy

We are evaluating using the kafka for integrating into a platform. One option is to use a rest proxy to simplify interaction with an API backend, as opposed to directly having a consumer in a backend technology like Node or Laravel. One concern that cropped up is the performance numbers for REST p