Issue with Samza 0.14.1 and Kafka 1.1.0 in handling OffSetOutOfRangeException

Hi We are facing an issue with Samza 0.14.1 and Kafka 1.1.0. The details have been posted in samza mailing list and stackoverflow

GenericRecord.toString produces invalid JSON for logical types

Hi All, I have a serialized avro binary data represented by byte[] where one of the field is long with a logical type of TimeStamp. Timestamp tsp = new Timestamp(1530228588182l); Schema schema = SchemaBuilder.builder() .record("hello") .fields() .name("

Re: kafka stream latency

Hello Nan, Kafka does not tie up the processing thread to do disk flushing. However, since you are on an older version of Kafka I suspect you're bumping into some old issues that have been resolved in later versions. e.g. https://issues.apache.org/jira/browse/KAFKA-4614 I'd suggest you upgrading

Exposing Kafka on WAN

Hello, We have a requirement for opening Kafka on WAN where external producers and consumers need to be able to talk to Kafka. I was able to get Zookeeper and Kafka working with two way SSL and SASL for authentication and ACL for authorization. However, my concern with this approach was opening u

Re: kafka stream latency

I will wait for the expert’s opinion: Did the Transparent Huge Pages(THP) disabled on the broker machine? it’s a Linux kernel parameter. -Sudhir > On Aug 23, 2018, at 4:46 PM, Nan Xu wrote: > > I think I found where the problem is, how to solve and why, still not sure. > > it related to disk

Re: kafka stream latency

I think I found where the problem is, how to solve and why, still not sure. it related to disk (maybe flushing?). I did a single machine, single node, single topic and single partition setup. producer pub as 2000 message/s, 10K size message size. and single key. when I save kafka log to the mem

Can I configure with ACL for some clients but not others

Hi I have a system which is in network A (inside my premises) And all my current components publish to KAFKA with no security Now my customer needs to use my KAFKA from “outside” And therefore I now need to introduce SASL and ACL ACL so that I can limit to what they publish to and read from. BUT

Re: Performance Impact with Apache Kafka Security

Hi, Which Kafka version and Java version are you using? Did you try this with Java 9 which has 2.5x perf improvements over Java 8 for SSL? Can you try using a slightly weaker cipher suite to improve the performance? -Harsha On Wed, Aug 22, 2018, at 1:11 PM, Sri Harsha Chavali wrote: > Hi

Re: Kafka Connect REST connector with additional logging to kafka

Hi Andrew, Thank you for your answer. Sadly I'm not allowed to add this module to my technologies by now. 2018-08-23 16:00 GMT+02:00 Andrew Otto : > Hiya, this doesn’t help answer your question, but as an FYI, Wikimedia has > implemented https://github.com/wikimedia/change-propagation to do what

Re: Kafka Connect REST connector with additional logging to kafka

Hiya, this doesn’t help answer your question, but as an FYI, Wikimedia has implemented https://github.com/wikimedia/change-propagation to do what you are trying to do: issue HTTP requests (and other things) triggered by incoming messages in Kafka. On Thu, Aug 23, 2018 at 9:45 AM Andrea Spina wrot

Kafka Connect REST connector with additional logging to kafka

Dear community, I'd figure out a kafka connect HTTP/REST sink connector X capable of achieving PUT actions on an external web service A. I saw some side projects already implementing something similar [1]. In addition to this feature, I'd like to log to kafka the answers returning by A so that I c

Re: Cannot create topics when zookeeper.set.acl=true

But thanks anyway for the quick answer. Op do 23 aug. 2018 om 14:38 schreef HG : > Well it works fine when I do : > export > "KAFKA_OPTS=-Djava.security.auth.login.config=/u01/kafka/config/kafka_server_jaas.conf" > > Op do 23 aug. 2018 om 14:25 schreef Manikumar : > >> *zk does NOT support Pla

Re: Cannot create topics when zookeeper.set.acl=true

Well it works fine when I do : export "KAFKA_OPTS=-Djava.security.auth.login.config=/u01/kafka/config/kafka_server_jaas.conf" Op do 23 aug. 2018 om 14:25 schreef Manikumar : > *zk does NOT support PlainLoginModule.* > > On Thu, Aug 23, 2018 at 5:54 PM Manikumar > wrote: > > > No, zk does suppo

Re: Cannot create topics when zookeeper.set.acl=true

*zk does NOT support PlainLoginModule.* On Thu, Aug 23, 2018 at 5:54 PM Manikumar wrote: > No, zk does support PlainLoginModule. while using kafka-acls.sh script > with kerberized zk, > we need to pass required kerberos credentials. > > AdminClient API is an api to perform administrative action

Re: Cannot create topics when zookeeper.set.acl=true

No, zk does support PlainLoginModule. while using kafka-acls.sh script with kerberized zk, we need to pass required kerberos credentials. AdminClient API is an api to perform administrative actions (create/delete topics, create/delete acls etc..). This avoids direct communication with zk. Check be

Re: Cannot create topics when zookeeper.set.acl=true

Hi, I am not using kerberos only Client { org.apache.kafka.common.security.plain.PlainLoginModule required username="user" password="user-secret"; }; Does that make a difference? What do you mean with AdminClient API? Regards Hans Op do 23 aug. 2018 om 13:34 schreef Manikumar : > We

Re: Cannot create topics when zookeeper.set.acl=true

We can pass jaas conf by exporting below variable before starting the kafka-acls.sh script. another option is to use AdminClient API. export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=zk_client_jaas.conf" zk_client_jaas.conf: // Zookeeper client authent

Re: Cannot create topics when zookeeper.set.acl=true

Hi, I searched for an option with which I can provide credentials but I did not find them. Is there another way to reach the same goal? Regards Hans Op do 23 aug. 2018 om 13:00 schreef Manikumar : > "kafka-acls.sh" script communicates directly with zookeeper. > We should run kafka-acls.sh as ka

Re: Cannot create topics when zookeeper.set.acl=true

"kafka-acls.sh" script communicates directly with zookeeper. We should run kafka-acls.sh as kafka user (super user) to get write permission on zk. We should pass required jaas conf to the script. On Thu, Aug 23, 2018 at 3:02 PM HG wrote: > Hi, > > I have an environment with SSL, SASL and ACL's

Cannot create topics when zookeeper.set.acl=true

Hi, I have an environment with SSL, SASL and ACL's enabled. When I set zookeeper.set_acl=true in the server.properties file of the brokers I cannot create topics , ACL's etc. [root@host201 kafka]# bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal