Writing a customized principal builder for authorization using ACLs

Hi, Can anyone help me or point me to any resources that can be of help for writing a customized principal builder to use in Authorization using ACLs? I've enabled SSL authentication scheme for both clients and brokers but I would like to change the principal name to just the original name and Orga

Re: OOM errors

My apologies. In fact the 'aggregate' step includes this: 'TimeWindows.of(20 * 60 * 1000L).advanceBy(60 * 1000L)' On Tue, Nov 29, 2016 at 9:12 PM, Guozhang Wang wrote: > Where does the "20 minutes" come from? I thought the "aggregate" operator > in your > > stream->aggregate->filter->foreach > >

Message order different each time stream is replayed?

While I was connected to console-consumer.sh, I posted a few messages to a Kafka topic, one message at a time, across a few hours. I'd post a message, see it arrive in console-consumer, a few mins later I'd post the next message, and so on. They all arrived in order. However, when I now try to v

Re: Writing a customized principal builder for authorization using ACLs

i would suggest understanding SunJSSE security provider(s) implemented by Oracle https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#SunJSSE i would also suggest understanding complimentary security providers from BouncyCastle http://www.bouncycastle.org/wi

Re: Initializing StateStores takes *really* long for large datasets

I'd like to quickly reinforce Frank's opinion regarding the rocksdb memory usage. I was also surprised by the amount of non-JVM-heap memory being used and had to tune the 100 MB default down considerably. It's also unfortunate that it's hard to estimate the memory requirements for a KS app becaus

Release Kafka 0.9.0.2?

Hello Apache Kafka community, Would it be possible to release 0.9.0.2? It has few important fixes, like KAFKA-3594 and would be helpful for us that cannot upgrade to 0.10.x yet. Kind regards, Stevo Slavic.

Re: Initializing StateStores takes *really* long for large datasets

Mathieu, You are absolutely right. We've written about the memory management strategy below: https://cwiki.apache.org/confluence/display/KAFKA/Discussion%3A+Memory+Management+in+Kafka+Streams

Re: Convert log file in Avro json encoding to Avro binary encoding and send into Kafka

Hi Ewen, Thank you for your advise. I reached a following compromise: My application sends records in the Avro Binary encoding to Kafka directly using Confluent's serializers(io.confluent/kafka-avro-serializer), which integrated with schema registry, and also does logging in the Avro JSON encoding

Re: Messages intermittently get lost

Hi Martin, makes sense. When I SSH into all 3 of my ZK nodes and run: sh zkServer.sh status All three of them give me the following output: JMX enabled by default zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi") Looks like a bug in the ZK shell

Writing a customized principal builder for authorization

Hi, Can anyone help me or point me to any resources that can be of help for writing a customized principal builder to use in Authorization using ACLs? I've enabled SSL authentication scheme for both clients and brokers but I would like to change the principal name to just the original name and Orga

Re: question about property advertised.listeners

The advertised.listeners property would not help you to map a different port. If you need to talk to brokers at a different port you should also set the listeners property accordingly. Say if you want the broker to listen at port 10092 as an example, the server.properties needs to contain these 2

Re: log.dirs balance?

Thanks for the report, Karolis. I have a potential theory for how this happened and I'm wondering if it's possibly valid: I have 9 partitions on a machine with 3 disks and they get assigned exactly as you'd expect: d1: t{1,2,3} d2: t{4,5,6} d3: t{7,8,9} Then, a disk fails or something somewhere

Re: Writing a customized principal builder for authorization

Hi Kriti, You will have to implement the Principal Builder interface and provide the full class path in broker config. I don't remember the exact config name right now, but you can search for some config by name "principalbuilder.class" in the broker configs. Once you do this, Kafka will auto

Re: Writing a customized principal builder for authorization

"principal.builder.class" is the name of the property. Thanks, Mayuresh On Wed, Nov 30, 2016 at 9:30 AM, wrote: > Hi Kriti, > > You will have to implement the Principal Builder interface and provide the > full class path in broker config. I don't remember the exact config name > right now, but

Re: Writing a customized principal builder for authorization

How do you then modify Kafka's searchable classpath to pick up this new principal.builder.class classfile from a JAR somewhere on the filesystem? In other words, I change my server.properties to: principal.builder.class=com.example.kafkautils.MyCustomKafkaPrincipalBuilder How will Kafka be a

KStreams app in YARN

Hello, I was wondering if anyone has deployed a KStreams app to YARN, perhaps using Slider? If so, what was your experience? Thanks! -Aaron

Re: Messages intermittently get lost

a shock when a zk script goes fubar zookeeper devs can we get some help for sh zkServer.sh status? Thanks! Martin __ From: Zac Harvey Sent: Wednesday, November 30, 2016 10:25 AM To: users@kafka.apache.org Subject:

Re: Initializing StateStores takes *really* long for large datasets

+1 on this. Ara. > On Nov 30, 2016, at 5:18 AM, Mathieu Fenniak > wrote: > > I'd like to quickly reinforce Frank's opinion regarding the rocksdb memory > usage. I was also surprised by the amount of non-JVM-heap memory being > used and had to tune the 100 MB default down considerably. It's al

Save the date: ApacheCon Miami, May 15-19, 2017

Dear Apache enthusiast, ApacheCon and Apache Big Data will be held at the Intercontinental in Miami, Florida, May 16-18, 2017. Submit your talks, and register, at http://apachecon.com/ Talks aimed at the Big Data section of the event should go to http://events.linuxfoundation.org/events/apache-bi

mirror and schema topics

Hello! I'm trying to mirror a kafka cluster, then run connect on the mirror. It seems the schemas are not getting moved in the mirror though, so I get the following error. Is this a configuration problem? Thank you for the help! Mirror>curl -X GET http://localhost:8081/subjects [] Machine1>cu

FW: mirror and schema topics

Hello! I'm trying to mirror a kafka cluster, then run connect on the mirror. It seems the schemas are not getting moved in the mirror though, so I get the following error. Is this a configuration problem? Thank you for the help! Mirror>curl -X GET http://localhost:8081/subjects [] Machine1>c

Re: Writing a customized principal builder for authorization

this is a quick and dirty test you can use: org.apache.kafka.common.network.SSLSelectorTest: //Truststore needs to contain keystore/cert that contains the actual principal you will use File trustStoreFile = File.createTempFile("truststore", ".jks"); Map sslServerConfigs = org.

Re: Messages intermittently get lost

Hey Martin, Do you want to create a jira for this and report the issue? -Flavio > On 30 Nov 2016, at 18:33, Martin Gainty wrote: > > a shock when a zk script goes fubar > > > zookeeper devs can we get some help for sh zkServer.sh status? > > > Thanks! > > Martin >

Re: Message order different each time stream is replayed?

How many partitions do you have in that topic. Kafka only guarantees a total ordering of messages within a partition, not across partitions of a topic. If you want total ordering over the entire topic, you need to create a topic with a single partition. On Wed, Nov 30, 2016 at 4:10 AM, Ali Akhtar

Re: Message order different each time stream is replayed?

Ah, That explains it. I have 3 partitions. On 1 Dec 2016 5:42 a.m., "Apurva Mehta" wrote: How many partitions do you have in that topic. Kafka only guarantees a total ordering of messages within a partition, not across partitions of a topic. If you want total ordering over the entire topic, you

How to get consumer creating time and last consuming time

Hi all, In kafka0.8, I can get these infomations from zk.But how to get them in kafka0.10? Thanks Yuanjia Li