Re: Kafka tuning - consultant work

2019-04-15 Thread Manoj Murumkar
Damian, Let me know when we can talk. I have done extensive work on Kafka and run a boutique consulting firm specializes in this work. Let me know when it's a good time to talk. I am located in San Francisco Bay area and can do a bit of travel, if needed. Thanks, Manoj (650) 417.5847 > On A

Enabling SASL_PLAINTEXT for Kafka REST

2017-10-18 Thread Manoj Murumkar
Folks, I am having trouble enabling SASL_PLAINTEXT protocol for Kafka REST component to work with secure cluster (that also uses same protocol). I am sure I am missing something trivial. If someone can help, I'd really appreciate it. Here're my configs: Startup script: cat /bin/kafka-rest-start

Re: Re: Using ACLs without Kerberos

2017-09-08 Thread Manoj Murumkar
This is fixed as well, btw. My ZK config wasn't done properly. On Tue, Sep 5, 2017 at 11:08 PM, Manoj Murumkar wrote: > Thanks, SASL setup for ZK does the trick. > > Now, I am continuing the security setup with schema registry. I am > configuring it to use same PLAINTE

Re: Re: Using ACLs without Kerberos

2017-09-05 Thread Manoj Murumkar
g/jira/browse/ZOOKEEPER-2125 once released ZK > could also be secured using SSL. > > Also remember not use any OS user, anyone user on n\w who can connect to > ZK host:port will be able to modify the ACLs. > > - Shri > > -Original Message- > From: Manoj Murumkar

Re: Using ACLs without Kerberos

2017-08-31 Thread Manoj Murumkar
gt; Any OS user who got access to zookeeper can create ACLs for any Kafka > principal. > This can be restricted by enabling Zk Kerberos authentication. > http://kafka.apache.org/documentation/#zk_authz > > From Kafka 0.11.0.0, We can create ACLs using Java Admin API. > > On Th

Re: Using ACLs without Kerberos

2017-08-31 Thread Manoj Murumkar
afka supports SASL > authentication framework. > KAFKA SASL supports GSSAPI (Kerberos), PLAIN or SCRAM mechanisms. you can > enable SSL encryption also > > http://kafka.apache.org/documentation.html#security > > > On Thu, Aug 31, 2017 at 7:28 PM, Manoj Murumkar > wrote:

Re: Using ACLs without Kerberos

2017-08-31 Thread Manoj Murumkar
and > it seems like it's possible. > How are you trying? Can you give more details about your setup? What > authentication mechanism, etc.? > > > Thanks, > > > On Thu, Aug 31, 2017 at 2:49 AM, Manoj Murumkar > wrote: > > > Hi, > > > > We

Using ACLs without Kerberos

2017-08-30 Thread Manoj Murumkar
Hi, We are evaluating how to put authorization in place for Kafka (around topics, mostly). Is it a good idea to do this without Kerberos? I was testing if a non-admin principal (OS user) can modify (add/remove) ACLs and it seems like it's possible. If this is right behavior, it's insecure and unus

Re: auto.offset.reset defaulting to earliest

2017-08-09 Thread Manoj Murumkar
. On Wed, Aug 9, 2017 at 9:25 AM, Manoj Murumkar wrote: > Hi, > > I am running standalone connector and explicitly setting auto.offset.reset > parameter to latest in the properties file (which is default value). > However, when I look at the logs, here's what I see: > &g

auto.offset.reset defaulting to earliest

2017-08-09 Thread Manoj Murumkar
Hi, I am running standalone connector and explicitly setting auto.offset.reset parameter to latest in the properties file (which is default value). However, when I look at the logs, here's what I see: 2017-08-09 15:34:24,731] INFO Instantiated task hdfs-sink-cdpos_cdhdr_view-0 with version 3.2.0

JdbcSourceConnector connector configuration is invalid

2017-06-26 Thread Manoj Murumkar
Hi, We are trying to pull some data from HANA into Kafka. When run in Stand-alone mode, the following configuration works as expected (I have SAP jar in right place). name=hana-jdbc connector.class=io.confluent.connect.jdbc.JdbcSourceConnector tasks.max=5 connection.url=jdbc:sap://:30015/?databas

Mirroring Schema registry using mirror maker

2017-06-20 Thread Manoj Murumkar
Hi, I am trying to mirror _schemas topic (essentially schema registry) using mirror maker, which is not working. It only replicates newly created schemas even though auto.offset.reset is set to earliest. Any ideas? Producer Config: bootstrap.servers=:9092 group.id=mirrormaker exclude.interna

Re: ORC plugin for Kafka HDFS connector

2017-03-23 Thread Manoj Murumkar
>> It wants to extract Avro schema from ORC record. Should say: It wants to extract connect schema from ORC record. On Thu, Mar 23, 2017 at 11:14 PM, Manoj Murumkar wrote: > Hi, > > I am developing a connector to support ORC data type in HDFS connector. > Everything is in pla

ORC plugin for Kafka HDFS connector

2017-03-23 Thread Manoj Murumkar
Hi, I am developing a connector to support ORC data type in HDFS connector. Everything is in place except for hive integration. Specifically, in the SchemaFileReader implementation. It wants to extract Avro schema from ORC record. However, I am unable to get record name from ORC record in order to