Kafka MirrorMaker error

2017-06-08 Thread karan alang
Hi All, I'm trying to transfer data between kafka clusters using Kafka MirrorMaker & running into issues. I've created a consumer.config & producer.config files & using the command shown below. The error indicates - equirement failed: Missing required property 'zookeeper.connect' --

Re: Kafka MirrorMaker error

2017-06-08 Thread karan alang
Thanks, that fixed the issue ! On Thu, Jun 8, 2017 at 11:58 AM, Mostafa Zarifyar wrote: > maybe this will help. > > https://community.hortonworks.com/articles/24599/kafka-mirrormaker.html > > -M > > On Thu, Jun 8, 2017 at 11:42 AM, karan alang > wrote: > > &

Re-Balancing Kafka topics - Best practices

2017-06-13 Thread karan alang
Hi All, Fpr Re-balancing Kafka partitions, we can set property -> *auto.leader.rebalance.enable = true in server.properties file.* *Is that the recommended way or is it better to reBalance the kafka partitions manually ?(using *scripts - *kafka-preferred-replica-election.sh, * *kafka-reassign-

Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

2017-06-13 Thread karan alang
Hello All, I've streaming data flowing into Kafka topics. What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ? for eg. Can I add Broker & re-balance the topics while the data flows into Kafka topic ? Pls. advise. Thanks!

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

2017-06-13 Thread karan alang
, > > On Tue, 13 Jun 2017 at 10:58 pm, karan alang > wrote: > > > Hello All, > > > > I've streaming data flowing into Kafka topics. > > What are the best practices wrt. adding or removing Kafka Brokers while > > data is flowing into Kafka ? > > >

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

2017-06-13 Thread karan alang
ducer is pushing data into the Kafka Topic ? What are the implications of this ? Pls. advise. On Tue, Jun 13, 2017 at 4:29 PM, karan alang wrote: > > Hello, clarifying the question -> > > Within the same Kafka cluster, if i want to add Broker & rebalance topic > (say t

kafka-consumer-groups.sh vs ConsumerOffsetChecker

2017-06-15 Thread karan alang
Hi All - I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying to use the ConsumerOffsetChecker & bin/kafka-consumer-groups.sh to check for offsets. I'm seeing different behavior. Here is what i did -> a) When i use ConsumerOffsetChecker $KAFKA_HOME/bin/kafka-run-class.sh kaf

Re: Re-Balancing Kafka topics - Best practices

2017-06-15 Thread karan alang
Thanks Todd.. for the detailed reply. regds, Karan Alang On Tue, Jun 13, 2017 at 3:19 PM, Todd Palino wrote: > A few things here… > > 1) auto.leader.rebalance.enable can have serious performance impacts on > larger clusters. It’s currently in need of some development work to en

Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker

2017-06-15 Thread karan alang
CURRENT OFFSET, LOG END OFFSET, LAG, OWNER any ideas on what needs to be done? On Thu, Jun 15, 2017 at 11:55 AM, karan alang wrote: > Hi All - > I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying to > use the ConsumerOffsetChecker & bin/kafka-consumer-gr

Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker

2017-06-15 Thread karan alang
he topic 'newBroker1' exists under this ZK path? > Also, do you still get the rows below if you run the ConsumerOffsetChecker > without providing any topic? > > Thanks. > --Vahid > > > > > From: karan alang > To: users@kafka.apache.org > Date: 06/

Kafka-Spark Integration - build failing with sbt

2017-06-16 Thread karan alang
I'm trying to compile kafka & Spark Streaming integration code i.e. reading from Kafka using Spark Streaming, and the sbt build is failing with error - [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found Scala version

Re: Kafka-Spark Integration - build failing with sbt

2017-06-17 Thread karan alang
Thanks, i was able to get this working. here is what i added in build.sbt file -- scalaVersion := "2.11.7" val sparkVers = "2.1.0" // Base Spark-provided dependencies libraryDependencies ++= Seq( "org

Re: Kafka-Spark Integration - build failing with sbt

2017-06-17 Thread karan alang
Hey Jozef, Thanks for the quick response .. yes, you are right .. spark-sql dependency was missing .. added that & it worked fine. regds, Karan Alang On Sat, Jun 17, 2017 at 2:24 PM, Jozef.koval wrote: > Hey Karan, > I believe you are missing spark-sql dependency. > > Joz

Re: Kafka-Spark Integration - build failing with sbt

2017-06-19 Thread karan alang
through fine So, I assume additional libraries are being downloaded when i provide the appropriate packages in LibraryDependencies ? which ones would have helped compile this ? On Sat, Jun 17, 2017 at 2:52 PM, karan alang wrote: > Hey Jozef, > > Thanks for the quick response .. >

Kafka MirrorMaker - errors/warning

2017-06-20 Thread karan alang
Hi All - i've setup Kafka MirrorMaker using link - Kafka Mirror Maker best Practices , and am getting the following warnings -> [2017-06-20 12:29:31,381] WARN The configuration queue.time = 100 was suppl

New Kafka Producer or the Old One ???

2017-06-21 Thread karan alang
Hello All - I've *Kafka 0.9* & I'm running this command to publish records to Kafka topics - $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1 --max-messages 500 --broker-list localhost:9092,localhost:9093,localhost:9094,localhost:9095 --producer.config $KAFKA_HOME/config/producer.pro

Re: Kafka MirrorMaker - errors/warning

2017-06-21 Thread karan alang
owever, couple are still pending -> a) producer.type = async was supplied but isn't a known config *How do i ensure the producer is an "async" producer ?* b) queue.time = 100 was supplied but isn't a known config *What needs to be done for this ?* On Tue, Jun 20, 2017 at

Kafka MirrorMaker - not replicating messages after being brought up

2017-06-21 Thread karan alang
Hi All, I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker functionality. Here is what i did : 1) I have identical topics Topic1 on 2 Kafka clusters - Cluster1 & Cluster2 2) On Cluster1, I publish 100 messages on Topic1 3) I've 2 consumers reading messages from the 2 topics o

Re: Kafka MirrorMaker - not replicating messages after being brought up

2017-06-21 Thread karan alang
onfigs > > On Thu, Jun 22, 2017 at 2:00 AM, karan alang > wrote: > > > Hi All, > > > > I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker > > functionality. > > > > Here is what i did : > > > > 1) I hav

Deleting/Purging data from Kafka topics (Kafka 0.10)

2017-06-22 Thread karan alang
Hi All - How do i go about deleting data from Kafka Topics ? I've Kafka 0.10 installed. I tried setting the parameter of the topic as shown below -> $KAFKA10_HOME/bin/kafka-topics.sh --zookeeper localhost:2161 --alter --topic mmtopic6 --config retention.ms=1000 I was expecting to have the data p

Kafka 0.10 - kafka console consumer not reading the data in order that it was published

2017-06-22 Thread karan alang
Hi All - version - kafka 0.10 I'm publishing data into Kafka topic using command line, and reading the data using kafka console consumer *Publish command ->* $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1 --max-messages 100 --broker-list localhost:9092,localhost:9093,localhost:909

Re: Kafka 0.10 - kafka console consumer not reading the data in order that it was published

2017-06-22 Thread karan alang
Hi Subhash, number of partitions - 3 On Thu, Jun 22, 2017 at 12:37 PM, Subhash Sriram wrote: > How many partitions are in your topic? > > On Thu, Jun 22, 2017 at 3:33 PM, karan alang > wrote: > > > Hi All - > > > > version - kafka 0.10 > > I'm pu

Re: Kafka 0.10 - kafka console consumer not reading the data in order that it was published

2017-06-22 Thread karan alang
h Sriram > Sent: Thursday, 22 June, 21:37 > Subject: Re: Kafka 0.10 - kafka console consumer not reading the data in > order that it was published > To: users@kafka.apache.org > > > How many partitions are in your topic? On Thu, Jun 22, 2017 at 3:33 PM, > karan alang wrote: &g

Re: Kafka 0.10 - kafka console consumer not reading the data in order that it was published

2017-06-22 Thread karan alang
r. > > I hope that helps. > > Thanks, > Subhash > > On Thu, Jun 22, 2017 at 4:37 PM, karan alang > wrote: > > > Hi Subhash, > > > > number of partitions - 3 > > > > On Thu, Jun 22, 2017 at 12:37 PM, Subhash Sriram < > subhash.sri...@gmail.com

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

2017-06-22 Thread karan alang
ike in your example if this broker config > value is much higher, then the broker doesn't delete old logs regular > enough. > > --Vahid > > > > From: karan alang > To: users@kafka.apache.org > Date: 06/22/2017 12:27 PM > Subject:Deleting/Purging da

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

2017-06-22 Thread karan alang
te: 06/22/2017 06:43 PM > Subject:Re: Deleting/Purging data from Kafka topics (Kafka 0.10) > > > > Hi Karan, > > I think the issue is in verification step. Because the start and end > offsets are not going to be reset when messages are deleted. > H

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

2017-06-23 Thread karan alang
ok, thanks ... i'll check that and update in case i see issues. regds, Karan Alang On Fri, Jun 23, 2017 at 12:01 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Karan, > > I think what you are seeing with `--time -1` and '--time -2` confirms that > t

Kafka logs - server.log & controller.log

2017-06-28 Thread karan alang
Hi All - in Kafka, i see two different log files - server.log & controller.log files. What specifically gets logged in these log files ? i.e. what are these 2 files meant to log ?

updating startup script in Apache Amabari

2017-07-06 Thread karan alang
hi All i'm trying to configure/update Kafka startup script, so Prometheus is able to read metrics from Kafka. I need to add the following to the Broker startup script : -javaagent:/path/to/jmx_exporter.jar=$PORT:$CONFIG Where in Apache Ambari can i modify the Kafka Broker startup script ? Pls

Error in Kafka startup with the JMX exporter - java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds

2017-07-06 Thread karan alang
Hi All, I'm getting error in starting up Kafka, with the JMX exporter passed as java agent. Error is as shown below, any ideas ? I'm trying to integrate Prometheus to collect Kafka metrics. [root@nwk2-bdp-kafka-06 kafka]# KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/hdp/2.5.3.0-37/prometheus/jmx_prom

Re: Kafka logs - server.log & controller.log

2017-07-07 Thread karan alang
sorry, missed this email Thanks for the clarification. regds, Karan Alang On Thu, Jun 29, 2017 at 6:52 AM, Tauzell, Dave wrote: > All the brokers write to server.log. The broker that happens to be the > controller will also write to the controller.log file. > > -Dave >

Re: Kafka-Spark Integration - build failing with sbt

2017-07-10 Thread karan alang
he code in Eclipse, i was not able to do that > > eg. > import org.apache.spark.streaming.kafka.KafkaUtils > > gave errors saying KafaUtils was not part of the package. > However, when i used sbt to compile - the compilation went through fine > > So, I assume additional librari

Re: Kafka-Spark Integration - build failing with sbt

2017-07-12 Thread karan alang
ation, that fixed the issue regds, Karan Alang On Wed, Jul 12, 2017 at 1:01 PM, Jozef.koval wrote: > Hi Karan, > Why do you try to submit class which does not exist? Don't you want to > create it first? > Jozef > > Sent from [ProtonMail](https://protonmail.ch), encrypted

Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

2017-07-24 Thread karan alang
Hello - i've enabled SSL for Kafka, and Kafka is starting up fine with SSL enable. However, when i run the Kafka console producer, it is give me error as shown below -> 1. Command : 2. 3. /usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list nwk2-bdp-kafka-05.gdcs-qa

Re: Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

2017-07-24 Thread karan alang
.RequestSendThread.brokerReady(ControllerChannelManager.scala:233) > at > kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182) > at > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181) > at kafka.utils.ShutdownableThread.run(Shutdownabl

Re: Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

2017-07-24 Thread karan alang
017296247CC84BB1F7D7842BA9B1 Session-ID-ctx: Master-Key: 795C06945CBD2BABC55A269FF46EAE6848E3834E5EAB54886E10DFD5289498901A5169AFE268872F4B0A3439DA20A378 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1500936569 Timeout : 7200 (sec) Verify return code: 18 (self signed certificate) On Mon, Jul 24, 2017 at 2:08 PM, kar

Kafka with SSL enabled - Not able to publish messages (org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms)

2017-07-25 Thread karan alang
hi - I've enabled SSL for Kafka & i'm trying to publish messages using console Producer Error is as shown below, any ideas ? > >1. /usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list >nwk2-bdp-kafka-05.gdcs-qa.apple.com:6668,nwk2-bdp-kafka-04.gdcs-qa. >apple.com:6668

Re: Kafka with SSL enabled - Not able to publish messages (org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms)

2017-07-25 Thread karan alang
Brokers (1001, 1002, 1003) The question is why & what needs to be done to fix this ? On Tue, Jul 25, 2017 at 10:31 PM, Manikumar wrote: > enable debug logs to find out the actual error. > > On Wed, Jul 26, 2017 at 12:49 AM, karan alang > wrote: > > > hi - I'v

Confluent Control Center UI - not coming up

2017-07-26 Thread karan alang
hi All, I've installed Confluent Kafka 3.2.2 & am using the following link to start the services - http://docs.confluent.io/current/control-center/docs/quickstart.html I've started the services shown below, however the Contraol Center UI is not coming up. Any ideas on this ? screen -ls > There

Re: Confluent Control Center UI - not coming up

2017-07-26 Thread karan alang
Hello all- i was able to fix this by modifying the following entry confluent.controlcenter.rest.listeners=http://0.0.0.0:9021 -> confluent.controlcenter.rest.listeners=http://:9021 <http://0.0.0.0:9021/> On Wed, Jul 26, 2017 at 3:35 PM, karan alang wrote: > hi All, > &

Confluent 3.2.2 - Error in starting broker

2017-07-27 Thread karan alang
Hi All - i've installed Confluent 3.2.2 & i'm getting error in starting up the Broker Attached the server.properties file, any ideas on what the issue might be ? ./bin/kafka-server-start ./etc/kafka/server.properties > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in

Re: Confluent 3.2.2 - Error in starting broker

2017-07-27 Thread karan alang
HI All - this is fixed, this was due to class path issue. On Thu, Jul 27, 2017 at 3:30 PM, karan alang wrote: > Hi All - i've installed Confluent 3.2.2 & i'm getting error in starting up > the Broker > > Attached the server.properties file, any ideas on what the issue

Confluent Kafka 3.2.2 - rebalancing not happenning

2017-08-01 Thread karan alang
Hi All - i'm trying to rebalance Kafka topic (refer link -> http://docs.confluent.io/current/kafka/rebalancer/rebalancer.html), and somehow the rebalancing is not working. Here is what i'm doing -> - i've 4 Kafka brokers & i've made changes to the server.properties file to enable Confluent Metric

Re: Confluent Kafka 3.2.2 - rebalancing not happenning

2017-08-01 Thread karan alang
are even across brokers and racks on a per topic and cluster > level while minimising data movement. seems there is a disconnect here, pls let me know if anyone has inputs. On Tue, Aug 1, 2017 at 4:45 PM, karan alang wrote: > Hi All - > i'm trying to rebalance Kafka topi

Kafka SSL - console-consumer on node5 not receiving messages from producer on node4

2017-09-05 Thread karan alang
Hi All - I've Kafka 0.10 .. I've enabled SSL(Non-kerberized) for Kafka Broker on Node 4, and *i'm able to produce/consume messages using console-producer & console-consumer from Node 4.* However, i'm having issues enabling ssl connection between Node 4 & Node 5 & try to consume messages from Node

Kafka Streams - not able to see the text in console consumer

2017-09-17 Thread karan alang
Hello All - i've a basic word count Kafka streams code, which reads data from the input topic, splits the data (per the separator) and outputs the data into the output topic. I've a console producer, which is putting data into the input topic (kstreams3), and a console consumer which is reading th

Re: Kafka Streams - not able to see the text in console consumer

2017-09-18 Thread karan alang
.kafka.common.serialization.LongDeserializer > > > Vito > > > > > > -- > 鄭紹志 Vito Jeng > 亦思科技股份有限公司 研究發展處 > TEL: 03-5630345 Ext.16 > > On Mon, Sep 18, 2017 at 10:58 AM, karan alang > wrote: > > > Hello All > > - i've a basic word count

Kafka SSL error

2017-09-19 Thread karan alang
Hello All - I was able to set up SSL for the Kafka brokers, using OpenSSL. however, I'm having issues with setting up SSL using the pem file (i.e. SSL certificate - certified by CA, provided by the company) Here is what i've done - created the server/client keystore & truststore files and importe

Re: Kafka SSL error

2017-09-20 Thread karan alang
2 key and that can > be converted into keystore. > > BTW: If you run your application with system property “javax.net.debug” set > to “ssl” it will generate a lot of useful debug information which will help > to understand what is going on and fix this. > > Jakub > > On T

Re: Kafka SSL error

2017-09-20 Thread karan alang
at the broker basically > uses the self-signed cert you generated. This is probably the default > subject distinguished name of the self signed certificate. And since it is > self-signed openssl cannot verify its identity (unless you pass it the > public key of this exact self signed

Re: Kafka SSL error

2017-09-20 Thread karan alang
uld be in following order -> 1) Root 2) Intermediate 3) Server On Wed, Sep 20, 2017 at 4:48 PM, karan alang wrote: > Hi Jakub, > > Thanks for the detailed note... > > here is the update -> > > I was able to convert the host.cert.pem to PKCS#12 & import the cert into

Re: Kafka SSL error

2017-09-25 Thread karan alang
the chain parts separately into the keystore will not help. > > In the openssl s_client you can also use the -CAfile option to point it to > the intermediate certificate - that would also wirk but using the whole > chain is probably better. > > Regards > Jakub > > On Thu,

left join between PageViews(KStream) & UserProfile (KTable)

2017-10-22 Thread karan alang
Hello all - I'm trying to run the sample PageView Kafka streams example, (ref - https://github.com/apache/kafka/tree/0.11.0/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview ) and have a question .. There is a leftJoin between PageView (Kstream) and UserProfile(Ktable) as s

Re: left join between PageViews(KStream) & UserProfile (KTable)

2017-10-24 Thread karan alang
ctions for how to > run it. > > It would be great if we had the same setup for the Apache Kafka page view > example, of course. Pull requests are very welcome. :-) > > Hope this helps, > Michael > > > > > > > On Sun, Oct 22, 2017 at 9:30 PM, karan al

Kafka producer/consumer quotas (Kafka 0.10)

2017-10-27 Thread karan alang
I've Kafka 0.10 .. I've set the following in kafka broker.properties & restarted the brokers - quota.producer.default=10485760 quota.consumer.default=10485760 when i fire the following command, i was expecting to see the quota limits : /bin/kafka-configs --zookeeper localhost:3181 --describe -

what drives deletion of kafka topics

2018-01-30 Thread karan alang
I've Kafka 10. I've a basic question - what determines when the Kafka topic marked for deletion gets deleted ? Today, i marked a topic for deletion, and it got deleted immediately (possibly because the topic was not being used for last few months ?) .. In earlier instances, i'd to wait for some ti

multiple tenants accessing Confluent kafka on GCP

2021-10-27 Thread karan alang
Hello All - For multiple external tenants accessing Kafka, is there any issue with providing the same endpoint (ie. set of brokers, ports) to the multiple producers ? what are the best practices wrt multiple tenants producing data on (Confluent) Kafka topics, Kafka being installed on GCP. tia!

GCP Dataproc - Failed to construct kafka consumer, Failed to load SSL keystore dataproc-versa-sase-p12-1.jks of type JKS

2022-02-02 Thread karan alang
Hello All, I'm trying to run a Structured Streaming program on GCP Dataproc, which accesses the data from Kafka and prints it. Access to Kafka is using SSL, and the truststore and keystore files are stored in buckets. I'm using Google Storage API to access the bucket, and store the file in the cu

Re: GCP Dataproc - Failed to construct kafka consumer, Failed to load SSL keystore dataproc-versa-sase-p12-1.jks of type JKS

2022-02-02 Thread karan alang
re-checking to see if there is any suggestion on this issue. On Wed, Feb 2, 2022 at 3:36 PM karan alang wrote: > Hello All, > > I'm trying to run a Structured Streaming program on GCP Dataproc, which > accesses the data from Kafka and prints it. > > Access to Kafk

GCP Dataproc - getting error in importing KafkaProducer

2022-02-17 Thread karan alang
Hello All, I've a GCP Dataproc cluster, and i'm running a Spark StructuredStreaming job on this. I'm trying to use KafkaProducer to push aggregated data into a Kafka topic, however when i import KafkaProducer (from kafka import KafkaProducer), it gives error ``` Traceback (most recent call last

Confluent Schema Registry - auto.register.schemas

2022-11-11 Thread karan alang
Hello All, ref stackoverflow : https://stackoverflow.com/questions/74396652/confluent-schema-registry-why-is-auto-register-schemas-not-applicable-at-schem --- I've installed the Schema Registry and want to set the auto.register.schema set to false - so only avro messages conforming to the regist

Re: Confluent Schema Registry - auto.register.schemas

2022-11-14 Thread karan alang
hello All - checking to see if anyone has updates on this. tia! On Fri, Nov 11, 2022 at 1:55 PM karan alang wrote: > Hello All, > > ref stackoverflow : > https://stackoverflow.com/questions/74396652/confluent-schema-registry-why-is-auto-register-schemas-not-applicable-at-schem >

Kafka on GKE - automating distribution of TLS certs to external Kafka clients on certs expiry

2023-01-05 Thread karan alang
Hello All, We have a Strimzi kafka cluster on GKE with TLS implemented, the TLS certs have a default expiry of 1 year i.e. the certs get renewed every year Currently, on expiry - we have to provide the external Kafka clients with new certs, the objective is to automate this process i.e. One shoul

MirrorMaker2 on GKE

2023-02-23 Thread karan alang
Hello All, Anyone has installed KafkaMirrorMaker2 on GKE ? Need some help to debug/resolve issues that I've been having. I've 2 clusters on GKE with strimzi kafka installed. cluster1(nossl-e) on use-east1, cluster2(nossl-e) on us-west1. Both the clusters are Auto-pilot cluster (mentioning this,

Re: MirrorMaker2 on GKE

2023-02-23 Thread karan alang
n Thu, Feb 23, 2023 at 2:12 PM karan alang wrote: > Hello All, > > Anyone has installed KafkaMirrorMaker2 on GKE ? > Need some help to debug/resolve issues that I've been having. > > I've 2 clusters on GKE with strimzi kafka installed. > cluster1(nossl-e) on use-east1

Re: MirrorMaker2 on GKE

2023-02-23 Thread karan alang
data from src topic to target topic On Thu, Feb 23, 2023 at 3:04 PM karan alang wrote: > Logs with the Disconnect Exception (when i set the loglevel to - INFO) : > > > ``` > > [SourceTaskOffsetCommitter-1] > > 2023-02-23 22:54:13,546 INFO [Consumer clientId=consumer-null-

Strimzi kafka on k8s - only bootstrap server has public IP, brokers have Cluster IP

2024-11-03 Thread karan alang
Hello All - Anyone has installed Kafka on k8s, with only bootstrap server having Public IP, while the brokers have Cluster IP. Objective being - to reduce the number of public IPs, this reducing cost & also making the cluster secure However, what would the implication of this be on performance si

Schema Registry - serving multiple Kafka clusters

2025-01-07 Thread karan alang
Hello All, Can we have multiple Kafka clusters using a single Schema Registry (either Karapace ? If anyone has set up a similar system, pls let me know. Thanks!

Re: Schema Registry - serving multiple Kafka clusters

2025-01-09 Thread karan alang
ce we don't use Confluent cloud) .. If there are other specific limitations in using Strimzi Schema Registry, I'll need to check if I need to integrate Karapace Schema Registry instead. Pls let me know your thoughts. thanks, Karan Alang On Tue, Jan 7, 2025 at 5:40 PM Shahar Cizer K

Re: Schema Registry options for Strimzi Kafka

2025-04-10 Thread karan alang
ed OR I need to be aware of. Also, from what I understand - this has an Apache 2.0 license, so we use this without restrictions in commercial products as well. regds, Karan Alang On Fri, Mar 21, 2025 at 12:49 AM Ömer Şiar Baysal wrote: > Hi, > > Like Strimzi, Red Hat folks has its s

Apicurio Schema Registry 3.0.7 - producer failing in Confluent compatibility mode

2025-05-04 Thread karan alang
I have Apicurio Schema Registry 3.0.7 installed on GKE. I'm trying to use this in Confluent Compatibility mode .. primarily because on production, my Kafka Producer is C++ client, and Apicurio serde is not supported for C++ client. Here is the Java code: ``` import com.versa.apicurio.confluent.E

Re: Schema Registry options for Strimzi Kafka

2025-04-24 Thread karan alang
Hi All - anyone has any info on this ? Also, I've created a discussion post on the github .. but haven't received any response from the apicurio community yet. Is there a better way to reach out to the apicurio community ? tia! On Mon, Apr 21, 2025 at 12:28 PM karan alang wrote:

Re: Schema Registry options for Strimzi Kafka

2025-04-24 Thread karan alang
mpared to version 2, > you can also compare what has been done via milestones on GitHub, the only > thing I remember about version 3 is they dropped support for Confluent API > version 6. > > Kind regards. > OSB > > On Thu, Apr 24, 2025, 20:43 karan alang wrote: > >

Apicurio Schema Registry 3.0.7 - unable to update multiple versions of schema for artifactId

2025-05-02 Thread karan alang
I have Apicurio Schema Registry 3.0.7 installed, and I'm trying to use the following code to publish a version 2 of schema where GROUP = default, ARTIFACT_ID = com.versa.apicurio.confluent.Employee here is the code import requests import json import urllib.parse # ───

Re: Schema Registry options for Strimzi Kafka

2025-04-21 Thread karan alang
hema Registry version to 3.x when using the Operator ? Any feedback on this ? tia! On Thu, Apr 10, 2025 at 4:08 PM karan alang wrote: > Thanks Omer .. I was checking the options and it seems this is one of the > most popular Open Source options for Schema Registry. > I'm planning o

Schema Registry options for Strimzi Kafka

2025-04-04 Thread karan alang
Hello - We have Strimzi Kafka installed on kubernetes (on GCP, as well as onPrem). I'm looking for feedback on the Schema Registry options - 1. Confluent Schema Registry 2. Karapace 3. any others There is a strimzi registry operator as well, but that has not been updated in the last 3 years htt

Re: Apicurio Schema Registry 3.0.7 - producer failing in Confluent compatibility mode

2025-05-06 Thread karan alang
well, I was hoping someone in the group would have used apicurio schema registry and would be able to help out (esp since it is used with Kafka) Responses to issues created on apicurio github are very slow, if at all. thanks, Karan Alang On Sun, May 4, 2025 at 8:04 PM Luke Chen wrote: >