few kafka topics got deleted post upgrade

Dear Kafka Community, We are upgrading our kafka instance from 2.8.1 version to 3.5.2. The setup we have was of 3 kafka broker nodes and 3 zookeeper nodes. We performed a rolling upgrade of the broker with the 3.5.2 version and observed that some topics got deleted post upgrade. we don't see an

Delete Kafka topics replicated by mirror maker 2

Hi Team, I have configured two Kafka clusters with 2-way replication using MirrorMaker2 (mm2). I want to delete a topic from source cluster and its replica from destination cluster. When I delete a topic in the source cluster and then delete the replica in the destination cluster, the desti

Re: Kafka Topics

contact the sender and > > delete this email and its contents from any computer. Any views expressed > > in this email are those of the individual sender and may not necessarily > > reflect the views of the company. > > > >Please consider the environmet b

Re: Kafka Topics

sed > in this email are those of the individual sender and may not necessarily > reflect the views of the company. > >Please consider the environmet before printing this email. > > > -Original Message- > From: Wes Peng > Sent: Thursday, December

RE: Kafka Topics

flect the views of the company. Please consider the environmet before printing this email. -Original Message- From: Wes Peng Sent: Thursday, December 23, 2021 10:11 PM To: users@kafka.apache.org Subject: Re: Kafka Topics That depends on your resources

Re: Kafka Topics

That depends on your resources such as ram, disk etc. General speaking there is no problem. Regards > Dears, > > I'm new to using Kafka, and I was wondering up to how many topics can > Kafka Handle. I'm trying to use Kafka but using it I will be obliged to > create thousands of topics to keep up

Kafka Topics

Dears, I'm new to using Kafka, and I was wondering up to how many topics can Kafka Handle. I'm trying to use Kafka but using it I will be obliged to create thousands of topics to keep up with my data. Will Kafka on my server handle this kind of data? Thank you, Best Regards Ola Bissani De

[Spark SQL] - Not able to consume Kafka topics

Hello, Issues : * I and my team are trying to consume some kafka topics based on the timestamps using startingOffsetsByTimestamps option, and the code works fine when we run via a Databricks notebook. * There is a need to setup the whole process in a local system (IntelliJ), but the

[Spark SQL] - Not able to consume Kafka topics

Hello, Issues : * I and my team are trying to consume some kafka topics based on the timestamps using startingOffsetsByTimestamps option, and the code works fine when we run via a Databricks notebook. * There is a need to setup the whole process in a local system (IntelliJ), but the

Re: Unable to list Kafka topics

il chaudhari > > > > wrote: > > > > > Hi, > > > > > Please check if it helps: > > > > > > > > > > > http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_supportedPlatforms > > > > > > > >

Re: Unable to list Kafka topics

tforms > > > > > > > > You have similar symptoms in point 6 in your original email. > > > > > > > > Try running it manually with below steps: > > > > Clear all logs > > > > Start zookeeper manually, watch logs for any erro

Re: Unable to list Kafka topics

> > > > > Try running it manually with below steps: > > > Clear all logs > > > Start zookeeper manually, watch logs for any error. Rectify it referring > > > above link. > > > Start kafka manually. Watch info/error logs. > > > Then try to list topics. >

Re: Unable to list Kafka topics

ng > > above link. > > Start kafka manually. Watch info/error logs. > > Then try to list topics. > > > > Regards, > > Sunil. > > > > > > On Sat, 21 Nov 2020 at 10:16 PM, prat 007 wrote: > > > > > Hi Team, > > > &g

Re: Unable to list Kafka topics

server. (on Dev env, standalone > > setup) but my playbook is getting failed at kafka role for task listing of > > Kafka topics. Can you please help. > > below are the logs. &g

Re: Unable to list Kafka topics

led at kafka role for task listing of > Kafka topics. Can you please help. > below are the logs. > > 1. > playbook getting failed with error: - > ``` > TASK [k

Re: Unable to list Kafka topics

Sat, Nov 21, 2020, 22:16 prat 007 wrote: > > > > > > > Hi Team, > > > > > > > > I am running ansible to install elk on one server. (on Dev env, > > standalone > > > > setup) but my playbook is getting failed at kafka role for task >

Re: Unable to list Kafka topics

gt; > > > > > I am running ansible to install elk on one server. (on Dev env, > standalone > > > setup) but my playbook is getting failed at kafka role for task > listing of > > > Kafka topics. Can you please help. > > &g

Re: Unable to list Kafka topics

7 wrote: > > > Hi Team, > > > > I am running ansible to install elk on one server. (on Dev env, standalone > > setup) but my playbook is getting failed at kafka role for task listing of > > Kafka topics. Can you please help. > > below are the logs. > > >

Re: Unable to list Kafka topics

t kafka role for task listing of > Kafka topics. Can you please help. > below are the logs. > > 1. > playbook getting failed with error: - > ``` > TASK [k

Re: Unable to list Kafka topics

k listing of > Kafka topics. Can you please help. > below are the logs. > > 1. > playbook getting failed with error: - > ``` > TASK [kafka : Kafka Topics | List topics] >

Unable to list Kafka topics

Hi Team, I am running ansible to install elk on one server. (on Dev env, standalone setup) but my playbook is getting failed at kafka role for task listing of Kafka topics. Can you please help. below are the logs. 1. playbook getting failed with error: - ``` TASK [kafka : Kafka Topics | List

Re: Priority queues using kafka topics and a consumer group?

Richard, There is; but it doesn't look like what you would do in a typical messaging technology. As you may know Apache Kafka® is not messaging but a distributed streaming platform based on the concept of commit log. The main characteristic of a commit log is that they keep records in the sam

Priority queues using kafka topics and a consumer group?

Trying to see if there is a way to implement priority topics where a single consumer group is reading from multiple topics each denoting a level of priority and having the Kafka client automatically prioritize reads against the message in the higher priority topics.

Re: KSQL to search for data in Kafka Topics

I'm not sure what you mean by "not using stream topology". What does that mean to you that you'd rather avoid? However, you can indeed use KSQL to define streams & tables that process data from a number of topics. However, I think you may have the misimpression that KSQL is designed so you can exe

Re: Persist Kafka Topics and ksqldb

You need to externalise your container data stores. Here's an example Docker Compose that does that: https://github.com/confluentinc/demo-scene/blob/master/wifi-fun/docker-compose.yml -- Robin Moffatt | Sen

Persist Kafka Topics and ksqldb

After a *docker-compose restart. *All topics, ksqldb types are lost. Is there any way to safely persist all data ? docker-compse : --- version: "2" services: zookeeper: image: confluentinc/cp-zookeeper:latest container_name: zookeeper environment: ZOOKEEPER_CLIENT_PORT: 2181

KSQL to search for data in Kafka Topics

Hello, I am quite new to KSQL, so apologise for misunderstanding it's concept. I have a list of topics that I want to search data for. I am not using stream process, but plain topics which has data retained for 14 days. All i want to do is search for data in SQL-like way as long as it's within th

Re: java.net.BindException: Address already in use (Bind failed) with kafka-topics command

a.net.BindException: Address already in use (Bind failed) > with kafka-topics command > > Hello, > > Looks like your server is running out of ephemeral ports. The below > command confirms it is running out of ephemeral ports. We can restart the > process to solve the issu

RE: java.net.BindException: Address already in use (Bind failed) with kafka-topics command

use (Bind failed) with kafka-topics command Hello, Looks like your server is running out of ephemeral ports. The below command confirms it is running out of ephemeral ports. We can restart the process to solve the issue. Must probably root cause of the issue will be wrong configuration of

Re: java.net.BindException: Address already in use (Bind failed) with kafka-topics command

| wc -l Regards, Kalai On Thu, 9 Jan, 2020, 10:46 AM JOHN, BIBIN, wrote: > Could you please let me know why sometimes I am getting, below exception? > Cluster is up and running. Kafka and ZK is running in same server. We also > have jmx exporter configured for monitoring. > &g

java.net.BindException: Address already in use (Bind failed) with kafka-topics command

Could you please let me know why sometimes I am getting, below exception? Cluster is up and running. Kafka and ZK is running in same server. We also have jmx exporter configured for monitoring. kafka-topics --zookeeper localhost:2181 --list Error: Exception thrown by the agent

How spark structured streaming consumers initiated and invoked while reading multi-partitioned kafka topics?

Hi, Any suggestions regarding below issue? https://stackoverflow.com/questions/56524921/how-spark-structured-streaming-consumers-initiated-and-invoked-while-reading-mul Thanks, Shyam

Re: Kafka Topics

Team > > Could you please tell me about kafka topics, > > we have getting data from source to kafka about 200GB for daily basics > > how many partitions are required to pull data from source. > > > tell me about how to set no.of partitions for a topic?? > > is there a

Kafka Topics

Dear Team Could you please tell me about kafka topics, we have getting data from source to kafka about 200GB for daily basics how many partitions are required to pull data from source. tell me about how to set no.of partitions for a topic?? is there any mathematical rules for partitions

Re: How to balance messages in kafka topics with newly added partitions?

Yes but I find this even easier to do with KSQL. CREATE STREAM OUTPUTTOPIC AS SELECT * FROM INPUTTOPIC; There are similar examples like this that also filter messages while copying, or change the message format while copying on the KSQL Recipe page here https://www.confluent.io/stream-processin

Re: How to balance messages in kafka topics with newly added partitions?

You can use MirrorMaker to copy data between topics. Ryanne On Sun, Jan 27, 2019, 7:12 AM jaaz jozz Thanks, Sönke > Is there any available kafka tool to move messages between topics? > > On Sun, Jan 27, 2019 at 2:40 PM Sönke Liebau > wrote: > > > Hi Jazz, > > > > I'm afraid the only way of reba

Re: How to balance messages in kafka topics with newly added partitions?

Thanks, Sönke Is there any available kafka tool to move messages between topics? On Sun, Jan 27, 2019 at 2:40 PM Sönke Liebau wrote: > Hi Jazz, > > I'm afraid the only way of rebalancing old messages is indeed to > rewrite them to the topic - thus creating duplication. > Once a message has been

Re: How to balance messages in kafka topics with newly added partitions?

Hi Jazz, I'm afraid the only way of rebalancing old messages is indeed to rewrite them to the topic - thus creating duplication. Once a message has been written to a partition by Kafka this assignment is final, there is no way of moving it to another partition. Changing the partition count of top

How to balance messages in kafka topics with newly added partitions?

Hello, I have kafka cluster with certain topic that had too few partitions, so a large backlog of messages was collected. After i added additional partitions, only the newly messages balanced between all the new partitions. What is the preferred way to balance the "old" backlog of messages inside

Re: Deploying Kafka topics in a kerberized Zookeeper without superuser (in a CI flow)

gt; > Is it possible to use kafka-topics.sh and kafka-configs.sh in a kerberized > environment with a non-superuser Kerberos principal and how can this be > made to happen? > Can you suggest an alternate solution to achieve CI for Kafka topics? > > Best regards, > Kristjan Peil

Deploying Kafka topics in a kerberized Zookeeper without superuser (in a CI flow)

a kerberized environment with a non-superuser Kerberos principal and how can this be made to happen? Can you suggest an alternate solution to achieve CI for Kafka topics? Best regards, Kristjan Peil

Kafka topics lost after system reboot

Hello, I have a 3 node,3 zookeeper Kafka cluster of a version 0.11.x.x I could see everytime after a reboot,all of my topics are lost. I found in Google that the data is lost if it's stored in /TMP/Kafka-logs directory But in our case the Kafka and the zookeeper data is stored in one of our persis

Re: Visualizing kafka topics data

Hello Akash, we use for simple cases Kafdrop: https://github.com/HomeAdvisor/Kafdrop Is a jar that raises an HTTP server. Hope it helps. -- Jonathan On Sat, Aug 4, 2018 at 12:56 AM Rahul Singh wrote: > Graylog, or Kibana (connected to ElasticSearch) > > > Rahul > On Aug 3, 2018, 5:12 PM -04

Re: Visualizing kafka topics data

Graylog, or Kibana (connected to ElasticSearch) Rahul On Aug 3, 2018, 5:12 PM -0400, Akash Jain , wrote: > In my Java application, I am using a third party library to collect the > memory profile, and send to a Kafka topic as JSON data. > > How can I make sense out of that JSON data? Is there a t

Visualizing kafka topics data

In my Java application, I am using a third party library to collect the memory profile, and send to a Kafka topic as JSON data. How can I make sense out of that JSON data? Is there a tool which can ingest data from a kafka topic and visualize it? Thanks, Akash

Re: Creating and deleting Kafka Topics in Scala App

figuration. > >> > >> > >> -Matthias > >> > >> On 7/31/18 8:57 AM, Pulkit Manchanda wrote: > >>> HI All, > >>> > >>> I am want to create and delete Kafka topics on runtime in my > Application. > >>> I

Re: Creating and deleting Kafka Topics in Scala App

t. But it doesn't work either. > > Pulkit > > On Tue, Jul 31, 2018 at 9:22 PM, Matthias J. Sax > wrote: > >> Is `delete.topic.enable` set to `true`? It's a broker configuration. >> >> >> -Matthias >> >> On 7/31/18 8:57 AM, Pulkit

Re: Creating and deleting Kafka Topics in Scala App

lkit Manchanda wrote: > > HI All, > > > > I am want to create and delete Kafka topics on runtime in my Application. > > I followed few projects on GitHub like > > https://github.com/simplesteph/kafka-0.11-examples/blob/master/src/main/ > scala/au/com/simplesteph/kafk

Re: Creating and deleting Kafka Topics in Scala App

Is `delete.topic.enable` set to `true`? It's a broker configuration. -Matthias On 7/31/18 8:57 AM, Pulkit Manchanda wrote: > HI All, > > I am want to create and delete Kafka topics on runtime in my Application. > I followed few projects on GitHub like > https://github.c

Creating and deleting Kafka Topics in Scala App

HI All, I am want to create and delete Kafka topics on runtime in my Application. I followed few projects on GitHub like https://github.com/simplesteph/kafka-0.11-examples/blob/master/src/main/scala/au/com/simplesteph/kafka/kafka0_11/demo/KafkaAdminClientDemo.scala But to no avail. The code runs

Avro Schema Generation from Json/Csv/SQL for Kafka topics

I have a Kafka related challenge and hoping someone else has faced this or has some pointers. This is NOT a *schema registry* question, it is a question regarding the generation of schemas. I already know how I’m managing these schemas once they are created. I need to manage potentially several

Re: Reliable way to purge data from Kafka topics

be able to handle duplicated messages. > > > > > > > > > On 25. May 2018, at 08:54, Shantanu Deshmukh < > shantanu...@gmail.com> > > > > wrote: > > > > > > > > > > Hello, > > > > > > > > > &g

Re: Reliable way to purge data from Kafka topics

; We have cross data center replication. Using Kafka mirror maker we > are > > > > replicating data from our primary cluster to backup cluster. Problem > > > arises > > > > when we start operating from backup cluster, in case of drill or > actual > > > >

Re: Reliable way to purge data from Kafka topics

t; > outage. Data gathered at backup cluster needs to be reverse-replicated > to > > > primary. To do that I can only think of two options. 1) Use a different > > CG > > > every time for mirror maker 2) Purge topics so that data sent by > primary > > > doesn't g

Re: Reliable way to purge data from Kafka topics

2) Purge topics so that data sent by primary > > doesn't get replicated back to primary again due to reverse replication. > > > > We have opted for purging Kafka topics which are under replication. I use > > kafka-topics.sh --alter command to set retention of topic to 5 second

Re: Reliable way to purge data from Kafka topics

e to reverse replication. > > We have opted for purging Kafka topics which are under replication. I use > kafka-topics.sh --alter command to set retention of topic to 5 seconds to > purge data. But this doesn't see to be a fool proof mechanism. Thread > responsible for doing this ev

Reliable way to purge data from Kafka topics

to primary. To do that I can only think of two options. 1) Use a different CG every time for mirror maker 2) Purge topics so that data sent by primary doesn't get replicated back to primary again due to reverse replication. We have opted for purging Kafka topics which are under replication.

subscribe kafka topics

what drives deletion of kafka topics

I've Kafka 10. I've a basic question - what determines when the Kafka topic marked for deletion gets deleted ? Today, i marked a topic for deletion, and it got deleted immediately (possibly because the topic was not being used for last few months ?) .. In earlier instances, i'd to wait for some ti

RE: Is it possible to disable caching for some kafka topics?

I don't think that is possible since Kafka uses the file system cache for this. -Dave -Original Message- From: Archie [mailto:anubhavnidhi1...@gmail.com] Sent: Monday, August 28, 2017 4:14 PM To: users@kafka.apache.org Subject: Re: Is it possible to disable caching for some kafka t

Re: Is it possible to disable caching for some kafka topics?

ta fast) and topic B's > consumers are offline. Since topic B's consumers are offline, I don't need > to cache any message related to topic B right now and want topic A to get > the whole cache. Is it possible to disable caching for some kafka topics? > > Thanks, > Archie >

Is it possible to disable caching for some kafka topics?

want topic A to get the whole cache. Is it possible to disable caching for some kafka topics? Thanks, Archie

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

> Thanks. > --Vahid > > > > > From: karan alang > To: users@kafka.apache.org > Date: 06/22/2017 11:14 PM > Subject:Re: Deleting/Purging data from Kafka topics (Kafka 0.10) > > > > Hi Vahid, > here is the output of the GetOffsetShell com

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

--time -2`. I hope this answers your question. Thanks. --Vahid From: karan alang To: users@kafka.apache.org Date: 06/22/2017 11:14 PM Subject:Re: Deleting/Purging data from Kafka topics (Kafka 0.10) Hi Vahid, here is the output of the GetOffsetShell commands (with --ti

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

-1` you should get test:0:105 > - with `--time -2` you should get test:0:105 > > Could you please advise whether you're seeing a different behavior? > > Thanks. > --Vahid > > > > > From: "Vahid S Hashemian" > To: users@kafka.apache.org > Da

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

ld you please advise whether you're seeing a different behavior? Thanks. --Vahid From: "Vahid S Hashemian" To: users@kafka.apache.org Date: 06/22/2017 06:43 PM Subject:Re: Deleting/Purging data from Kafka topics (Kafka 0.10) Hi Karan, I think the issue is in verif

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

@kafka.apache.org Date: 06/22/2017 06:09 PM Subject:Re: Deleting/Purging data from Kafka topics (Kafka 0.10) Hi Vahid, somehow, the changes suggested don't seem to be taking effect, and i dont see the data being purged from the topic. Here are the steps i followed - 1) topic is set with

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

ike in your example if this broker config > value is much higher, then the broker doesn't delete old logs regular > enough. > > --Vahid > > > > From: karan alang > To: users@kafka.apache.org > Date: 06/22/2017 12:27 PM > Subject:Deleting/Purging da

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

To: users@kafka.apache.org Date: 06/22/2017 12:27 PM Subject:Deleting/Purging data from Kafka topics (Kafka 0.10) Hi All - How do i go about deleting data from Kafka Topics ? I've Kafka 0.10 installed. I tried setting the parameter of the topic as shown below -> $KAFKA10_H

Deleting/Purging data from Kafka topics (Kafka 0.10)

Hi All - How do i go about deleting data from Kafka Topics ? I've Kafka 0.10 installed. I tried setting the parameter of the topic as shown below -> $KAFKA10_HOME/bin/kafka-topics.sh --zookeeper localhost:2161 --alter --topic mmtopic6 --config retention.ms=1000 I was expecting to have

Re: Re-Balancing Kafka topics - Best practices

> > > > > Hi All, > > > > > > Fpr Re-balancing Kafka partitions, we can set property -> > > > > > > > > > *auto.leader.rebalance.enable = true in server.properties file.* > > > > > > *Is that th

Re: Re-Balancing Kafka topics - Best practices

-> > > > > > > *auto.leader.rebalance.enable = true in server.properties file.* > > > > *Is that the recommended way or is it better to reBalance the kafka > > partitions manually ?(using *scripts - *kafka-preferred-replica- > > electi

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

ate Topic t1 on cluster c1 => Topic t1 on Cluster t2 > > > > Is that the best way of doing it ? > > > > > > > > > > On Tue, Jun 13, 2017 at 3:14 PM, Mohammed Manna > > wrote: > > > >> You should plan this ahead and make sure that your

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

g it ? > > > > > On Tue, Jun 13, 2017 at 3:14 PM, Mohammed Manna > wrote: > >> You should plan this ahead and make sure that your topic partition and >> replication factors are updated correctly using kafka topics admin or a >> batch job setup for Kafka Mirror Maker

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

aker to replicate Topic t1 on cluster c1 => Topic t1 on Cluster t2 Is that the best way of doing it ? On Tue, Jun 13, 2017 at 3:14 PM, Mohammed Manna wrote: > You should plan this ahead and make sure that your topic partition and > replication factors are updated correctly using kafka t

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

> Regards, > > On Tue, 13 Jun 2017 at 10:58 pm, karan alang > wrote: > > > Hello All, > > > > I've streaming data flowing into Kafka topics. > > What are the best practices wrt. adding or removing Kafka Brokers while > > data is flowing into Kaf

Re: Re-Balancing Kafka topics - Best practices

eassign-partition.sh)* > *One of the blogs mentioned that - it is preferable to Re-balance Kafka > topics manually, since setting * > > *auto.leader.rebalance.enable = true causes issues.* > > Pls let me know. > Any other best practices wrt. Re-balancing kafka topics ? > > t

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

You should plan this ahead and make sure that your topic partition and replication factors are updated correctly using kafka topics admin or a batch job setup for Kafka Mirror Maker. Apache wiki explains how Mirror maker can guarantee a failsafe mirroring. I cannot quite remember how deep the

Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Hello All, I've streaming data flowing into Kafka topics. What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ? for eg. Can I add Broker & re-balance the topics while the data flows into Kafka topic ? Pls. advise. Thanks!

Re-Balancing Kafka topics - Best practices

ign-partition.sh)* *One of the blogs mentioned that - it is preferable to Re-balance Kafka topics manually, since setting * *auto.leader.rebalance.enable = true causes issues.* Pls let me know. Any other best practices wrt. Re-balancing kafka topics ? thanks!

Re: kafka-topics[.sh]: fail to support connecting via broker / v0.10 style

I can be updated once the Kafka AdminAPI is available and does everything over the Kafka wire protocol that the current kafka-topics command does by talking directly with zookeeper. For example create a topic or delete a topic. Unfortunately is has to remain this way for just a little while longer

kafka-topics[.sh]: fail to support connecting via broker / v0.10 style

If I understand Kafka correctly, since v0.9 / v0.10, users are often recommended to connect consumers to the Kafka cluster via bootstrap.servers AKA broker node addresses. However, the kafka-topics shell scripts fails to support this interface, still requiring the legacy zookeeper connect string

Kafka Topics best practice for logging data pipeline use case

We are using latest Kafka and Logstash versions for ingesting several business apps logs(now few but eventually 100+) into ELK. We have a standardized logging structure for business apps to log data into Kafka topics and able to ingest into ELK via Kafka topics input plugin. Currently, we are

Kafka topics best practice

We are using latest Kafka and Logstash versions for ingesting several business apps(now few but eventually 100+) log data into ELK. We have a standardized logging structure for business apps to log data into Kafka topic and able to ingest into ELK via Kafka input plugin. Currently, we are using on

Kafka topics for logging from multiple apps

Hello, We are using Kafka as data pipeline for log aggregation into ElasticSearch. We have over 500 apps that will eventually send logs to Kafka topics for aggregation into ElasticSearch. Currently, we are using app level topic for sending log messages into Kafka. Wondering what is the best

Kafka - topics not getting deleted

Hi Kafka-users, Could anyone please suggest why am I not being able to delete the kafka topic from broker. Below are the commands I've executed $ ./kafka-topics.sh --list --zookeeper /kafka Shows ProducerConsumerOffsetTestTopic001 - marked for deletion $ ./kafka-topics.sh --describe -zookee

kafka topics failed UnknownHostException

Hi, Guys. Any idea about this error? I installed and add Kafka service through Cloudera Manager, which looks fine. But when I try to test it with list topics, it gave error like: ./kafka-topics.sh --list --zookeeper localhost:2181 java.net.UnknownHostException: opt at java.net.AbstractPlainS

Re: Kafka topics with infinite retention?

Since Kafka itself has replication, I'm not sure what HDFS backups would bring – how would you recover from e.g. all Kafka nodes blowing up if you only have an HDFS backup? Why not use MirrorMaker to replicate the cluster to a remote DC, with a process of reversing the direction in case you need to

Re: Kafka topics with infinite retention?

> I'm not sure what HDFS backups would bring I’m not sure how realistic the threat is, but I was thinking a case in which bug in Kafka corrupts the log files. I would personally sleep better knowing there is a Kafka-independent backup of all data. > how would you recover from e.g. all Kafka n

Re: Kafka topics with infinite retention?

If you use camus to backup you could use something like this? https://groups.google.com/forum/#!searchin/camus_etl/sumac/camus_etl/hIi1cbOCFvU/grqd7UHLoFUJ Please note I haven't used this, but remember seeing it ages ago. On Wed, Mar 16, 2016 at 2:24 PM, Giidox wrote: > > I'm not sure what HDF

Re: Kafka topics with infinite retention?

Good points. I would backup all partitions to HDFS (etc.), as fast as the data arrives. In case the Kafka becomes corrupted, the topics can be repopulated from the backup. In my case, all clients track their own offsets, so they should in theory be able to continue as if nothing had happened.

Re: Kafka topics with infinite retention?

new materialized views based on past >> events, and only having to deal with Kafka is simpler. >> 2) Instead of having some sort of two-phased import process where I need to >> first import historical data and then do a switchover to the Kafka topics, >> I'd rather just start

Re: Kafka topics with infinite retention?

d only having to deal with Kafka is simpler. > 2) Instead of having some sort of two-phased import process where I need to > first import historical data and then do a switchover to the Kafka topics, > I'd rather just start from offset 0 in the Kafka topics. > 3) I'd like t

Re: Kafka topics with infinite retention?

of events, since: > > > > > > 1) I want to make it easy to spin up new materialized views based on > past > > > events, and only having to deal with Kafka is simpler. > > > 2) Instead of having some sort of two-phased import process where I > need > > t

Re: Kafka topics with infinite retention?

terialized views based on past > > events, and only having to deal with Kafka is simpler. > > 2) Instead of having some sort of two-phased import process where I need > to > > first import historical data and then do a switchover to the Kafka > topics, > > I'd rat

Re: Kafka topics with infinite retention?

> first import historical data and then do a switchover to the Kafka topics, > I'd rather just start from offset 0 in the Kafka topics. > 3) I'd like to be able to use standard tooling where possible, and most > tools for ingesting events into e.g. Spark Streaming would be diff

Kafka topics with infinite retention?

simpler. 2) Instead of having some sort of two-phased import process where I need to first import historical data and then do a switchover to the Kafka topics, I'd rather just start from offset 0 in the Kafka topics. 3) I'd like to be able to use standard tooling where possible, and most

Re: How can i secure Kafka topics

:15 AM, "prateek arora" wrote: >Hi > >Can i secure Kafka topics like Apache hbase and hdfs have kerberos >security >mechanism ? > >i am not able to find out any document related to Kafka security . > > >please suggest how can i secure Kafka . > > > >Regards >Prateek

How can i secure Kafka topics

Hi Can i secure Kafka topics like Apache hbase and hdfs have kerberos security mechanism ? i am not able to find out any document related to Kafka security . please suggest how can i secure Kafka . Regards Prateek

Re: Number of kafka topics/partitions supported per cluster of n nodes

@Jiefu Gong, Are the results of your tests available publicly? Regards, Prabhjot On Tue, Jul 28, 2015 at 10:35 PM, Prabhjot Bharaj wrote: > I would be using the servers available at my place of work. I dont have > access to AWS servers. I would starting off with a small number of nodes in > th

  1   2   >