Assistance Needed: Retrieving Current User in CreateTopicPolicy for Topic Creation Restriction

Dear Kafka Community, I am reaching out for guidance on how to retrieve the current user who is attempting to create a topic within the CreateTopicPolicy implementation in Apache Kafka. Our goal is to restrict topic creation based on a configured maximum limit for each user. While working on

Re: Mirror maker disable auto topic creation

Thanks Ryanne Sent from Yahoo Mail for iPhone On Thursday, September 15, 2022, 12:00 AM, Ryanne Dolan wrote: The link is correct. Connect and mm2 create those internal topics at startup, whether or not auto topic creation is enabled. On Wed, Sep 14, 2022, 11:15 PM Mcs Vemuri wrote: > He

Mirror maker disable topic creation

Hello, Is there any way to disable topic creation in MM2? I tried setting the topic.creation.enable to False but the MM-offsets/configs/status topics are still created(broker auto topic creation is also set to false) I found this- https://groups.google.com/g/confluent-platform/c/JRZmpCEZElo

Re: Mirror maker disable auto topic creation

The link is correct. Connect and mm2 create those internal topics at startup, whether or not auto topic creation is enabled. On Wed, Sep 14, 2022, 11:15 PM Mcs Vemuri wrote: > Hello, > Is there any way to disable topic creation in MM2? I tried setting the > topic.creation.enable to

Mirror maker disable auto topic creation

Hello, Is there any way to disable topic creation in MM2? I tried setting the topic.creation.enable to False but the MM-offsets/configs/status topics are still created(broker auto topic creation is also set to false) I found this- https://groups.google.com/g/confluent-platform/c/JRZmpCEZElo

Re: Automation Script : Kafka topic creation

Thank you Men and Ran On Sat, Nov 6, 2021 at 7:23 PM Men Lim wrote: > I'm currently using Kafka-gitops. > > On Sat, Nov 6, 2021 at 3:35 AM Kafka Life wrote: > > > Dear Kafka experts > > > > does anyone have ready /automated script to create /delete /alter topics > in > > different environment

Re: Automation Script : Kafka topic creation

I'm currently using Kafka-gitops. On Sat, Nov 6, 2021 at 3:35 AM Kafka Life wrote: > Dear Kafka experts > > does anyone have ready /automated script to create /delete /alter topics in > different environments? > taking Configuration parameter as input . > > if yes i request you to kindly share i

Re: Automation Script : Kafka topic creation

https://github.com/kafka-ops/julie Note that I didn't try it myself, only heard about it... and I am not related to this project בתאריך שבת, 6 בנוב׳ 2021, 12:35, מאת Kafka Life ‏: > Dear Kafka experts > > does anyone have ready /automated script to create /delete /alter topics in > different en

Automation Script : Kafka topic creation

Dear Kafka experts does anyone have ready /automated script to create /delete /alter topics in different environments? taking Configuration parameter as input . if yes i request you to kindly share it with me .. please

Connect: Setting topic creation config via file (vs. API)

Hi all, I’m trying to make use of the somewhat new functionality of Connect to create new topics on the fly as described in KIP-158. My test i

How to restrict kafka Topic creation/deletion and ACL create/delete for a TLS user?

I have AWS MSK Kafka Cluster (only TLS) and i want to restrict Topic Creation/Deletion for all the tls users except one user ( i am restricting all the zookeeper nodes through security) Issue: Currently i am able to create/delete the topics and also update the ACLs from all the tls users (with

Re: Kafka topic creation issue

No success with using the internal zookeeper. Tried a fresh install and it started working. Thank you! On Sun, May 26, 2019 at 4:23 PM M. Manna wrote: > Hi, > > Can you try with zookeeper as the online QuickStart suggests ? See if you > can see any difference? > > Also, some topic may not be

Re: Kafka topic creation issue

Hi, Can you try with zookeeper as the online QuickStart suggests ? See if you can see any difference? Also, some topic may not be listed because zookeeper-handled topics may not be visible. Try that and let us know. Regards, On Sun, 26 May 2019 at 17:03, Kasthuri R Raman wrote: > Hi , > I'm

Kafka topic creation issue

Hi , I'm running zookeeper full version outside of kafka. When creating the topic, I do not get any output or error in the log file. kafka-topics.sh --list --bootstrap-server localhost:9092 is returning empty topic. *My configuration :* Zookeeper : 3.5.5 Kafka : 2.12-2 MacOS - 10.13.4 By using p

Re: Topic Creation two different config(s)

You can use AdminClient.createTopics with the necessary configuration. On Wed, May 8, 2019 at 12:08 PM SenthilKumar K wrote: > Hello Experts , We have a requirement to create topic dynamically with two > different config(s). Is this possible in Kafka ? > > Kafka Version : 2.2.0 > > Topics with d

Topic Creation two different config(s)

Hello Experts , We have a requirement to create topic dynamically with two different config(s). Is this possible in Kafka ? Kafka Version : 2.2.0 Topics with different settings: #1 - Set retention as 24 hours for free tier customers # 2 - Set retention as 72 hours for paid customers Note : Right

Re: Topic Creation fails - Need help

Broker is 100% running. ZK path shows /broker/ids/1 On Fri, Aug 18, 2017 at 1:02 AM, Yang Cui wrote: > please use zk client to check the path：/brokers/ids in ZK > > 发自我的 iPhone > > > 在 2017年8月18日，下午3:14，Raghav 写道： > > > > Hi > > > > I have a 1 broker and 1 zookeeper on the same VM. I am using K

Re: Topic Creation fails - Need help

please use zk client to check the path：/brokers/ids in ZK 发自我的 iPhone > 在 2017年8月18日，下午3:14，Raghav 写道： > > Hi > > I have a 1 broker and 1 zookeeper on the same VM. I am using Kafka 10.2.1. > I am trying to create a topic using below command: > > "bin/kafka-topics.sh --create --zookeeper local

Re: Topic Creation fails - Need help

your broker is not running 发自我的 iPhone > 在 2017年8月18日，下午3:14，Raghav 写道： > > Hi > > I have a 1 broker and 1 zookeeper on the same VM. I am using Kafka 10.2.1. > I am trying to create a topic using below command: > > "bin/kafka-topics.sh --create --zookeeper localhost:2181 > --replication-facto

Topic Creation fails - Need help

Hi I have a 1 broker and 1 zookeeper on the same VM. I am using Kafka 10.2.1. I am trying to create a topic using below command: "bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 16 --topic topicTest04" It fails with the below error. Just wondering why

Topic creation fails on Kerberos/SASL enabled brokers

I have setup a Kerberos/SASL enabled Kafka cluster with three brokers. Everything is working except an issue I have with the topic creation. When a topic is created with a principal different from the principal of the leader broker, a znode is create under 'kafka/brokers/topics' wit

Topic creation fails on Kerberos/SASL enabled brokers

Hello, I have setup a Kerberos/SASL enabled Kafka cluster with three brokers. Everything is working except an issue I have with the topic creation. When a topic is created with a principal different from the principal of the leader broker, a znode is create under 'kafka/brokers/topics&

Replica selection at the time of topic creation

next several topics? I understand that one can (a) specify the replicas list for each topic during topic creation (b) use kafka-reassign-partitions.sh to do a reassignment of partitions and (c) set auto.leader.rebalance.enable=true. Thanks, Buvana private def assignReplicasToBrokersRackUna

Re: Need a help in understanding __consumer_offsets topic creation in Kafka Cluster

Hi, We are using Kafka 0.9.0.2, and by default log cleaner is not enabled. When we enable the log cleaner, internal topic “__consumer_offsets” (around 1TB in size)starts compaction, and during compaction we are unable to consume/produce messages. Also, consumer groups failed in leader election.

Re: Need a help in understanding __consumer_offsets topic creation in Kafka Cluster

We changed the policy to "delete" dynamically for the __consumer_offsets topic and it was a better option than doing a cluster restart after enabling log compaction. Also, we found problems when you are replicating to a log compacted topic from a non-compacted topic (which is leader). On Mon, Mar

Re: Need a help in understanding __consumer_offsets topic creation in Kafka Cluster

Thanks @Stevo Slavić *Thanks, Kunal* *+91-9958189589* *Data Analyst* *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798 * *Blog:- **http://learnhardwithkunalgupta.blogspot.in * On Mon

Re: Need a help in understanding __consumer_offsets topic creation in Kafka Cluster

You are affected by this 0.9.0.0 bug https://issues.apache.org/jira/browse/KAFKA-2988 It was fixed for 0.9.0.1. You could just apply same fix to your 0.9.0.0 cluster but I'd recommend upgrading to 0.9.0.1. Kind regards, Stevo Slavic. On Mon, Mar 14, 2016, 07:10 Kunal Gupta wrote: > Hi everyone

Need a help in understanding __consumer_offsets topic creation in Kafka Cluster

Hi everyone, I am new here, recently join the group. I faced a problem in Kafka Cluster, a problem is described below. I am using Kafka version 0.9.0.0 We have established a Kafka Cluster of 3 machines where 2 machines are utilized for Kafka broker and same 3 machines utilized for zookeeper. Whe

Re: Topic creation using new Client Jar - 0.9.0

0.9.0 client does not yet supporting admin requests like topic creation, so you still need to do it through AdminUtils for now. We plan to add this support after KIP-4 is adopted: https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations

Topic creation using new Client Jar - 0.9.0

Hi All, Is it possible to create a topic programmatically with a specific topic configuration (number of partitions, replication factor, retention time, etc) using just the new 0.9.0 client jar? -Erik

Re: Unable to create a topic from console producer even though topic creation is enabled

It turns out that "auto.create.topics.enable=true" was actually getting overridden to false somewhere else, and ended up causing this issue. On Tue, Nov 3, 2015 at 8:19 PM, Artem Ervits wrote: > change the order of your commands > > *bin/kafka-console-producer.sh --broker-list localhost:9092 -

Re: Unable to create a topic from console producer even though topic creation is enabled

change the order of your commands *bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test* On Tue, Nov 3, 2015 at 7:23 AM, Kishore N C wrote: > Hi all, > > I have a 3-node Kafka cluster. I'm running into the following error when I > try to use the console producer to write to

Unable to create a topic from console producer even though topic creation is enabled

Hi all, I have a 3-node Kafka cluster. I'm running into the following error when I try to use the console producer to write to a topic that does *not* yet exist. I have ensured that "auto.create.topics.enable=true" in server.properties. The error: ubuntu@ip-XX-X-XXX-XX:/usr/local/kafka$ bin/kafk

Re: Issue with auto topic creation as well

t; > On Wed, Feb 4, 2015 at 9:23 PM, Jun Rao wrote: > > > >> Could you try the 0.8.2.0 release? It fixed one issue related to topic > >> creation. > >> > > > Jun, > > If you need more info let me know. Seems like TopicMetadataResponse i

Re: Auto topic creation occurring when auto.create.topics.enable=false

> is > > now created! > > > > I'm using poseidon/poseidon-cluster ruby libraries for the comms, but > don't > > believe they should be over-riding the broker configuration in terms of > > topic creation. Nothing in source code suggests anything, and I don't > > believe the protocol allows for it anyway. > > > > We're using Kafka 0.8.0 final for all brokers. Anyone come across this > > issue before? > > > > /Dave > > >

Re: Auto topic creation occurring when auto.create.topics.enable=false

no > exception, and kafka-list-topics shows the previously non-existent topic is > now created! > > I'm using poseidon/poseidon-cluster ruby libraries for the comms, but don't > believe they should be over-riding the broker configuration in terms of > topic creation. Not

Auto topic creation occurring when auto.create.topics.enable=false

every time I try to fetch messages from the broker, I get no exception, and kafka-list-topics shows the previously non-existent topic is now created! I'm using poseidon/poseidon-cluster ruby libraries for the comms, but don't believe they should be over-riding the broker configuration in te

Re: Issue with auto topic creation as well

On Thu, Feb 5, 2015 at 12:37 PM, Sumit Rangwala wrote: > > > On Wed, Feb 4, 2015 at 9:23 PM, Jun Rao wrote: > >> Could you try the 0.8.2.0 release? It fixed one issue related to topic >> creation. >> > Jun, If you need more info let me know. Seems like TopicM

Re: Issue with auto topic creation as well

On Wed, Feb 4, 2015 at 9:23 PM, Jun Rao wrote: > Could you try the 0.8.2.0 release? It fixed one issue related to topic > creation. > Jun, Tried with 0.8.2.0 and I still see the same error. I see the error given below almost incessantly on the client side for topic LAX1-GR

Re: Issue with auto topic creation as well

Could you try the 0.8.2.0 release? It fixed one issue related to topic creation. Thanks, Jun On Wed, Feb 4, 2015 at 12:58 AM, Sumit Rangwala wrote: > I am observing the following exception with kafka client: > > 2015-02-04 00:17:27,345 > > (LAX1-GRIFFIN-r8-14230374680

Issue with auto topic creation as well

$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) # Background on what my code is doing: In my setup kafka brokers are set for auto topic creation. In the scenario above a node informs other nodes (currently 5 in total) about

Re: Topic Creation

We have a ticket open for making admin tools like topic creation to be sync: KAFKA-1125 <https://issues.apache.org/jira/browse/KAFKA-1125> On Tue, Nov 18, 2014 at 9:25 PM, Jun Rao wrote: > Yes, that's the best option for now. The creation is async. > > Thanks, > > Ju

Re: Topic Creation

Yes, that's the best option for now. The creation is async. Thanks, Jun On Tue, Nov 18, 2014 at 8:55 AM, Bryan Baugher wrote: > Hi everyone, > > I'm looking for the API to create/manager topics. Is this class[1] the one > I should be using? Also what if I'm wanting to create the topic > synchr

Topic Creation

Hi everyone, I'm looking for the API to create/manager topics. Is this class[1] the one I should be using? Also what if I'm wanting to create the topic synchronously? [1] - https://github.com/apache/kafka/blob/0.8.1.1/core/src/main/scala/kafka/admin/AdminUtils.scala#L149-L157 Bryan

Re: Explicit topic creation and topic metadata availability

from same command on 0.8.1 branch is better, but still > same > > > > >>> exception: > > > > >>> > > > > >>> ~/git/oss/kafka [0.8.1|✔] > > > > >>> 21:12 $ bin/kafka-topics.sh --zookeeper 127.0.0.1:50194 --topic

Re: Explicit topic creation and topic metadata availability

topic command null > > > >>> java.util.NoSuchElementException > > > >>> at > > scala.collection.IterableLike$class.head(IterableLike.scala:101) > > > >>> at scala.collection.immutable.Map$EmptyMap$.head(Map.scala:73) >

Re: Explicit topic creation and topic metadata availability

1.apply(TopicCommand.scala:137) > > >>> at > > >>> > > > kafka.admin.TopicCommand$$anonfun$describeTopic$1.apply(TopicCommand.scala:127) > > >>> at > > >>> > > > scala.collection.mutable.ResizableArray$class.foreach(Re

Re: Explicit topic creation and topic metadata availability

.admin.TopicCommand$$anonfun$describeTopic$1.apply(TopicCommand.scala:127) > >>> at > >>> > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57) > >>> at > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43) >

Re: Explicit topic creation and topic metadata availability

at kafka.admin.TopicCommand.main(TopicCommand.scala) >>> >>> On Wed, Oct 22, 2014 at 5:30 PM, Guozhang Wang >>> wrote: >>> >>>> Hello Stevo, >>>> >>>> Your understanding about the configs are correct, and it is indeed w

Re: Explicit topic creation and topic metadata availability

tanding about the configs are correct, and it is indeed wired >>> that the producer gets the exception after topic is created. Could you >>> use >>> the kafka-topics command to check if the leaders exist? >>> >>> kafka-topics.sh --zookeeper XXX --top

Re: Explicit topic creation and topic metadata availability

eption after topic is created. Could you use >> the kafka-topics command to check if the leaders exist? >> >> kafka-topics.sh --zookeeper XXX --topic [topic-name] describe >> >> Guozhang >> >> On Wed, Oct 22, 2014 at 5:57 AM, Stevo Slavić wrote

Re: Explicit topic creation and topic metadata availability

4 at 5:57 AM, Stevo Slavić wrote: > > > Hello Apache Kafka users, > > > > Using Kafka 0.8.1.1 (single instance with single ZK 3.4.6 running > locally), > > with auto topic creation disabled, in a test I have topic created with > > AdminUtils.createTopic (AdminUtils.

Re: Explicit topic creation and topic metadata availability

, Oct 22, 2014 at 5:57 AM, Stevo Slavić wrote: > Hello Apache Kafka users, > > Using Kafka 0.8.1.1 (single instance with single ZK 3.4.6 running locally), > with auto topic creation disabled, in a test I have topic created with > AdminUtils.createTopic (AdminUtils.topicExists re

Explicit topic creation and topic metadata availability

Hello Apache Kafka users, Using Kafka 0.8.1.1 (single instance with single ZK 3.4.6 running locally), with auto topic creation disabled, in a test I have topic created with AdminUtils.createTopic (AdminUtils.topicExists returns true) but KafkaProducer on send request keeps throwing

Re: auto topic creation not working for attempts to consume non-existing topic

> On Fri, Oct 3, 2014 at 1:31 AM, Stevo Slavić wrote: > > > > > OK, thanks, > > > > > > Do you agree then that the docs for auto topic creation configuration > > > parameter are misleading and should be changed? > > > > > > An

Re: auto topic creation not working for attempts to consume non-existing topic

t; jun > > On Fri, Oct 3, 2014 at 1:31 AM, Stevo Slavić wrote: > > > OK, thanks, > > > > Do you agree then that the docs for auto topic creation configuration > > parameter are misleading and should be changed? > > > > Another issue is that when the topic

Re: auto topic creation not working for attempts to consume non-existing topic

Yes, the docs can be improved. Could you file a jira? For the 2nd issue, the new java producer handles this better. Thanks, jun On Fri, Oct 3, 2014 at 1:31 AM, Stevo Slavić wrote: > OK, thanks, > > Do you agree then that the docs for auto topic creation configuration > p

Re: Changing number of partitions after topic creation tool

Thanks. On Sat, Oct 4, 2014 at 7:36 PM, Neha Narkhede wrote: > You can do that with the topics command. Refer to the docs here - > http://kafka.apache.org/documentation.html#basic_ops_add_topic > > On Sat, Oct 4, 2014 at 4:29 PM, jova wrote: > > > I was reading online that a tool was coming to

Re: Changing number of partitions after topic creation tool

You can do that with the topics command. Refer to the docs here - http://kafka.apache.org/documentation.html#basic_ops_add_topic On Sat, Oct 4, 2014 at 4:29 PM, jova wrote: > I was reading online that a tool was coming to change the number of > partitions of a topic after it has been created. I

Changing number of partitions after topic creation tool

I was reading online that a tool was coming to change the number of partitions of a topic after it has been created. Is this tool available yet? -- Jovaughn Lockridge {developer : Github } {professional : LinkedIn

Re: auto topic creation not working for attempts to consume non-existing topic

OK, thanks, Do you agree then that the docs for auto topic creation configuration parameter are misleading and should be changed? Another issue is that when the topic auto creation is disabled, attempts to publish a message on a non-existing topic using high-level api will throw a generic

Re: auto topic creation not working for attempts to consume non-existing topic

In general, only writers should trigger auto topic creation, but not the readers. So, a topic can be auto created by the producer, but not the consumer. Thanks, Jun On Thu, Oct 2, 2014 at 2:44 PM, Stevo Slavić wrote: > Hello Apache Kafka community, > > auto.create.topics.enable conf

auto topic creation not working for attempts to consume non-existing topic

h positive consumer.timeout.ms configured, will not result in topic creation (I cannot see one registered in ZooKeeper). Additionally, for non-blocking consumer with timeout, no offset will be recorded. This further means, if such consumer had auto.offset.reset set to largest, that it will miss at

Re: calling ad-hoc topic creation from code

is thrown (rep. factor greater > than num. of brokers). > > There is no difference in this code and the script, apparently, i can only > see changes in the constructor used in zkClient. > > Any clue? > > > Thanks. > > > 2014-02-25 18:41 GMT+01:00 Neha Narkhede : &

Re: calling ad-hoc topic creation from code

clue? Thanks. 2014-02-25 18:41 GMT+01:00 Neha Narkhede : > David, > > Topic creation can fail if you specify the replication factor > # of > brokers in the cluster. Can you check if that is true in your case? > Unfortunately, I don't think we fail the createTopic() API

Re: calling ad-hoc topic creation from code

David, Topic creation can fail if you specify the replication factor > # of brokers in the cluster. Can you check if that is true in your case? Unfortunately, I don't think we fail the createTopic() API with the appropriate exception since there is still a race condition where the broker

Re: calling ad-hoc topic creation from code

This is the code that i can see in CreateTopicCommand var zkClient: ZkClient = null try { zkClient = new ZkClient(zkConnect, 3, 3, ZKStringSerializer) createTopic(zkClient, topic, nPartitions, replicationFactor, replicaAssignmentStr) println("creation succeeded!")

Re: calling ad-hoc topic creation from code

Is the ZK connection string + namespace the same btw the code and the script? Thanks, Jun On Tue, Feb 25, 2014 at 3:01 AM, David Morales de Frías < dmora...@paradigmatecnologico.com> wrote: > Hi there, > > I'm trying to create a topic from java code, by calling CreateTopicCommand: > > > *ZkCli

calling ad-hoc topic creation from code

Hi there, I'm trying to create a topic from java code, by calling CreateTopicCommand: *ZkClient zkClient = new ZkClient(zookeeperCluster, 3, 3);* *CreateTopicCommand.createTopic(zkClient, topic, numPartitions.intValue(),replicationFactor.intValue(), "");* *zkClient.close();* The prog

Re: Topic creation on restart

Ok, I updated https://issues.apache.org/jira/browse/KAFKA- with this scenario also. Jason On Tue, Nov 5, 2013 at 9:11 PM, Neha Narkhede wrote: > >. Could a fetch request from a consumer cause a Topic creation request > (seems > implausible). > > Yes, that seems lik

Re: Topic creation on restart

>. Could a fetch request from a consumer cause a Topic creation request (seems implausible). Yes, that seems like a way the broker can get into this situation. Thanks, Neha On Tue, Nov 5, 2013 at 4:58 PM, Jason Rosenberg wrote: > I don't know if I have a way to see the acc

Re: Topic creation on restart

I don't know if I have a way to see the access logs on the LB..(still trying to track that down). One thing I do see though, is that there are fetch requests from consumers, that are then followed by these Topic creation log messages, e.g. (replaced some of the specific strings in thi

Re: Topic creation on restart

dding a delay before enabling a freshly started broker in the > > > metadata vip that clients use, it seems to have drastically reduced the > > > number of these topic creation requests. However, not all of them. > > > > > > I still see (only some times) a handful of &quo

Re: Topic creation on restart

Can this unintentional topic creation be avoided by setting auto.create.topics.enable=false? Yes, but that is not the right fix. Thanks, Neha On Nov 4, 2013 10:21 PM, "Priya Matpadi" wrote: > Can this unintentional topic creation be avoided by setting > auto.create.top

Re: Topic creation on restart

Can this unintentional topic creation be avoided by setting auto.create.topics.enable=false? On Mon, Nov 4, 2013 at 9:40 PM, Jason Rosenberg wrote: > Ok, so this can happen, even if the node has not been placed back into > rotation, at the metadata vip? > > > On Tue, Nov 5, 2

Re: Topic creation on restart

> Thanks, > Neha > > > On Mon, Nov 4, 2013 at 11:47 AM, Jason Rosenberg wrote: > > > Ok, > > > > After adding a delay before enabling a freshly started broker in the > > metadata vip that clients use, it seems to have drastically reduced the > > numb

Re: Topic creation on restart

ata vip that clients use, it seems to have drastically reduced the > number of these topic creation requests. However, not all of them. > > I still see (only some times) a handful of "Topic creation" log messages, > that happen on a freshly started broker, during the time k

Re: Topic creation on restart

Ok, After adding a delay before enabling a freshly started broker in the metadata vip that clients use, it seems to have drastically reduced the number of these topic creation requests. However, not all of them. I still see (only some times) a handful of "Topic creation" log mess

Re: Topic creation on restart

nd delay before placing a broker into the > vip for metadata requests, it never seems to have this issue. So I'm not > sure about the 104 seconds number, other than that was how long the flood > of "Topic creation" log messages went on for (over 500 of these). > > Which

Re: Topic creation on restart

Neha, This cluster has on the order of 750 topics. It looks like if I add a 20 second delay before placing a broker into the vip for metadata requests, it never seems to have this issue. So I'm not sure about the 104 seconds number, other than that was how long the flood of "Topic cre

Re: Topic creation on restart

restart, I see lots of these messages > > > during > > > > > the restart (these happened in the logs of the 2nd server in the > > > cluster > > > > to > > > > > restart (broker 11), after having restarted started, and during the > > > time > &

Re: Topic creation on restart

gt; > Some times during a rolling restart, I see lots of these messages > > during > > > > the restart (these happened in the logs of the 2nd server in the > > cluster > > > to > > > > restart (broker 11), after having restarted started, and during the

Re: Topic creation on restart

time > > > the 3rd node (broker 9) is doing a controlled shutdown before > stopping). > > > > > > Is it creating new topics (it shouldn't have been!). Or is it moving > > > topics around, or what? > > > > > > 2013-11-01 03:25:55,6

Re: Topic creation on restart

> topics around, or what? > > > > 2013-11-01 03:25:55,661 INFO [kafka-request-handler-7] > admin.AdminUtils$ - > > Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 } > > 2013-11-01 03:26:01,577 INFO [kafka-request-handler-0

Re: Topic creation on restart

t; 2013-11-01 03:25:55,661 INFO [kafka-request-handler-7] admin.AdminUtils$ - > Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 } > 2013-11-01 03:26:01,577 INFO [kafka-request-handler-0] admin.AdminUtils$ - > Topic creation { "partitions

Re: Topic creation on restart

around, or what? > > 2013-11-01 03:25:55,661 INFO [kafka-request-handler-7] admin.AdminUtils$ - > Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 } > 2013-11-01 03:26:01,577 INFO [kafka-request-handler-0] admin.AdminUtils$ - > Topic creati

Topic creation on restart

it creating new topics (it shouldn't have been!). Or is it moving topics around, or what? 2013-11-01 03:25:55,661 INFO [kafka-request-handler-7] admin.AdminUtils$ - Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 } 2013-11-01 03:26:01

Re: high level consumer usage question with auto topic creation

One thing is that you need to make sure the consumer starts consuming from the beginning of the topic, otherwise by default, it will start from the latest message in the topic, from the time it starts up. Since the consumer and producer are asynchronous, it's hard to assert that the consumer is co

high level consumer usage question with auto topic creation

I am trying to write a junit test for working with an embedded Kafka 0.8 server where I send and receive messages just to verify my embedded server, producer and consumer wrappers work. Order of operation in the junit looks something like: -Start zk. [own thread] (wait for init) -Start Kafka [own