maybe I should highlight, I only publish 1 key. so only one broker is going
to handle it. and only 1 stream instance handle it too. what's the typical
throughput/latency I should expect in this case? assuming the processing
logic is very very simple, just get data(integer) and sum. I am more
expecting 100,000 m/s and less than 10ms latency for a single powerful
broker.
Nan
On Mon, Aug 20, 2018 at 12:45 AM Nan Xu <nanxu1...@gmail.com> wrote:
> I did several test. one is with 10 brokers (remote server),
> one with 3 brokers. (local docker)
>
> both exhibit the same behavior, I was thinking the same but from at least
> the kafka log, I don't see a rebalance happening. and I am sure my cpu is
> only used about half. and all broker still running.
>
> Nan
>
>
>
> On Mon, Aug 20, 2018 at 12:18 AM Shantanu Deshmukh <shantanu...@gmail.com>
> wrote:
>
>> How many brokers are there in your cluster? This error usually comes when
>> one of the brokers who is leader for a partition dies and you are trying
>> to
>> access it.
>>
>> On Fri, Aug 17, 2018 at 9:23 PM Harish K <harish...@gmail.com> wrote:
>>
>> > Hi,
>> > I have installed Kafka and created topic but while data ingestion i
>> get
>> > some errors as follows.Any help would be really appreciated
>> >
>> >
>> > [2018-08-17 06:12:49,838] WARN Error while fetching metadata with
>> > correlation id 24 :
>> > {wikipedia=LEADER_NOT_AVAILABLE}(org.apache.kafka.clients.NetworkClient)
>> >
>> > *server log:*
>> >
>> > [2018-08-17 06:06:00,719] INFO Creating /controller (is it secure?
>> false)
>> > (kafka.utils.ZKCheckedEphemeral)
>> > [2018-08-17 06:06:00,720] INFO Result of znode creation is: OK
>> > (kafka.utils.ZKCheckedEphemeral)
>> > [2018-08-17 06:06:00,720] INFO 0 successfully elected as leader
>> > (kafka.server.ZookeeperLeaderElector)
>> > [2018-08-17 06:06:00,736] ERROR Error while electing or becoming leader
>> on
>> > broker 0 (kafka.server.ZookeeperLeaderElector)
>> > kafka.common.KafkaException: Can't parse json string: null
>> > at kafka.utils.Json$.liftedTree1$1(Json.scala:40)
>> > at kafka.utils.Json$.parseFull(Json.scala:36)
>> > at
>> >
>> >
>> kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:660)
>> > at
>> >
>> >
>> kafka.utils.ZkUtils$$anonfun$getReplicaAssignmentForTopics$1.apply(ZkUtils.scala:656)
>> > at
>> >
>> >
>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>> > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>> > at kafka.utils.ZkUtils.getReplicaAssignmentForTopics(ZkUtils.scala:656)
>> > at
>> >
>> >
>> kafka.controller.KafkaController.initializeControllerContext(KafkaController.scala:742)
>> > at
>> >
>> >
>> kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:333)
>> > at
>> >
>> >
>> kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:160)
>> > at
>> >
>> kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:85)
>> > at
>> >
>> >
>> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply$mcZ$sp(ZookeeperLeaderElector.scala:154)
>> > at
>> >
>> >
>> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:154)
>> > at
>> >
>> >
>> kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:154)
>> > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
>> > at
>> >
>> >
>> kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:153)
>> > at org.I0Itec.zkclient.ZkClient$9.run(ZkClient.java:825)
>> > at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:72)
>> > Caused by: java.lang.NullPointerException
>> > at
>> >
>> >
>> scala.util.parsing.combinator.lexical.Scanners$Scanner.<init>(Scanners.scala:44)
>> > at scala.util.parsing.json.JSON$.parseRaw(JSON.scala:51)
>> > at scala.util.parsing.json.JSON$.parseFull(JSON.scala:65)
>> > at kafka.utils.Json$.liftedTree1$1(Json.scala:37)
>> > ... 17 more
>> >
>> >
>> > *Controller Log:*
>> >
>> > [2018-08-17 06:05:54,644] INFO [Controller 0]: Controller starting up
>> > (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,659] INFO [Controller 0]: Broker 0 starting become
>> > controller state transition (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,661] INFO [Controller 0]: Initialized controller
>> epoch
>> > to 2294948 and zk version 2294947 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,664] INFO [Controller 0]: Controller 0 incremented
>> > epoch to 2294949 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,665] DEBUG [Controller 0]: Registering
>> > IsrChangeNotificationListener (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,705] INFO [Controller 0]: Controller startup
>> complete
>> > (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,715] DEBUG [Controller 0]: Controller resigning,
>> > broker id 0 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,715] DEBUG [Controller 0]: De-registering
>> > IsrChangeNotificationListener (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,717] INFO [Partition state machine on Controller
>> 0]:
>> > Stopped partition state machine (kafka.controller.PartitionStateMachine)
>> > [2018-08-17 06:05:54,718] INFO [Replica state machine on controller 0]:
>> > Stopped replica state machine (kafka.controller.ReplicaStateMachine)
>> > [2018-08-17 06:05:54,718] INFO [Controller 0]: Broker 0 resigned as the
>> > controller (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,727] INFO [Controller 0]: Broker 0 starting become
>> > controller state transition (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,728] INFO [Controller 0]: Initialized controller
>> epoch
>> > to 2294949 and zk version 2294948 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,730] INFO [Controller 0]: Controller 0 incremented
>> > epoch to 2294950 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,730] DEBUG [Controller 0]: Registering
>> > IsrChangeNotificationListener (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,777] INFO [Controller 0]: Broker 0 starting become
>> > controller state transition (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,778] INFO [Controller 0]: Initialized controller
>> epoch
>> > to 2294950 and zk version 2294949 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,784] INFO [Controller 0]: Controller 0 incremented
>> > epoch to 2294951 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,791] DEBUG [Controller 0]: Registering
>> > IsrChangeNotificationListener (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,835] INFO [Controller 0]: Broker 0 starting become
>> > controller state transition (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,838] INFO [Controller 0]: Initialized controller
>> epoch
>> > to 2294951 and zk version 2294950 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,839] INFO [Controller 0]: Controller 0 incremented
>> > epoch to 2294952 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,839] DEBUG [Controller 0]: Registering
>> > IsrChangeNotificationListener (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,925] INFO [Controller 0]: Broker 0 starting become
>> > controller state transition (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,926] INFO [Controller 0]: Initialized controller
>> epoch
>> > to 2294952 and zk version 2294951 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,928] INFO [Controller 0]: Controller 0 incremented
>> > epoch to 2294953 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,929] DEBUG [Controller 0]: Registering
>> > IsrChangeNotificationListener (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,965] INFO [Controller 0]: Broker 0 starting become
>> > controller state transition (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,966] INFO [Controller 0]: Initialized controller
>> epoch
>> > to 2294953 and zk version 2294952 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,967] INFO [Controller 0]: Controller 0 incremented
>> > epoch to 2294954 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,968] DEBUG [Controller 0]: Registering
>> > IsrChangeNotificationListener (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,992] INFO [Controller 0]: Broker 0 starting become
>> > controller state transition (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,993] INFO [Controller 0]: Initialized controller
>> epoch
>> > to 2294954 and zk version 2294953 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,995] INFO [Controller 0]: Controller 0 incremented
>> > epoch to 2294955 (kafka.controller.KafkaController)
>> > [2018-08-17 06:05:54,996] DEBUG [Controller 0]: Registering
>> > IsrChangeNotificationListener (kafka.controller.KafkaController)
>> >
>>
>
