Mayuresh,
Thanks for the details. I'll need to do some more tests to get back with
specific numbers re delay and check for timeouts.
For now (pre KIP-291 being implemented), the only parameters that will tune
leader election will be the zookeeper timeout and increasing the number of
network threa
Hi Mark,
Reducing the zookeeper session timeouts would enable the broker change
zookeeper listener on the controller to fire earlier than later. This will
enable the controller to detect that the broker is down earlier.
Increasing the network threads (processor threads) would not help here,
unles
Hi, there,
We will deploy a large-scale kafka cluster using kafka 2.0.0 in production
environment recently, as JunRao recommend in cluster-limit lately, to
accommodate for the rare event of a hard failure of the controller, it is
better to limit each broker to have up to 4,000 partitions an
Mike, sending to Kafka is about as fast as writing to a file and is
generally asynchronous. You might consider sending short messages as your
data is collected, and tagging each one with an execution ID. Sending from
onExecutionFinish() also sounds very reasonable.
Ryanne
On Mon, Dec 10, 2018, 1:
