As part of our load tests on Kafka, we are trying to create a certain
number of sink connectors, each one with a unique name and topic and
therefore, each sink connector is in his own consumer group.
For every new connector added, a rebalance is triggered for all the
connectors (the more connectors created before, the longer the current
rebalance).
Our creation is done by sending parallel create requests to the connector
RestAPI (we have tried doing this in several ways – 1000 requests, the
desired amount, at once, and also in smaller “batches” – 10/20 requests).
We have a few questions:
Why is the rebalance happening?
Why does it take so long (can be hours)?
Is the rebalance configurable?
The cluster is deployed using strimzi on kuberentes.
The connectors are created with the following template:
POST connectors/HttpSinkConnector-{index}
{
"name": "HttpSinkConnector-{index}",
"config": {
"connector.class": "com. connectors.rest.sink.HttpSinkConnector",
"tasks.max": "1",
"topics": "http-sink-connector-{index}",
"http.api.url": "{ some url }",
"headers": "Content-Type:application/json"
}
}
Connector Worker configuration:
offset.storage.topic=connect-cluster-offsets
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=connect-cluster-configs
key.converter=org.apache.kafka.connect.json.JsonConverter
group.id=connect-cluster
status.storage.topic=connect-cluster-status
config.storage.replication.factor=3
connector.client.config.override.policy=All
consumer.override.auto.offset.reset=latest
heartbeat.interval.ms=100000
key.converter.schemas.enable=false
offset.storage.replication.factor=3
session.timeout.ms=300000
status.storage.replication.factor=3
value.converter.schemas.enable=false
