[
https://issues.apache.org/jira/browse/KAFKA-6503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch updated KAFKA-6503:
---------------------------------
Priority: Critical (was: Major)
> Connect: Plugin scan is very slow
> ---------------------------------
>
> Key: KAFKA-6503
> URL: https://issues.apache.org/jira/browse/KAFKA-6503
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 1.0.0
> Reporter: Per Steffensen
> Priority: Critical
>
> Just upgraded to 1.0.0. It seems some plugin scan has been introduced. It is
> very slow - see logs from starting my Kafka-Connect instance at the bottom.
> It takes almost 4 minutes scanning. I am running Kafka-Connect in docker
> based on confluentinc/cp-kafka-connect:4.0.0. I set plugin.path to
> /usr/share/java. The only thing I have added is a 13MB jar in
> /usr/share/java/kafka-connect-file-streamer-client containing two connectors
> and a converter. That one alone seems to take 20 secs.
> If it was just scanning in the background, and everything was working it
> probably would not be a big issue. But it does not. Right after starting the
> Kafka-Connect instance I try to create a connector via the /connectors
> endpoint, but it will not succeed before the plugin scanning has finished (4
> minutes)
> I am not even sure why scanning is necessary. Is it not always true that
> connectors, converters etc are mentioned by name, so to see if it exists,
> just try to load the class - the classloader will tell if it is available.
> Hmmm, there is probably a reason.
> Anyway, either it should be made much faster, or at least Kafka-Connect
> should be fully functional (or as functional as possible) while scanning is
> going on.
> {code}
> [2018-01-30 13:52:26,834] INFO Scanning for plugin classes. This might take a
> moment ... (org.apache.kafka.connect.cli.ConnectDistributed)
> [2018-01-30 13:52:27,218] INFO Loading plugin from:
> /usr/share/java/kafka-connect-file-streamer-client
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,037] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-file-streamer-client/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,038] INFO Added plugin
> 'com.tlt.common.files.streamer.client.kafka.connect.OneTaskPerStreamSourceConnectorManager'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,039] INFO Added plugin
> 'com.tlt.common.files.streamer.client.kafka.connect.OneTaskPerFilesStreamerServerSourceConnectorManager'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,040] INFO Added plugin
> 'com.tlt.common.files.streamer.client.kafka.connect.KafkaConnectByteArrayConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,049] INFO Loading plugin from:
> /usr/share/java/kafka-connect-elasticsearch
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:47,595] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-elasticsearch/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:47,611] INFO Added plugin
> 'io.confluent.connect.elasticsearch.ElasticsearchSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:47,651] INFO Loading plugin from:
> /usr/share/java/kafka-connect-jdbc
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:49,491] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-jdbc/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:49,491] INFO Added plugin
> 'io.confluent.connect.jdbc.JdbcSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:49,492] INFO Added plugin
> 'io.confluent.connect.jdbc.JdbcSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:49,663] INFO Loading plugin from:
> /usr/share/java/kafka-connect-s3
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:51,055] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-s3/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:51,055] INFO Added plugin
> 'io.confluent.connect.s3.S3SinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:51,061] INFO Added plugin
> 'io.confluent.connect.storage.tools.SchemaSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:51,064] INFO Added plugin
> 'io.confluent.connect.avro.AvroConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:52,745] INFO Loading plugin from:
> /usr/share/java/kafka-connect-storage-common
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:25,800] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-storage-common/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:26,402] INFO Loading plugin from:
> /usr/share/java/kafka-connect-hdfs
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:58,104] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-hdfs/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:58,104] INFO Added plugin
> 'io.confluent.connect.hdfs.tools.SchemaSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:58,104] INFO Added plugin
> 'io.confluent.connect.hdfs.HdfsSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:58,508] INFO Loading plugin from:
> /usr/share/java/monitoring-interceptors
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:00,714] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/monitoring-interceptors/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:00,715] INFO Loading plugin from:
> /usr/share/java/kafka-serde-tools
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:03,900] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-serde-tools/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:03,911] INFO Loading plugin from:
> /usr/share/java/confluent-rebalancer
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:11,702] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/confluent-rebalancer/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:11,722] INFO Loading plugin from:
> /usr/share/java/confluent-control-center
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,874] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/confluent-control-center/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,875] INFO Added plugin
> 'org.apache.kafka.connect.tools.MockSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,875] INFO Added plugin
> 'org.apache.kafka.connect.tools.SchemaSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,876] INFO Added plugin
> 'org.apache.kafka.connect.tools.MockConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,876] INFO Added plugin
> 'org.apache.kafka.connect.tools.MockSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,876] INFO Added plugin
> 'org.apache.kafka.connect.tools.VerifiableSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,877] INFO Added plugin
> 'org.apache.kafka.connect.tools.VerifiableSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,878] INFO Added plugin
> 'org.apache.kafka.connect.converters.ByteArrayConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,878] INFO Added plugin
> 'org.apache.kafka.connect.storage.StringConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,879] INFO Added plugin
> 'org.apache.kafka.connect.json.JsonConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,879] INFO Added plugin
> 'org.apache.kafka.connect.transforms.Cast$Value'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,879] INFO Added plugin
> 'org.apache.kafka.connect.transforms.Flatten$Key'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,880] INFO Added plugin
> 'org.apache.kafka.connect.transforms.TimestampConverter$Value'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,880] INFO Added plugin
> 'org.apache.kafka.connect.transforms.Cast$Key'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,881] INFO Added plugin
> 'org.apache.kafka.connect.transforms.MaskField$Value'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,881] INFO Added plugin
> 'org.apache.kafka.connect.transforms.TimestampRouter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,882] INFO Added plugin
> 'org.apache.kafka.connect.transforms.ExtractField$Value'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,882] INFO Added plugin
> 'org.apache.kafka.connect.transforms.TimestampConverter$Key'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,896] INFO Added plugin
> 'org.apache.kafka.connect.transforms.MaskField$Key'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,904] INFO Added plugin
> 'org.apache.kafka.connect.transforms.ExtractField$Key'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,904] INFO Added plugin
> 'org.apache.kafka.connect.transforms.Flatten$Value'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,905] INFO Added plugin
> 'org.apache.kafka.connect.transforms.ValueToKey'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,905] INFO Added plugin
> 'org.apache.kafka.connect.transforms.HoistField$Value'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,905] INFO Added plugin
> 'org.apache.kafka.connect.transforms.HoistField$Key'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,906] INFO Added plugin
> 'org.apache.kafka.connect.transforms.ReplaceField$Value'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,914] INFO Added plugin
> 'org.apache.kafka.connect.transforms.InsertField$Key'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,914] INFO Added plugin
> 'org.apache.kafka.connect.transforms.InsertField$Value'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,915] INFO Added plugin
> 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,915] INFO Added plugin
> 'org.apache.kafka.connect.transforms.RegexRouter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,915] INFO Added plugin
> 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,915] INFO Added plugin
> 'org.apache.kafka.connect.transforms.ReplaceField$Key'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,919] INFO Loading plugin from:
> /usr/share/java/confluent-common
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:23,165] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/confluent-common/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:23,175] INFO Loading plugin from: /usr/share/java/kafka
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:41,996] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/kafka/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:41,997] INFO Added plugin
> 'org.apache.kafka.connect.file.FileStreamSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:41,998] INFO Added plugin
> 'org.apache.kafka.connect.file.FileStreamSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:42,021] INFO Loading plugin from:
> /usr/share/java/confluent-serializers
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:44,081] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/confluent-serializers/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:44,089] INFO Loading plugin from:
> /usr/share/java/schema-registry
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:49,504] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/schema-registry/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:49,541] INFO Loading plugin from:
> /usr/share/java/rest-utils
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:52,094] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/usr/share/java/rest-utils/}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,632] INFO Registered loader:
> sun.misc.Launcher$AppClassLoader@18b4aac2
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,659] INFO Added aliases
> 'OneTaskPerFilesStreamerServerSourceConnectorManager' and
> 'OneTaskPerFilesStreamerServerSource' to plugin
> 'com.tlt.common.files.streamer.client.kafka.connect.OneTaskPerFilesStreamerServerSourceConnectorManager'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,663] INFO Added aliases
> 'OneTaskPerStreamSourceConnectorManager' and 'OneTaskPerStreamSource' to
> plugin
> 'com.tlt.common.files.streamer.client.kafka.connect.OneTaskPerStreamSourceConnectorManager'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,665] INFO Added aliases 'ElasticsearchSinkConnector' and
> 'ElasticsearchSink' to plugin
> 'io.confluent.connect.elasticsearch.ElasticsearchSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,667] INFO Added aliases 'HdfsSinkConnector' and
> 'HdfsSink' to plugin 'io.confluent.connect.hdfs.HdfsSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,668] INFO Added aliases 'JdbcSinkConnector' and
> 'JdbcSink' to plugin 'io.confluent.connect.jdbc.JdbcSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,670] INFO Added aliases 'JdbcSourceConnector' and
> 'JdbcSource' to plugin 'io.confluent.connect.jdbc.JdbcSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,672] INFO Added aliases 'S3SinkConnector' and 'S3Sink'
> to plugin 'io.confluent.connect.s3.S3SinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,673] INFO Added aliases 'FileStreamSinkConnector' and
> 'FileStreamSink' to plugin
> 'org.apache.kafka.connect.file.FileStreamSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,695] INFO Added aliases 'FileStreamSourceConnector' and
> 'FileStreamSource' to plugin
> 'org.apache.kafka.connect.file.FileStreamSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,697] INFO Added aliases 'MockConnector' and 'Mock' to
> plugin 'org.apache.kafka.connect.tools.MockConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,699] INFO Added aliases 'MockSinkConnector' and
> 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,700] INFO Added aliases 'MockSourceConnector' and
> 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,702] INFO Added aliases 'VerifiableSinkConnector' and
> 'VerifiableSink' to plugin
> 'org.apache.kafka.connect.tools.VerifiableSinkConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,705] INFO Added aliases 'VerifiableSourceConnector' and
> 'VerifiableSource' to plugin
> 'org.apache.kafka.connect.tools.VerifiableSourceConnector'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,706] INFO Added aliases 'KafkaConnectByteArrayConverter'
> and 'KafkaConnectByteArray' to plugin
> 'com.tlt.common.files.streamer.client.kafka.connect.KafkaConnectByteArrayConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,711] INFO Added aliases 'AvroConverter' and 'Avro' to
> plugin 'io.confluent.connect.avro.AvroConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,711] INFO Added aliases 'ByteArrayConverter' and
> 'ByteArray' to plugin
> 'org.apache.kafka.connect.converters.ByteArrayConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,715] INFO Added aliases 'JsonConverter' and 'Json' to
> plugin 'org.apache.kafka.connect.json.JsonConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,715] INFO Added aliases 'StringConverter' and 'String'
> to plugin 'org.apache.kafka.connect.storage.StringConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,727] INFO Added alias 'RegexRouter' to plugin
> 'org.apache.kafka.connect.transforms.RegexRouter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,738] INFO Added alias 'TimestampRouter' to plugin
> 'org.apache.kafka.connect.transforms.TimestampRouter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,739] INFO Added alias 'ValueToKey' to plugin
> 'org.apache.kafka.connect.transforms.ValueToKey'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
