Ewen Cheslack-Postava created KAFKA-5498:
--------------------------------------------
Summary: Connect validation API stops returning recommendations
for some fields after the right sequence of requests
Key: KAFKA-5498
URL: https://issues.apache.org/jira/browse/KAFKA-5498
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava
Fix For: 0.11.0.0
If you issue the right sequence of requests against this API, it starts
behaving differently, omitting certain fields (at a minimum recommended
values, which is how I noticed this). If you start with
{code}
$ curl -X PUT -H "Content-Type: application/json" --data '{"connector.class":
"org.apache.kafka.connect.file.FileStreamSourceConnector", "name": "file",
"transforms": "foo"}'
http://localhost:8083/connector-plugins/org.apache.kafka.connect.file.FileStreamSourceConnector/config/validate
| jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5845 100 5730 100 115 36642 735 --:--:-- --:--:-- --:--:-- 36496
{
"name": "org.apache.kafka.connect.file.FileStreamSourceConnector",
"error_count": 4,
"groups": [
"Common",
"Transforms",
"Transforms: foo"
],
"configs": [
{
"definition": {
"name": "name",
"type": "STRING",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "Globally unique name to use for this connector.",
"group": "Common",
"width": "MEDIUM",
"display_name": "Connector name",
"dependents": [],
"order": 1
},
"value": {
"name": "name",
"value": "file",
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "connector.class",
"type": "STRING",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "Name or alias of the class for this connector. Must
be a subclass of org.apache.kafka.connect.connector.Connector. If the connector
is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either
specify this full name, or use \"FileStreamSink\" or
\"FileStreamSinkConnector\" to make the configuration a bit shorter",
"group": "Common",
"width": "LONG",
"display_name": "Connector class",
"dependents": [],
"order": 2
},
"value": {
"name": "connector.class",
"value": "org.apache.kafka.connect.file.FileStreamSourceConnector",
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "tasks.max",
"type": "INT",
"required": false,
"default_value": "1",
"importance": "HIGH",
"documentation": "Maximum number of tasks to use for this connector.",
"group": "Common",
"width": "SHORT",
"display_name": "Tasks max",
"dependents": [],
"order": 3
},
"value": {
"name": "tasks.max",
"value": "1",
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "key.converter",
"type": "CLASS",
"required": false,
"default_value": null,
"importance": "LOW",
"documentation": "Converter class used to convert between Kafka Connect
format and the serialized form that is written to Kafka. This controls the
format of the keys in messages written to or read from Kafka, and since this is
independent of connectors it allows any connector to work with any
serialization format. Examples of common formats include JSON and Avro.",
"group": "Common",
"width": "SHORT",
"display_name": "Key converter class",
"dependents": [],
"order": 4
},
"value": {
"name": "key.converter",
"value": null,
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "value.converter",
"type": "CLASS",
"required": false,
"default_value": null,
"importance": "LOW",
"documentation": "Converter class used to convert between Kafka Connect
format and the serialized form that is written to Kafka. This controls the
format of the values in messages written to or read from Kafka, and since this
is independent of connectors it allows any connector to work with any
serialization format. Examples of common formats include JSON and Avro.",
"group": "Common",
"width": "SHORT",
"display_name": "Value converter class",
"dependents": [],
"order": 5
},
"value": {
"name": "value.converter",
"value": null,
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "transforms",
"type": "LIST",
"required": false,
"default_value": null,
"importance": "LOW",
"documentation": "Aliases for the transformations to be applied to
records.",
"group": "Transforms",
"width": "LONG",
"display_name": "Transforms",
"dependents": [],
"order": 6
},
"value": {
"name": "transforms",
"value": "foo",
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "transforms.foo.type",
"type": "CLASS",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "Class for the 'foo' transformation.",
"group": "Transforms: foo",
"width": "LONG",
"display_name": "Transformation type for foo",
"dependents": [],
"order": 0
},
"value": {
"name": "transforms.foo.type",
"value": null,
"recommended_values": [
"org.apache.kafka.connect.transforms.Cast$Key",
"org.apache.kafka.connect.transforms.Cast$Value",
"org.apache.kafka.connect.transforms.ExtractField$Key",
"org.apache.kafka.connect.transforms.ExtractField$Value",
"org.apache.kafka.connect.transforms.Flatten$Key",
"org.apache.kafka.connect.transforms.Flatten$Value",
"org.apache.kafka.connect.transforms.HoistField$Key",
"org.apache.kafka.connect.transforms.HoistField$Value",
"org.apache.kafka.connect.transforms.InsertField$Key",
"org.apache.kafka.connect.transforms.InsertField$Value",
"org.apache.kafka.connect.transforms.MaskField$Key",
"org.apache.kafka.connect.transforms.MaskField$Value",
"org.apache.kafka.connect.transforms.RegexRouter",
"org.apache.kafka.connect.transforms.ReplaceField$Key",
"org.apache.kafka.connect.transforms.ReplaceField$Value",
"org.apache.kafka.connect.transforms.SetSchemaMetadata$Key",
"org.apache.kafka.connect.transforms.SetSchemaMetadata$Value",
"org.apache.kafka.connect.transforms.TimestampConverter$Key",
"org.apache.kafka.connect.transforms.TimestampConverter$Value",
"org.apache.kafka.connect.transforms.TimestampRouter",
"org.apache.kafka.connect.transforms.ValueToKey"
],
"errors": [
"Missing required configuration \"transforms.foo.type\" which has no
default value.",
"Invalid value null for configuration transforms.foo.type: Not a
Transformation"
],
"visible": true
}
},
{
"definition": {
"name": "file",
"type": "STRING",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "Source filename.",
"group": null,
"width": "NONE",
"display_name": "file",
"dependents": [],
"order": -1
},
"value": {
"name": "file",
"value": null,
"recommended_values": [],
"errors": [
"Missing required configuration \"file\" which has no default value."
],
"visible": true
}
},
{
"definition": {
"name": "topic",
"type": "STRING",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "The topic to publish data to",
"group": null,
"width": "NONE",
"display_name": "topic",
"dependents": [],
"order": -1
},
"value": {
"name": "topic",
"value": null,
"recommended_values": [],
"errors": [
"Missing required configuration \"topic\" which has no default value."
],
"visible": true
}
}
]
}
{code}
You can see that the recommended values for transformations are there. However,
if you do
{code}
curl -X PUT -H "Content-Type: application/json" --data '{"connector.class":
"org.apache.kafka.connect.file.FileStreamSourceConnector", "name": "file"}'
http://localhost:8083/connector-plugins/org.apache.kafka.connect.file.FileStreamSourceConnector/config/validate
| jq
{code}
before that command, the recommended values will not appear.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
