Csaba Galyo created KAFKA-9546:
----------------------------------
Summary: Make FileStreamSourceTask extendable with generic streams
Key: KAFKA-9546
URL: https://issues.apache.org/jira/browse/KAFKA-9546
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Reporter: Csaba Galyo
Use case: I want to read a ZIP compressed text file with a file connector and
send it to Kafka.
Currently, we have FileStreamSourceConnector which reads a \n delimited text
file. This connector always returns a task of type FileStreamSourceTask.
The FileStreamSourceTask reads from stdio or opens a file InputStream. The
issue with this approach is that the input needs to be a text file, otherwise
it won't work.
The code should be modified so that users could change the default InputStream
to eg. ZipInputStream, or any other format. The code is currently written in
such a way that it's not possible to extend it, we cannot use a different input
stream.
See example here where the code got copy-pasted just so it could read from a
ZstdInputStream (which reads ZSTD compressed files):
[https://github.com/gcsaba2/kafka-zstd/tree/master/src/main/java/org/apache/kafka/connect/file]
I suggest 2 changes:
# FileStreamSourceConnector should be extendable to return tasks of different
types. These types would be input by the user through the configuration map
# FileStreamSourceTask should be modified so it could be extended and child
classes could define different input streams.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
