[
https://issues.apache.org/jira/browse/KAFKA-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Hauch updated KAFKA-6831:
---------------------------------
Component/s: KafkaConnect
> FileStreamSink is very slow
> ---------------------------
>
> Key: KAFKA-6831
> URL: https://issues.apache.org/jira/browse/KAFKA-6831
> Project: Kafka
> Issue Type: Test
> Components: KafkaConnect
> Affects Versions: 1.1.0
> Reporter: Prabhat Verma
> Priority: Major
>
> Hi Team,
>
> I am very new in kafka. My project requirement is fetch data from source
> location and place it in other other location (consumer location). I am using
> FileStreamSink class to perform above action.
> I am using Linux machine having memory of 32 GB.
> When i start FIleStreamSink , It is syncing to consumer location very very
> slowly. Not sure why it is taking 2000 message at a time and then sync it.
> After that it wait for few second then sync again. This waiting time
> increases per run .
>
> I am processing 600K message but it took 1 hrs to process only 60K message.
>
> Below are my config details :
>
> connect-file-sink.property
> Name = local-file
> Connector.class = FileStreamSource
> task.max=20
> file=/d/d1/kafka/destination/outfile.txt
> topic=abc_partion_20
> connect-file-source.property
> Name = local-file
> Connector.class = FileStreamSource
> task.max=20
> file=/d/d1/kafka/source/infile.txt
> topic=abc_partion_20
>
> Can you please help ?
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
