Hello, I am learning Kafka, and I am trying to connect to a remote server/machine to pull data from a file which is continuously updating every 5 minutes. I tried to connect to the remote server/machine but couldn't get any success. I try to find examples but I couldn't get any which explain how i can connect to remote server/machine using kafka. Does kafka needed to be install on remote server also?According to Kafka documentation 'http://kafka.apache.org/quickstart' I follow step 7 on local machine and it works perfectly fine.Now I need to connect to a remote machine to pull data from a file. I usually enter to that machine via ssh [username + password]. but I had no idea how to configure Kafka for that remote server. I tried to make changes in config/server properties and change these properties listeners=PLAINTEXT://:9092 # The port the socket server listens on#port=9092 # Hostname the broker will bind to. If not set, the server will bind to all interfaces#host.name=localhost # Hostname the broker will advertise to producers and consumers. If not set, it uses the# value for "host.name" if configured. Otherwise, it will use the value returned from# java.net.InetAddress.getCanonicalHostName().advertised.host.name= i give here IP address of remote machine and in config/connect-file-source.properties
name=local-file-sourceconnector.class=org.apache.kafka.connect.file.FileStreamSourceConnectortasks.max=1file=/home/rob/data/lio.dat [path and filename on remote server which is updated continuously ]topic=connect-test and in config/connect-file-sink.properties name=local-file-sinkconnector.class=org.apache.kafka.connect.file.FileStreamSinkConnectortasks.max=1file=/home/lai/Desktop/see.txt [store data here]topics=connect-test when i run this command './bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties' i got this error 2017-03-07 17:02:47,272] INFO Sink task org.apache.kafka.connect.runtime.WorkerSinkTask@34a64b06 finished initialization and start (org.apache.kafka.connect.runtime.WorkerSinkTask:155)[2017-03-07 17:02:48,122] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:49,123] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:50,123] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:51,123] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:52,124] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:53,124] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:54,124] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:55,125] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:56,125] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:57,125] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:57,145] INFO org.apache.kafka.connect.runtime.WorkerSinkTask@34a64b06 Committing offsets (org.apache.kafka.connect.runtime.WorkerSinkTask:187)[2017-03-07 17:02:58,126] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:02:59,126] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:03:00,126] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98)[2017-03-07 17:03:01,127] WARN Couldn't find file for FileStreamSourceTask, sleeping to wait for it to be created (org.apache.kafka.connect.file.FileStreamSourceTask:98) if anybody help me or guide me toward any example which shows how can I connect to the remote server and pull data via Kafka. I can do the same job easily with rsync , but I am trying to implement Kafka because I would like to use additional features in the system. I would really thankful because I am struggling with this things from a couple of days. thanx
