Hi Ewen, Thanks for the explanation. with your suggested setting, I was able to start just a sink connector like below :
>* bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-sink.properties* But I have a couple of issues yet, 1) Since I am only testing a simple file sink connector, I am manually producing some messages to the 'connect-test' kafka topic, where the sink-Task is reading from. And it works only if the message is within double-quotes. 2) Once I hit the above JsonParser error on the SinkTask, the connector is hung, doesn't take any more messages even proper ones. On Tue, Nov 10, 2015 at 1:59 PM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > Hi Venkatesh, > > If you're using the default settings included in the sample configs, it'll > expect JSON data in a special format to support passing schemas along with > the data. This is turned on by default because it makes it possible to work > with a *lot* more connectors and data storage systems (many require > schemas!), though it does mean consuming regular JSON data won't work out > of the box. You can easily switch this off by changing these lines in the > worker config: > > key.converter.schemas.enable=true > value.converter.schemas.enable=true > > to be false instead. However, note that this will only work with connectors > that can work with "schemaless" data. This wouldn't work for, e.g., writing > Avro files in HDFS since they need schema information, but it might work > for other formats. This would allow you to consume JSON data from any topic > it already existed in. > > Note that JSON is not the only format you can use. You can also substitute > other implementations of the Converter interface. Confluent has implemented > an Avro version that works well with our schema registry ( > https://github.com/confluentinc/schema-registry/tree/master/avro-converter > ). > The JSON implementation made sense to add as the one included with Kafka > simply because it didn't introduce any other dependencies that weren't > already in Kafka. It's also possible to write implementations for other > formats (e.g. Thrift, Protocol Buffers, Cap'n Proto, MessagePack, and > more), but I'm not aware of anyone who has started to tackle those > converters yet. > > -Ewen > > On Tue, Nov 10, 2015 at 1:23 PM, Venkatesh Rudraraju < > venkatengineer...@gmail.com> wrote: > > > Hi, > > > > I am trying out the new kakfa connect service. > > > > version : kafka_2.11-0.9.0.0 > > mode : standalone > > > > I have a conceptual question on the service. > > > > Can I just start a sink connector which reads from Kafka and writes to > say > > HDFS ? > > From what I have tried, it's expecting a source-connector as well because > > the sink-connector is expecting a particular pattern of the message in > > kafka-topic. > > > > Thanks, > > Venkat > > > > > > -- > Thanks, > Ewen > -- Victory awaits him who has everything in order--luck, people call it.
