Redirecting to a file will not work, we would not be able to create a
different filenames after some time or filesize. HDFS connect sink already
has those options , also i could use some minor transformations while using
the connect

On Tue, May 14, 2019 at 7:28 AM Hans Jespersen <h...@confluent.io> wrote:

> Can you just use kafka-console-consumer and just redirect the output into a
> file?
>
> -hans
>
>
> On Mon, May 13, 2019 at 1:55 PM Vinay Jain <vina...@gmail.com> wrote:
>
> > Hi
> >
> > The data needs to be transferred to some other system in other network,
> and
> > due to some security reasons, the other systems cannot be exposed . So
> the
> > available mechanism is file based integration. Is there a production
> ready
> > Kafka connect adapter which can create files in local directory.
> >
> > Regards
> > Vinay Jain
> >
> > On Mon, May 13, 2019 at 3:41 PM Robin Moffatt <ro...@confluent.io>
> wrote:
> >
> > > Can you explain more about why you're writing a file with the data?
> > > Presumably, this is for another application to consume; could it not
> take
> > > the data from Kafka directly, whether with a native client or over the
> > REST
> > > proxy?
> > > Oftentimes local files are unnecessary 'duck tape' for integration that
> > can
> > > be done in a better way.
> > >
> > >
> > > --
> > >
> > > Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff
> > >
> > >
> > > On Mon, 13 May 2019 at 01:35, Vinay Jain <vina...@gmail.com> wrote:
> > >
> > > > Hi
> > > >
> > > > I would like to consume a Topic and save the AVRO messages in local
> > > > directory in the AVRO file formats. As per Kafka Connect File Stream
> > > > documentation, it is not for production use.
> > > >
> > > > Other Option I am thinking to use Kafka Connect - HDFS Sink but I am
> > not
> > > > sure if it can also write to the Local directory if we pass in the
> > > variable
> > > > hdfs.url the URL for local file system instead of HDFS path.
> > > >
> > > > Will this work or are there any other ready made options which can be
> > > used
> > > > for the same.
> > > >
> > > > Regards
> > > >
> > > > Vinay
> > > >
> > >
> >
>

Reply via email to