Kafka Streaming Question : reset offset

2016-07-20 Thread Pariksheet Barapatre
Hi Experts, I have written Kafka Streaming app that just filters rows based on some condition and load it to MongoDB. The streaming process is working fine but due to some flaw in my code, I want to reprocess whole data again. One way is to do this - kill streaming app , change consumer group i

Re: Kafka HDFS Connector

2016-06-22 Thread Pariksheet Barapatre
gt; > Dave Tauzell | Senior Software Engineer | Surescripts > > O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com > > Connect with us: Twitter I LinkedIn I Facebook I YouTube > > > > > > -Original Message- > > From: Pariksheet Ba

Re: Kafka HDFS Connector

2016-06-22 Thread Pariksheet Barapatre
ment? > 1) plaintext Kafka data --> plaintext HDFS data readable by hive > 2) plaintext Kafka data --> avro/parquet HDFS data readable by hive > > Regards, > > > > On Wed, Jun 22, 2016 at 6:02 AM, Pariksheet Barapatre < > pari.data...@gmail.com> wrote: > > &

Re: Kafka HDFS Connector

2016-06-22 Thread Pariksheet Barapatre
an use flume also. > > Thanks, > Mudit > > > > > On 6/22/16, 12:29 PM, "Pariksheet Barapatre" > wrote: > > >Anybody have any idea on this? > > > >Thanks > >Pari > > > >On 20 June 2016 at 14:36, Pariksheet Barapatre &g

Re: Kafka HDFS Connector

2016-06-22 Thread Pariksheet Barapatre
Anybody have any idea on this? Thanks Pari On 20 June 2016 at 14:36, Pariksheet Barapatre wrote: > Hello All, > > I have data coming from sensors into kafka cluster in text format > delimited by comma. > > How to offload this data to Hive periodically from Kafka. I gues

Kafka HDFS Connector

2016-06-20 Thread Pariksheet Barapatre
Hello All, I have data coming from sensors into kafka cluster in text format delimited by comma. How to offload this data to Hive periodically from Kafka. I guess, Kafka Connect should solve my problem but when I checked documentation, examples have only avro formatted data. Can you please provid

Re: Kafka Streaming - Window expiration

2016-06-13 Thread Pariksheet Barapatre
s > Eno > > > > On 13 Jun 2016, at 08:31, Pariksheet Barapatre > wrote: > > > > Hello Experts, > > > > As per documentation in kafka docs - > > *Windowing* is a common prerequisite for stateful transformations which > > group records in a

Kafka Streaming - Window expiration

2016-06-13 Thread Pariksheet Barapatre
Hello Experts, As per documentation in kafka docs - *Windowing* is a common prerequisite for stateful transformations which group records in a stream, for example, by their timestamps. A local state store is usually needed for a windowing operation to store recently received records based on the w

Kafka basic doubts

2016-02-22 Thread Pariksheet Barapatre
Hi All, Greetings..!!! This is my first email to Kafka Community. I have just started exploring Kafka on CDH5.5 cluster which ships with Kafka 0.8.2.1. I am able to run sample programs for producer as well as consumer (both high level and low level). Now I am trying to load messages from Kafka