Re: Spark streaming readind avro from kafka

Hi Neeraj, You might find Kafka-Direct useful. BTW, are you using something like Confluent for you Kafka setup. If that's the case you might leverage Schema registry to get hold of the associated schema without additional effor

Re: Spark streaming readind avro from kafka

Avro file contains metadata with schema(writer schema) in Kafka there is no such thing, you should put message that will contain some reference to known schema(put whole schema will have big overhead) some people use schema registry solution On 1 June 2016 at 21:02, justneeraj wrote: > +1 > > I

Re: Spark streaming readind avro from kafka

+1 I am trying to read avro from kafka and I don't want to limit to a small set of schema. So I want to dynamically load the schema from avro file (as avro contains schema as well). And then from this I want to create a dataframe and run some queries on that. Any help would be really thankful.