Hi Michael,
Yes, I did try spark-avro 0.2.0 databricks project. I am using CHD5.3 which
is based on spark 1.2. Hence I'm bound to use spark-avro 0.2.0 instead of
the latest.
I'm not sure how spark-avro project can help me in this scenario.
1. I have JavaDStream of type avro generic record
:JavaDStream<GenericRecord> [This is the data being read from kafka topics]
2. I'm able to get JavaSchemaRDD using the avro file like this
final JavaSchemaRDD schemaRDD2 = AvroUtils.avroFile(sqlContext,
"/xyz-Project/trunk/src/main/resources/xyz.avro");
3. I don't know how I can apply schema in step 2 to data in step 1.
I chose to do something like this
JavaSchemaRDD schemaRDD = sqlContext.applySchema(genericRecordJavaRDD,
xyz.class);
Used avro maven plugin to generate xyz class in Java. But this is not
good because avro maven plugin creates a field SCHEMA which is not
supported in applySchema method.
Please let me know how to deal with this.
Appreciate your help
Thanks,
Yamini
On Tue, Apr 7, 2015 at 1:57 PM, Michael Armbrust <[email protected]>
wrote:
> Have you looked at spark-avro?
>
> https://github.com/databricks/spark-avro
>
> On Tue, Apr 7, 2015 at 3:57 AM, Yamini <[email protected]> wrote:
>
>> Using spark(1.2) streaming to read avro schema based topics flowing in
>> kafka
>> and then using spark sql context to register data as temp table. Avro
>> maven
>> plugin(1.7.7 version) generates the java bean class for the avro file but
>> includes a field named SCHEMA$ of type org.apache.avro.Schema which is not
>> supported in the JavaSQLContext class[Method : applySchema].
>> How to auto generate java bean class for the avro file and over come the
>> above mentioned problem.
>>
>> Thanks.
>>
>>
>>
>>
>> -----
>> Thanks,
>> Yamini
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/scala-MatchError-class-org-apache-avro-Schema-of-class-java-lang-Class-tp22402.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>>
>
