[
https://issues.apache.org/jira/browse/AVRO-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108110#comment-17108110
]
Ryan Skraba commented on AVRO-2490:
-----------------------------------
Hello! This looks like a misunderstanding about what schema evolution means,
and probably not a bug.
The linked Oracle NoSQL documentation says that you can change an Avro schema
by adding a field with a default. This **IS** always true, but when you are
using the Java SDK by itself, you need to always provide the actual original
schema (i.e. a.k.a the old "writer" schema that generated the serialized data)
*_as well as_* the evolved schema (i.e. the new "reader" schema with your new
field).
The binary data is read using the original schema and mapped into the evolved
schema. Your last line of code should use the two parameter constructor for
[GenericDatumReader|https://avro.apache.org/docs/1.8.2/api/java/org/apache/avro/generic/GenericDatumReader.html#GenericDatumReader(org.apache.avro.Schema,%20org.apache.avro.Schema)].
There's a test case [demonstrating
this|https://github.com/apache/avro/blob/70260919426f89825ca148f5ee815f3b2cf4764d/lang/java/avro/src/test/java/org/apache/avro/TestSchemaBuilder.java#L779].
{code}
GenericRecord record = (GenericRecord)(new GenericDatumReader(originalSchema,
evolvedSchema)).read((Object)null, binaryDecoder);
{code}
The Oracle NoSQL doc implies that it is keeping track of the schemas being
used, and using the the "correct" writer and reader schemas automatically. If
you are writing your own code with the Java SDK, you need to keep track of your
schemas yourself.
> Unable to add field to existing avro schema
> -------------------------------------------
>
> Key: AVRO-2490
> URL: https://issues.apache.org/jira/browse/AVRO-2490
> Project: Apache Avro
> Issue Type: Bug
> Reporter: Martin Mucha
> Priority: Major
> Fix For: 1.9.0, 1.8.2
>
>
> As described in
> [https://docs.oracle.com/database/nosql-12.1.3.4/GettingStartedGuide/schemaevolution.html#changeschema-rules]
>
> I should be able to extend AVRO schema by adding optional and(or?)
> default-having field. If I do that though, deserialization fails for me. It
> happens for version 1.8.2 and 1.9.0 as well.
>
> Some cases:
> A) If I add new field defined as:
> {code:java}
> { "name": "newColumn", "type": ["null","string"], "default": null}{code}
> deserialization will fail with:
>
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 5 at
> org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:424) at
> org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290) at
> org.apache.avro.io.parsing.Parser.advance(Parser.java:88) at
> org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267) at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
> at
> tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToJson(JsonAvroConverter.java:83){code}
>
> if I add new field defined as:
> {code:java}
> { "name": "newColumn", "type": "string", "default": ""}{code}
> {{or}}
> {code:java}
> { "name": "newColumn", "type": "string", "default": "incorrect"}{code}
> (assuming I will fix it later in app)
> deserialization will fail with:
>
> {code:java}
> Caused by: org.apache.avro.AvroRuntimeException: Malformed data. Length is
> negative: -1 at
> org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336) at
> org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263) at
> org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201) at
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)
> at
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:414)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:181)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
> at
> tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToJson(JsonAvroConverter.java:83){code}
>
> {{what code does fail like that(that ± content of method convertToJson in
> JsonAvroConverter as seen in stacktrace.):}}
> {code:java}
> BinaryDecoder binaryDecoder = DecoderFactory.get().binaryDecoder(avro,
> (BinaryDecoder)null);
> GenericRecord record = (GenericRecord)(new
> GenericDatumReader(schema)).read((Object)null, binaryDecoder);{code}
> the second line produces given errors. Without new column added,
> deserialization works.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
