Sorry for the late reply Rog, been kinda busy lately. Please look into the schema evolution of Avro. Confluent has an excellent article on this: https://docs.confluent.io/current/schema-registry/avro.html
Could you try again with optional fields? e.g. "type": ["null", "array"]. Since the names are different, I would expect the default value (or even an exception). If you do a cat on the Avro file, you can see that the original schema is in the header of the file. The B field is not there the record, so the reader field is not compatible, so it won't work. I'll check if we can come up with a more meaningful exception. Cheers, Fokko Op vr 17 jan. 2020 om 17:02 schreef roger peppe <rogpe...@gmail.com>: > > > On Fri, 17 Jan 2020 at 13:35, Ryan Skraba <r...@skraba.com> wrote: > >> Hello! I just created a JIRA for this as an improvement :D >> https://issues.apache.org/jira/browse/AVRO-2689 >> >> To check evolution, we'd probably want to specify the reader schema in >> the GenericDatumReader created here: >> >> https://github.com/apache/avro/blob/master/lang/java/tools/src/main/java/org/apache/avro/tool/DataFileReadTool.java#L75 >> >> The writer schema is automatically set when the DataFileStream is >> created. If we want to set a different reader schema (than the one >> found in the file), it should be set by calling >> reader.setExpected(readerSchema) just after the DataFileStream is >> created. >> > > Ah, that's a good pointer, thanks! I was looking for an appropriate > constructor, but there didn't seem to be one. > > >> >> I think it's a pretty good idea -- it feels like we're seeing more >> questions about schema evolution these days, so that would be a neat >> way for a user to test (or to create reproducible scenarios for bug >> reports). If you're interested, feel free to take the JIRA! I'd be >> happy to help out. >> > > So, I've had a go at it... see > https://github.com/rogpeppe-contrib/avro/commit/1236e9d33207a11d557c1eb2a171972e085dfcf2 > > I did the following to see if it was working ("avro" is my shell script > wrapper around the avro-tools jar): > > % cat schema.avsc > { > "name": "R", > "type": "record", > "fields": [ > { > "name": "A", > "type": { > "type": "array", > "items": "int" > } > } > ] > } > % cat schema1.avsc > { > "name": "R", > "type": "record", > "fields": [ > { > "name": "B", > "type": "string", > "default": "hello" > } > ] > } > % > AVRO_TOOLS_JAR=/home/rog/other/avro/lang/java/tools/target/avro-tools-1.10.0-SNAPSHOT.ja% > avro random --count 1 --schema-file schema.avsc x.out > % avro tojson x.out > {"A":[-890831012,1123049230,302974832]} > % cp schema.avsc schema1.avsc > % avro tojson --reader-schema-file schema1.avsc x.out > Exception in thread "main" java.lang.ClassCastException: class > org.apache.avro.util.Utf8 cannot be cast to class java.util.Collection > (org.apache.avro.util.Utf8 is in unnamed module of loader 'app'; > java.util.Collection is in module java.base of loader 'bootstrap') > at > org.apache.avro.generic.GenericDatumWriter.getArraySize(GenericDatumWriter.java:258) > at > org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:228) > at > org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:136) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82) > at > org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:206) > at > org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:195) > at > org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:130) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72) > at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:99) > at org.apache.avro.tool.Main.run(Main.java:66) > at org.apache.avro.tool.Main.main(Main.java:55) > % > > I am a bit clueless when it comes to interpreting that exception... sorry > for the ignorance - this is the first Java code I've ever written! > Any idea what's going on? This is maybe getting a bit too noisy for the > list - feel to reply directly. > > cheers, > rog. > > >> Ryan >> >> >> On Fri, Jan 17, 2020 at 2:22 PM roger peppe <rogpe...@gmail.com> wrote: >> > >> > On Thu, 16 Jan 2020 at 17:21, Ryan Skraba <r...@skraba.com> wrote: >> >> >> >> didn't find anything currently in the avro-tools that uses both >> >> reader and writer schemas while deserializing data... It should be a >> >> pretty easy feature to add as an option to the DataFileReadTool >> >> (a.k.a. tojson)! >> > >> > >> > Thanks for that suggestion. I've been delving into that code a bit and >> trying to understand what's going on. >> > >> > At the heart of it is this code: >> > >> > GenericDatumReader<Object> reader = new GenericDatumReader<>(); >> > try (DataFileStream<Object> streamReader = new >> DataFileStream<>(inStream, reader)) { >> > Schema schema = streamReader.getSchema(); >> > DatumWriter<Object> writer = new GenericDatumWriter<>(schema); >> > JsonEncoder encoder = EncoderFactory.get().jsonEncoder(schema, >> out, pretty); >> > >> > I'm trying to work out where the best place to put the specific reader >> schema (taken from a command line flag) might be. >> > >> > Would it be best to do it when creating the DatumReader (it looks like >> there might be a way to create that with a generic writer schema and a >> specific reader schema, although I can't quite see how to do that atm), or >> when creating the DatumWriter? >> > Or perhaps there's a better way? >> > >> > Thanks for any guidance. >> > >> > cheers, >> > rog. >> >> >> >> >> >> You are correct about running ./build.sh dist in the java directory -- >> >> it fails with JDK 11 (likely fixable: >> >> https://issues.apache.org/jira/browse/MJAVADOC-562). >> >> >> >> You should probably do a simple mvn clean install instead and find the >> >> jar in lang/java/tools/target/avro-tools-1.10.0-SNAPSHOT.jar. That >> >> should work with JDK11 without any problem (well-tested in the build). >> >> >> >> Best regards, Ryan >> >> >> >> >> >> >> >> On Thu, Jan 16, 2020 at 5:49 PM roger peppe <rogpe...@gmail.com> >> wrote: >> >> > >> >> > Update: I tried running `build.sh dist` in `lang/java` and it failed >> (at least, it looks like a failure message) after downloading a load of >> Maven deps with the following errors: >> https://gist.github.com/rogpeppe/df05d993254dc5082253a5ef5027e965 >> >> > >> >> > Any hints on what I should do to build the avro-tools jar? >> >> > >> >> > cheers, >> >> > rog. >> >> > >> >> > On Thu, 16 Jan 2020 at 16:45, roger peppe <rogpe...@gmail.com> >> wrote: >> >> >> >> >> >> >> >> >> On Thu, 16 Jan 2020 at 13:57, Ryan Skraba <r...@skraba.com> wrote: >> >> >>> >> >> >>> Hello! Is it because you are using brew to install avro-tools? >> I'm >> >> >>> not entirely familiar with how it packages the command, but using a >> >> >>> direct bash-like solution instead might solve this problem of >> mixing >> >> >>> stdout and stderr. This could be the simplest (and right) solution >> >> >>> for piping. >> >> >> >> >> >> >> >> >> No, I downloaded the jar and am directly running it with "java -jar >> ~/other/avro-tools-1.9.1.jar". >> >> >> I'm using Ubuntu Linux 18.04 FWIW - the binary comes from Debian >> package openjdk-11-jre-headless. >> >> >> >> >> >> I'm going to try compiling avro-tools myself to investigate but I'm >> a total Java ignoramus - wish me luck! >> >> >> >> >> >>> >> >> >>> alias avrotoolx='java -jar >> >> >>> >> ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar' >> >> >>> avrotoolx tojson x.out 2> /dev/null >> >> >>> >> >> >>> (As Fokko mentioned, the 2> /dev/null isn't even necessary -- the >> >> >>> warnings and logs should not be piped along with the normal >> content.) >> >> >>> >> >> >>> Otherwise, IIRC, there is no way to disable the first illegal >> >> >>> reflective access warning when running in Java 9+, but you can >> "fix" >> >> >>> these module errors, and deactivate the NativeCodeLoader logs with >> an >> >> >>> explicit log4j.properties: >> >> >>> >> >> >>> java -Dlog4j.configuration=file:///tmp/log4j.properties --add-opens >> >> >>> java.security.jgss/sun.security.krb5=ALL-UNNAMED -jar >> >> >>> >> ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar >> >> >>> tojson x.out >> >> >> >> >> >> >> >> >> Thanks for that suggestion! I'm afraid I'm not familiar with log4j >> properties files though. What do I need to put in /tmp/log4j.properties to >> make this work? >> >> >> >> >> >>> None of that is particularly satisfactory, but it could be a >> >> >>> workaround for your immediate use. >> >> >> >> >> >> >> >> >> Yeah, not ideal, because if something goes wrong, stdout will be >> corrupted, but at least some noise should go away :) >> >> >> >> >> >>> I'd also like to see a more unified experience with the CLI tool >> for >> >> >>> documentation and usage. The current state requires a bit of Avro >> >> >>> expertise to use, but it has some functions that would be pretty >> >> >>> useful for a user working with Avro data. I raised >> >> >>> https://issues.apache.org/jira/browse/AVRO-2688 as an improvement. >> >> >>> >> >> >>> In my opinion, a schema compatibility tool would be a useful and >> >> >>> welcome feature! >> >> >> >> >> >> >> >> >> That would indeed be nice, but in the meantime, is there really >> nothing in the avro-tools commands that uses a chosen schema to read a data >> file written with some other schema? That would give me what I'm after >> currently. >> >> >> >> >> >> Thanks again for the helpful response. >> >> >> >> >> >> cheers, >> >> >> rog. >> >> >> >> >> >>> >> >> >>> Best regards, Ryan >> >> >>> >> >> >>> >> >> >>> >> >> >>> On Thu, Jan 16, 2020 at 12:25 PM roger peppe <rogpe...@gmail.com> >> wrote: >> >> >>> > >> >> >>> > Hi Fokko, >> >> >>> > >> >> >>> > Thanks for your swift response! >> >> >>> > >> >> >>> > Stdout and stderr definitely seem to be merged on this platform >> at least. Here's a sample: >> >> >>> > >> >> >>> > % avrotool random --count 1 --schema '"int"' x.out >> >> >>> > % avrotool tojson x.out > x.json >> >> >>> > % cat x.json >> >> >>> > 125140891 >> >> >>> > WARNING: An illegal reflective access operation has occurred >> >> >>> > WARNING: Illegal reflective access by >> org.apache.hadoop.security.authentication.util.KerberosUtil >> (file:/home/rog/other/avro-tools-1.9.1.jar) to method >> sun.security.krb5.Config.getInstance() >> >> >>> > WARNING: Please consider reporting this to the maintainers of >> org.apache.hadoop.security.authentication.util.KerberosUtil >> >> >>> > WARNING: Use --illegal-access=warn to enable warnings of further >> illegal reflective access operations >> >> >>> > WARNING: All illegal access operations will be denied in a >> future release >> >> >>> > 20/01/16 11:00:37 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> >> >>> > % >> >> >>> > >> >> >>> > I've just verified that it's not a problem with the java >> executable itself (I ran a program that printed to System.err and the text >> correctly goes to the standard error). >> >> >>> > >> >> >>> > > Regarding the documentation, the CLI itself contains info on >> all the available commands. Also, there are excellent online resources: >> https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ >> Is there anything specific that you're missing? >> >> >>> > >> >> >>> > There's the single line summary produced for each command by >> running "avro-tools" with no arguments, but that's not as much info as I'd >> ideally like. For example, it often doesn't say what file format is being >> written or read. For some commands, the purpose is not very clear. >> >> >>> > >> >> >>> > For example the description of the recodec command is "Alters >> the codec of a data file". It doesn't describe how it alters it or how one >> might configure the alteration parameters. I managed to get some usage help >> by passing it more than two parameters (specifying "--help" gives an >> exception), but that doesn't provide much more info: >> >> >>> > >> >> >>> > % avro-tools recodec a b c >> >> >>> > Expected at most an input file and output file. >> >> >>> > Option Description >> >> >>> > ------ ----------- >> >> >>> > --codec <String> Compression codec (default: null) >> >> >>> > --level <Integer> Compression level (only applies to deflate >> and xz) (default: >> >> >>> > -1) >> >> >>> > >> >> >>> > For the record, I'm wondering it might be possible to get >> avrotool to tell me if one schema is compatible with another so that I can >> check hypotheses about schema-checking in practice without having to write >> Java code. >> >> >>> > >> >> >>> > cheers, >> >> >>> > rog. >> >> >>> > >> >> >>> > >> >> >>> > On Thu, 16 Jan 2020 at 10:30, Driesprong, Fokko >> <fo...@driesprong.frl> wrote: >> >> >>> >> >> >> >>> >> Hi Rog, >> >> >>> >> >> >> >>> >> This is actually a warning produced by the Hadoop library, that >> we're using. Please note that htis isn't part of the stdout: >> >> >>> >> >> >> >>> >> $ find /tmp/tmp >> >> >>> >> /tmp/tmp >> >> >>> >> /tmp/tmp/._SUCCESS.crc >> >> >>> >> >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro >> >> >>> >> >> /tmp/tmp/.part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro.crc >> >> >>> >> /tmp/tmp/_SUCCESS >> >> >>> >> >> >> >>> >> $ avro-tools tojson >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro >> >> >>> >> 20/01/16 11:26:10 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> >> >>> >> {"line_of_text":{"string":"Hello"}} >> >> >>> >> {"line_of_text":{"string":"World"}} >> >> >>> >> >> >> >>> >> $ avro-tools tojson >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro > >> /tmp/tmp/data.json >> >> >>> >> 20/01/16 11:26:20 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> >> >>> >> >> >> >>> >> $ cat /tmp/tmp/data.json >> >> >>> >> {"line_of_text":{"string":"Hello"}} >> >> >>> >> {"line_of_text":{"string":"World"}} >> >> >>> >> >> >> >>> >> So when you pipe the data, it doesn't include the warnings. >> >> >>> >> >> >> >>> >> Regarding the documentation, the CLI itself contains info on >> all the available commands. Also, there are excellent online resources: >> https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ >> Is there anything specific that you're missing? >> >> >>> >> >> >> >>> >> Hope this helps. >> >> >>> >> >> >> >>> >> Cheers, Fokko >> >> >>> >> >> >> >>> >> Op do 16 jan. 2020 om 09:30 schreef roger peppe < >> rogpe...@gmail.com>: >> >> >>> >>> >> >> >>> >>> Hi, >> >> >>> >>> >> >> >>> >>> I've been trying to use avro-tools to verify Avro >> implementations, and I've come across an issue. Perhaps someone here might >> be able to help? >> >> >>> >>> >> >> >>> >>> When I run avro-tools with some subcommands, it prints a bunch >> of warnings (see below) to the standard output. Does anyone know a way to >> disable this? I'm using openjdk 11.0.5 under Ubuntu 18.04 and avro-tools >> 1.9.1. >> >> >>> >>> >> >> >>> >>> The warnings are somewhat annoying because they can corrupt >> output of tools that print to the standard output, such as recodec. >> >> >>> >>> >> >> >>> >>> Aside: is there any documentation for the commands in >> avro-tools? Some seem to have some command-line help (though unfortunately >> there doesn't seem to be a standard way of showing it), but often that help >> often doesn't describe what the command actually does. >> >> >>> >>> >> >> >>> >>> Here's the output that I see: >> >> >>> >>> >> >> >>> >>> WARNING: An illegal reflective access operation has occurred >> >> >>> >>> WARNING: Illegal reflective access by >> org.apache.hadoop.security.authentication.util.KerberosUtil >> (file:/home/rog/other/avro-tools-1.9.1.jar) to method >> sun.security.krb5.Config.getInstance() >> >> >>> >>> WARNING: Please consider reporting this to the maintainers of >> org.apache.hadoop.security.authentication.util.KerberosUtil >> >> >>> >>> WARNING: Use --illegal-access=warn to enable warnings of >> further illegal reflective access operations >> >> >>> >>> WARNING: All illegal access operations will be denied in a >> future release >> >> >>> >>> 20/01/16 08:12:39 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> >> >>> >>> >> >> >>> >>> cheers, >> >> >>> >>> rog. >> >> >>> >>> >> >
