Hello! For a simple, silent log4j, I use: $ cat /tmp/log4j.properties log4j.rootLogger=off
I didn't find anything currently in the avro-tools that uses both reader and writer schemas while deserializing data... It should be a pretty easy feature to add as an option to the DataFileReadTool (a.k.a. tojson)! You are correct about running ./build.sh dist in the java directory -- it fails with JDK 11 (likely fixable: https://issues.apache.org/jira/browse/MJAVADOC-562). You should probably do a simple mvn clean install instead and find the jar in lang/java/tools/target/avro-tools-1.10.0-SNAPSHOT.jar. That should work with JDK11 without any problem (well-tested in the build). Best regards, Ryan On Thu, Jan 16, 2020 at 5:49 PM roger peppe <rogpe...@gmail.com> wrote: > > Update: I tried running `build.sh dist` in `lang/java` and it failed (at > least, it looks like a failure message) after downloading a load of Maven > deps with the following errors: > https://gist.github.com/rogpeppe/df05d993254dc5082253a5ef5027e965 > > Any hints on what I should do to build the avro-tools jar? > > cheers, > rog. > > On Thu, 16 Jan 2020 at 16:45, roger peppe <rogpe...@gmail.com> wrote: >> >> >> On Thu, 16 Jan 2020 at 13:57, Ryan Skraba <r...@skraba.com> wrote: >>> >>> Hello! Is it because you are using brew to install avro-tools? I'm >>> not entirely familiar with how it packages the command, but using a >>> direct bash-like solution instead might solve this problem of mixing >>> stdout and stderr. This could be the simplest (and right) solution >>> for piping. >> >> >> No, I downloaded the jar and am directly running it with "java -jar >> ~/other/avro-tools-1.9.1.jar". >> I'm using Ubuntu Linux 18.04 FWIW - the binary comes from Debian package >> openjdk-11-jre-headless. >> >> I'm going to try compiling avro-tools myself to investigate but I'm a total >> Java ignoramus - wish me luck! >> >>> >>> alias avrotoolx='java -jar >>> ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar' >>> avrotoolx tojson x.out 2> /dev/null >>> >>> (As Fokko mentioned, the 2> /dev/null isn't even necessary -- the >>> warnings and logs should not be piped along with the normal content.) >>> >>> Otherwise, IIRC, there is no way to disable the first illegal >>> reflective access warning when running in Java 9+, but you can "fix" >>> these module errors, and deactivate the NativeCodeLoader logs with an >>> explicit log4j.properties: >>> >>> java -Dlog4j.configuration=file:///tmp/log4j.properties --add-opens >>> java.security.jgss/sun.security.krb5=ALL-UNNAMED -jar >>> ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar >>> tojson x.out >> >> >> Thanks for that suggestion! I'm afraid I'm not familiar with log4j >> properties files though. What do I need to put in /tmp/log4j.properties to >> make this work? >> >>> None of that is particularly satisfactory, but it could be a >>> workaround for your immediate use. >> >> >> Yeah, not ideal, because if something goes wrong, stdout will be corrupted, >> but at least some noise should go away :) >> >>> I'd also like to see a more unified experience with the CLI tool for >>> documentation and usage. The current state requires a bit of Avro >>> expertise to use, but it has some functions that would be pretty >>> useful for a user working with Avro data. I raised >>> https://issues.apache.org/jira/browse/AVRO-2688 as an improvement. >>> >>> In my opinion, a schema compatibility tool would be a useful and >>> welcome feature! >> >> >> That would indeed be nice, but in the meantime, is there really nothing in >> the avro-tools commands that uses a chosen schema to read a data file >> written with some other schema? That would give me what I'm after currently. >> >> Thanks again for the helpful response. >> >> cheers, >> rog. >> >>> >>> Best regards, Ryan >>> >>> >>> >>> On Thu, Jan 16, 2020 at 12:25 PM roger peppe <rogpe...@gmail.com> wrote: >>> > >>> > Hi Fokko, >>> > >>> > Thanks for your swift response! >>> > >>> > Stdout and stderr definitely seem to be merged on this platform at least. >>> > Here's a sample: >>> > >>> > % avrotool random --count 1 --schema '"int"' x.out >>> > % avrotool tojson x.out > x.json >>> > % cat x.json >>> > 125140891 >>> > WARNING: An illegal reflective access operation has occurred >>> > WARNING: Illegal reflective access by >>> > org.apache.hadoop.security.authentication.util.KerberosUtil >>> > (file:/home/rog/other/avro-tools-1.9.1.jar) to method >>> > sun.security.krb5.Config.getInstance() >>> > WARNING: Please consider reporting this to the maintainers of >>> > org.apache.hadoop.security.authentication.util.KerberosUtil >>> > WARNING: Use --illegal-access=warn to enable warnings of further illegal >>> > reflective access operations >>> > WARNING: All illegal access operations will be denied in a future release >>> > 20/01/16 11:00:37 WARN util.NativeCodeLoader: Unable to load >>> > native-hadoop library for your platform... using builtin-java classes >>> > where applicable >>> > % >>> > >>> > I've just verified that it's not a problem with the java executable >>> > itself (I ran a program that printed to System.err and the text correctly >>> > goes to the standard error). >>> > >>> > > Regarding the documentation, the CLI itself contains info on all the >>> > > available commands. Also, there are excellent online resources: >>> > > https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ >>> > > Is there anything specific that you're missing? >>> > >>> > There's the single line summary produced for each command by running >>> > "avro-tools" with no arguments, but that's not as much info as I'd >>> > ideally like. For example, it often doesn't say what file format is being >>> > written or read. For some commands, the purpose is not very clear. >>> > >>> > For example the description of the recodec command is "Alters the codec >>> > of a data file". It doesn't describe how it alters it or how one might >>> > configure the alteration parameters. I managed to get some usage help by >>> > passing it more than two parameters (specifying "--help" gives an >>> > exception), but that doesn't provide much more info: >>> > >>> > % avro-tools recodec a b c >>> > Expected at most an input file and output file. >>> > Option Description >>> > ------ ----------- >>> > --codec <String> Compression codec (default: null) >>> > --level <Integer> Compression level (only applies to deflate and xz) >>> > (default: >>> > -1) >>> > >>> > For the record, I'm wondering it might be possible to get avrotool to >>> > tell me if one schema is compatible with another so that I can check >>> > hypotheses about schema-checking in practice without having to write Java >>> > code. >>> > >>> > cheers, >>> > rog. >>> > >>> > >>> > On Thu, 16 Jan 2020 at 10:30, Driesprong, Fokko <fo...@driesprong.frl> >>> > wrote: >>> >> >>> >> Hi Rog, >>> >> >>> >> This is actually a warning produced by the Hadoop library, that we're >>> >> using. Please note that htis isn't part of the stdout: >>> >> >>> >> $ find /tmp/tmp >>> >> /tmp/tmp >>> >> /tmp/tmp/._SUCCESS.crc >>> >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro >>> >> /tmp/tmp/.part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro.crc >>> >> /tmp/tmp/_SUCCESS >>> >> >>> >> $ avro-tools tojson >>> >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro >>> >> 20/01/16 11:26:10 WARN util.NativeCodeLoader: Unable to load >>> >> native-hadoop library for your platform... using builtin-java classes >>> >> where applicable >>> >> {"line_of_text":{"string":"Hello"}} >>> >> {"line_of_text":{"string":"World"}} >>> >> >>> >> $ avro-tools tojson >>> >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro > >>> >> /tmp/tmp/data.json >>> >> 20/01/16 11:26:20 WARN util.NativeCodeLoader: Unable to load >>> >> native-hadoop library for your platform... using builtin-java classes >>> >> where applicable >>> >> >>> >> $ cat /tmp/tmp/data.json >>> >> {"line_of_text":{"string":"Hello"}} >>> >> {"line_of_text":{"string":"World"}} >>> >> >>> >> So when you pipe the data, it doesn't include the warnings. >>> >> >>> >> Regarding the documentation, the CLI itself contains info on all the >>> >> available commands. Also, there are excellent online resources: >>> >> https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ >>> >> Is there anything specific that you're missing? >>> >> >>> >> Hope this helps. >>> >> >>> >> Cheers, Fokko >>> >> >>> >> Op do 16 jan. 2020 om 09:30 schreef roger peppe <rogpe...@gmail.com>: >>> >>> >>> >>> Hi, >>> >>> >>> >>> I've been trying to use avro-tools to verify Avro implementations, and >>> >>> I've come across an issue. Perhaps someone here might be able to help? >>> >>> >>> >>> When I run avro-tools with some subcommands, it prints a bunch of >>> >>> warnings (see below) to the standard output. Does anyone know a way to >>> >>> disable this? I'm using openjdk 11.0.5 under Ubuntu 18.04 and >>> >>> avro-tools 1.9.1. >>> >>> >>> >>> The warnings are somewhat annoying because they can corrupt output of >>> >>> tools that print to the standard output, such as recodec. >>> >>> >>> >>> Aside: is there any documentation for the commands in avro-tools? Some >>> >>> seem to have some command-line help (though unfortunately there doesn't >>> >>> seem to be a standard way of showing it), but often that help often >>> >>> doesn't describe what the command actually does. >>> >>> >>> >>> Here's the output that I see: >>> >>> >>> >>> WARNING: An illegal reflective access operation has occurred >>> >>> WARNING: Illegal reflective access by >>> >>> org.apache.hadoop.security.authentication.util.KerberosUtil >>> >>> (file:/home/rog/other/avro-tools-1.9.1.jar) to method >>> >>> sun.security.krb5.Config.getInstance() >>> >>> WARNING: Please consider reporting this to the maintainers of >>> >>> org.apache.hadoop.security.authentication.util.KerberosUtil >>> >>> WARNING: Use --illegal-access=warn to enable warnings of further >>> >>> illegal reflective access operations >>> >>> WARNING: All illegal access operations will be denied in a future >>> >>> release >>> >>> 20/01/16 08:12:39 WARN util.NativeCodeLoader: Unable to load >>> >>> native-hadoop library for your platform... using builtin-java classes >>> >>> where applicable >>> >>> >>> >>> cheers, >>> >>> rog. >>> >>>
