On Thu, 16 Jan 2020 at 13:57, Ryan Skraba <r...@skraba.com> wrote: > Hello! Is it because you are using brew to install avro-tools? I'm > not entirely familiar with how it packages the command, but using a > direct bash-like solution instead might solve this problem of mixing > stdout and stderr. This could be the simplest (and right) solution > for piping. >
No, I downloaded the jar and am directly running it with "java -jar ~/other/avro-tools-1.9.1.jar". I'm using Ubuntu Linux 18.04 FWIW - the binary comes from Debian package openjdk-11-jre-headless. I'm going to try compiling avro-tools myself to investigate but I'm a total Java ignoramus - wish me luck! > alias avrotoolx='java -jar > ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar' > avrotoolx tojson x.out 2> /dev/null > > (As Fokko mentioned, the 2> /dev/null isn't even necessary -- the > warnings and logs should not be piped along with the normal content.) > > Otherwise, IIRC, there is no way to disable the first illegal > reflective access warning when running in Java 9+, but you can "fix" > these module errors, and deactivate the NativeCodeLoader logs with an > explicit log4j.properties: > > java -Dlog4j.configuration=file:///tmp/log4j.properties --add-opens > java.security.jgss/sun.security.krb5=ALL-UNNAMED -jar > ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar > tojson x.out > Thanks for that suggestion! I'm afraid I'm not familiar with log4j properties files though. What do I need to put in /tmp/log4j.properties to make this work? None of that is particularly satisfactory, but it could be a > workaround for your immediate use. > Yeah, not ideal, because if something goes wrong, stdout will be corrupted, but at least some noise should go away :) I'd also like to see a more unified experience with the CLI tool for > documentation and usage. The current state requires a bit of Avro > expertise to use, but it has some functions that would be pretty > useful for a user working with Avro data. I raised > https://issues.apache.org/jira/browse/AVRO-2688 as an improvement. > > In my opinion, a schema compatibility tool would be a useful and > welcome feature! > That would indeed be nice, but in the meantime, is there really nothing in the avro-tools commands that uses a chosen schema to read a data file written with some other schema? That would give me what I'm after currently. Thanks again for the helpful response. cheers, rog. > Best regards, Ryan > > > > On Thu, Jan 16, 2020 at 12:25 PM roger peppe <rogpe...@gmail.com> wrote: > > > > Hi Fokko, > > > > Thanks for your swift response! > > > > Stdout and stderr definitely seem to be merged on this platform at > least. Here's a sample: > > > > % avrotool random --count 1 --schema '"int"' x.out > > % avrotool tojson x.out > x.json > > % cat x.json > > 125140891 > > WARNING: An illegal reflective access operation has occurred > > WARNING: Illegal reflective access by > org.apache.hadoop.security.authentication.util.KerberosUtil > (file:/home/rog/other/avro-tools-1.9.1.jar) to method > sun.security.krb5.Config.getInstance() > > WARNING: Please consider reporting this to the maintainers of > org.apache.hadoop.security.authentication.util.KerberosUtil > > WARNING: Use --illegal-access=warn to enable warnings of further illegal > reflective access operations > > WARNING: All illegal access operations will be denied in a future release > > 20/01/16 11:00:37 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > > % > > > > I've just verified that it's not a problem with the java executable > itself (I ran a program that printed to System.err and the text correctly > goes to the standard error). > > > > > Regarding the documentation, the CLI itself contains info on all the > available commands. Also, there are excellent online resources: > https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ > Is there anything specific that you're missing? > > > > There's the single line summary produced for each command by running > "avro-tools" with no arguments, but that's not as much info as I'd ideally > like. For example, it often doesn't say what file format is being written > or read. For some commands, the purpose is not very clear. > > > > For example the description of the recodec command is "Alters the codec > of a data file". It doesn't describe how it alters it or how one might > configure the alteration parameters. I managed to get some usage help by > passing it more than two parameters (specifying "--help" gives an > exception), but that doesn't provide much more info: > > > > % avro-tools recodec a b c > > Expected at most an input file and output file. > > Option Description > > ------ ----------- > > --codec <String> Compression codec (default: null) > > --level <Integer> Compression level (only applies to deflate and xz) > (default: > > -1) > > > > For the record, I'm wondering it might be possible to get avrotool to > tell me if one schema is compatible with another so that I can check > hypotheses about schema-checking in practice without having to write Java > code. > > > > cheers, > > rog. > > > > > > On Thu, 16 Jan 2020 at 10:30, Driesprong, Fokko <fo...@driesprong.frl> > wrote: > >> > >> Hi Rog, > >> > >> This is actually a warning produced by the Hadoop library, that we're > using. Please note that htis isn't part of the stdout: > >> > >> $ find /tmp/tmp > >> /tmp/tmp > >> /tmp/tmp/._SUCCESS.crc > >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro > >> /tmp/tmp/.part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro.crc > >> /tmp/tmp/_SUCCESS > >> > >> $ avro-tools tojson > /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro > >> 20/01/16 11:26:10 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > >> {"line_of_text":{"string":"Hello"}} > >> {"line_of_text":{"string":"World"}} > >> > >> $ avro-tools tojson > /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro > > /tmp/tmp/data.json > >> 20/01/16 11:26:20 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > >> > >> $ cat /tmp/tmp/data.json > >> {"line_of_text":{"string":"Hello"}} > >> {"line_of_text":{"string":"World"}} > >> > >> So when you pipe the data, it doesn't include the warnings. > >> > >> Regarding the documentation, the CLI itself contains info on all the > available commands. Also, there are excellent online resources: > https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ > Is there anything specific that you're missing? > >> > >> Hope this helps. > >> > >> Cheers, Fokko > >> > >> Op do 16 jan. 2020 om 09:30 schreef roger peppe <rogpe...@gmail.com>: > >>> > >>> Hi, > >>> > >>> I've been trying to use avro-tools to verify Avro implementations, and > I've come across an issue. Perhaps someone here might be able to help? > >>> > >>> When I run avro-tools with some subcommands, it prints a bunch of > warnings (see below) to the standard output. Does anyone know a way to > disable this? I'm using openjdk 11.0.5 under Ubuntu 18.04 and avro-tools > 1.9.1. > >>> > >>> The warnings are somewhat annoying because they can corrupt output of > tools that print to the standard output, such as recodec. > >>> > >>> Aside: is there any documentation for the commands in avro-tools? Some > seem to have some command-line help (though unfortunately there doesn't > seem to be a standard way of showing it), but often that help often doesn't > describe what the command actually does. > >>> > >>> Here's the output that I see: > >>> > >>> WARNING: An illegal reflective access operation has occurred > >>> WARNING: Illegal reflective access by > org.apache.hadoop.security.authentication.util.KerberosUtil > (file:/home/rog/other/avro-tools-1.9.1.jar) to method > sun.security.krb5.Config.getInstance() > >>> WARNING: Please consider reporting this to the maintainers of > org.apache.hadoop.security.authentication.util.KerberosUtil > >>> WARNING: Use --illegal-access=warn to enable warnings of further > illegal reflective access operations > >>> WARNING: All illegal access operations will be denied in a future > release > >>> 20/01/16 08:12:39 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > >>> > >>> cheers, > >>> rog. > >>> >
