Hi Fokko, Thanks for your swift response!
Stdout and stderr definitely seem to be merged on this platform at least. Here's a sample: % avrotool random --count 1 --schema '"int"' x.out % avrotool tojson x.out > x.json % cat x.json 125140891 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/rog/other/avro-tools-1.9.1.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 20/01/16 11:00:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable % I've just verified that it's not a problem with the java executable itself (I ran a program that printed to System.err and the text correctly goes to the standard error). > Regarding the documentation, the CLI itself contains info on all the available commands. Also, there are excellent online resources: https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ Is there anything specific that you're missing? There's the single line summary produced for each command by running "avro-tools" with no arguments, but that's not as much info as I'd ideally like. For example, it often doesn't say what file format is being written or read. For some commands, the purpose is not very clear. For example the description of the recodec command is "Alters the codec of a data file". It doesn't describe how it alters it or how one might configure the alteration parameters. I managed to get some usage help by passing it more than two parameters (specifying "--help" gives an exception), but that doesn't provide much more info: % avro-tools recodec a b c Expected at most an input file and output file. Option Description ------ ----------- --codec <String> Compression codec (default: null) --level <Integer> Compression level (only applies to deflate and xz) (default: -1) For the record, I'm wondering it might be possible to get avrotool to tell me if one schema is compatible with another so that I can check hypotheses about schema-checking in practice without having to write Java code. cheers, rog. On Thu, 16 Jan 2020 at 10:30, Driesprong, Fokko <fo...@driesprong.frl> wrote: > Hi Rog, > > This is actually a warning produced by the Hadoop library, that we're > using. Please note that htis isn't part of the stdout: > > $ find /tmp/tmp > /tmp/tmp > /tmp/tmp/._SUCCESS.crc > /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro > /tmp/tmp/.part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro.crc > /tmp/tmp/_SUCCESS > > $ avro-tools tojson > /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro > 20/01/16 11:26:10 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > {"line_of_text":{"string":"Hello"}} > {"line_of_text":{"string":"World"}} > > $ avro-tools tojson > /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro > > /tmp/tmp/data.json > 20/01/16 11:26:20 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > > $ cat /tmp/tmp/data.json > {"line_of_text":{"string":"Hello"}} > {"line_of_text":{"string":"World"}} > > So when you pipe the data, it doesn't include the warnings. > > Regarding the documentation, the CLI itself contains info on all the > available commands. Also, there are excellent online resources: > https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/ > Is > there anything specific that you're missing? > > Hope this helps. > > Cheers, Fokko > > Op do 16 jan. 2020 om 09:30 schreef roger peppe <rogpe...@gmail.com>: > >> Hi, >> >> I've been trying to use avro-tools to verify Avro implementations, and >> I've come across an issue. Perhaps someone here might be able to help? >> >> When I run avro-tools with some subcommands, it prints a bunch of >> warnings (see below) to the standard output. Does anyone know a way to >> disable this? I'm using openjdk 11.0.5 under Ubuntu 18.04 and avro-tools >> 1.9.1. >> >> The warnings are somewhat annoying because they can corrupt output of >> tools that print to the standard output, such as recodec. >> >> Aside: is there any documentation for the commands in avro-tools? Some >> seem to have some command-line help (though unfortunately there doesn't >> seem to be a standard way of showing it), but often that help often doesn't >> describe what the command actually does. >> >> Here's the output that I see: >> >> WARNING: An illegal reflective access operation has occurred >> WARNING: Illegal reflective access by >> org.apache.hadoop.security.authentication.util.KerberosUtil >> (file:/home/rog/other/avro-tools-1.9.1.jar) to method >> sun.security.krb5.Config.getInstance() >> WARNING: Please consider reporting this to the maintainers of >> org.apache.hadoop.security.authentication.util.KerberosUtil >> WARNING: Use --illegal-access=warn to enable warnings of further illegal >> reflective access operations >> WARNING: All illegal access operations will be denied in a future release >> 20/01/16 08:12:39 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> >> cheers, >> rog. >> >>
