Update: I tried running `build.sh dist` in `lang/java` and it failed (at
least, it looks like a failure message) after downloading a load of Maven
deps with the following errors:
https://gist.github.com/rogpeppe/df05d993254dc5082253a5ef5027e965
Any hints on what I should do to build the avro-tools jar?
cheers,
rog.
On Thu, 16 Jan 2020 at 16:45, roger peppe <rogpe...@gmail.com> wrote:
>
> On Thu, 16 Jan 2020 at 13:57, Ryan Skraba <r...@skraba.com> wrote:
>
>> Hello! Is it because you are using brew to install avro-tools? I'm
>> not entirely familiar with how it packages the command, but using a
>> direct bash-like solution instead might solve this problem of mixing
>> stdout and stderr. This could be the simplest (and right) solution
>> for piping.
>>
>
> No, I downloaded the jar and am directly running it with "java -jar
> ~/other/avro-tools-1.9.1.jar".
> I'm using Ubuntu Linux 18.04 FWIW - the binary comes from Debian
> package openjdk-11-jre-headless.
>
> I'm going to try compiling avro-tools myself to investigate but I'm a
> total Java ignoramus - wish me luck!
>
>
>> alias avrotoolx='java -jar
>> ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar'
>> avrotoolx tojson x.out 2> /dev/null
>>
>> (As Fokko mentioned, the 2> /dev/null isn't even necessary -- the
>> warnings and logs should not be piped along with the normal content.)
>>
>> Otherwise, IIRC, there is no way to disable the first illegal
>> reflective access warning when running in Java 9+, but you can "fix"
>> these module errors, and deactivate the NativeCodeLoader logs with an
>> explicit log4j.properties:
>>
>> java -Dlog4j.configuration=file:///tmp/log4j.properties --add-opens
>> java.security.jgss/sun.security.krb5=ALL-UNNAMED -jar
>> ~/.m2/repository/org/apache/avro/avro-tools/1.9.1/avro-tools-1.9.1.jar
>> tojson x.out
>>
>
> Thanks for that suggestion! I'm afraid I'm not familiar with log4j
> properties files though. What do I need to put in /tmp/log4j.properties to
> make this work?
>
> None of that is particularly satisfactory, but it could be a
>> workaround for your immediate use.
>>
>
> Yeah, not ideal, because if something goes wrong, stdout will be
> corrupted, but at least some noise should go away :)
>
> I'd also like to see a more unified experience with the CLI tool for
>> documentation and usage. The current state requires a bit of Avro
>> expertise to use, but it has some functions that would be pretty
>> useful for a user working with Avro data. I raised
>> https://issues.apache.org/jira/browse/AVRO-2688 as an improvement.
>>
>> In my opinion, a schema compatibility tool would be a useful and
>> welcome feature!
>>
>
> That would indeed be nice, but in the meantime, is there really nothing in
> the avro-tools commands that uses a chosen schema to read a data file
> written with some other schema? That would give me what I'm after currently.
>
> Thanks again for the helpful response.
>
> cheers,
> rog.
>
>
>> Best regards, Ryan
>>
>>
>>
>> On Thu, Jan 16, 2020 at 12:25 PM roger peppe <rogpe...@gmail.com> wrote:
>> >
>> > Hi Fokko,
>> >
>> > Thanks for your swift response!
>> >
>> > Stdout and stderr definitely seem to be merged on this platform at
>> least. Here's a sample:
>> >
>> > % avrotool random --count 1 --schema '"int"' x.out
>> > % avrotool tojson x.out > x.json
>> > % cat x.json
>> > 125140891
>> > WARNING: An illegal reflective access operation has occurred
>> > WARNING: Illegal reflective access by
>> org.apache.hadoop.security.authentication.util.KerberosUtil
>> (file:/home/rog/other/avro-tools-1.9.1.jar) to method
>> sun.security.krb5.Config.getInstance()
>> > WARNING: Please consider reporting this to the maintainers of
>> org.apache.hadoop.security.authentication.util.KerberosUtil
>> > WARNING: Use --illegal-access=warn to enable warnings of further
>> illegal reflective access operations
>> > WARNING: All illegal access operations will be denied in a future
>> release
>> > 20/01/16 11:00:37 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> > %
>> >
>> > I've just verified that it's not a problem with the java executable
>> itself (I ran a program that printed to System.err and the text correctly
>> goes to the standard error).
>> >
>> > > Regarding the documentation, the CLI itself contains info on all the
>> available commands. Also, there are excellent online resources:
>> https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/
>> Is there anything specific that you're missing?
>> >
>> > There's the single line summary produced for each command by running
>> "avro-tools" with no arguments, but that's not as much info as I'd ideally
>> like. For example, it often doesn't say what file format is being written
>> or read. For some commands, the purpose is not very clear.
>> >
>> > For example the description of the recodec command is "Alters the codec
>> of a data file". It doesn't describe how it alters it or how one might
>> configure the alteration parameters. I managed to get some usage help by
>> passing it more than two parameters (specifying "--help" gives an
>> exception), but that doesn't provide much more info:
>> >
>> > % avro-tools recodec a b c
>> > Expected at most an input file and output file.
>> > Option Description
>> > ------ -----------
>> > --codec <String> Compression codec (default: null)
>> > --level <Integer> Compression level (only applies to deflate and xz)
>> (default:
>> > -1)
>> >
>> > For the record, I'm wondering it might be possible to get avrotool to
>> tell me if one schema is compatible with another so that I can check
>> hypotheses about schema-checking in practice without having to write Java
>> code.
>> >
>> > cheers,
>> > rog.
>> >
>> >
>> > On Thu, 16 Jan 2020 at 10:30, Driesprong, Fokko <fo...@driesprong.frl>
>> wrote:
>> >>
>> >> Hi Rog,
>> >>
>> >> This is actually a warning produced by the Hadoop library, that we're
>> using. Please note that htis isn't part of the stdout:
>> >>
>> >> $ find /tmp/tmp
>> >> /tmp/tmp
>> >> /tmp/tmp/._SUCCESS.crc
>> >> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro
>> >> /tmp/tmp/.part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro.crc
>> >> /tmp/tmp/_SUCCESS
>> >>
>> >> $ avro-tools tojson
>> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro
>> >> 20/01/16 11:26:10 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> >> {"line_of_text":{"string":"Hello"}}
>> >> {"line_of_text":{"string":"World"}}
>> >>
>> >> $ avro-tools tojson
>> /tmp/tmp/part-00000-9300fba6-ccdd-4ecc-97cb-0c3ae3631be5-c000.avro >
>> /tmp/tmp/data.json
>> >> 20/01/16 11:26:20 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> >>
>> >> $ cat /tmp/tmp/data.json
>> >> {"line_of_text":{"string":"Hello"}}
>> >> {"line_of_text":{"string":"World"}}
>> >>
>> >> So when you pipe the data, it doesn't include the warnings.
>> >>
>> >> Regarding the documentation, the CLI itself contains info on all the
>> available commands. Also, there are excellent online resources:
>> https://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/
>> Is there anything specific that you're missing?
>> >>
>> >> Hope this helps.
>> >>
>> >> Cheers, Fokko
>> >>
>> >> Op do 16 jan. 2020 om 09:30 schreef roger peppe <rogpe...@gmail.com>:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I've been trying to use avro-tools to verify Avro implementations,
>> and I've come across an issue. Perhaps someone here might be able to help?
>> >>>
>> >>> When I run avro-tools with some subcommands, it prints a bunch of
>> warnings (see below) to the standard output. Does anyone know a way to
>> disable this? I'm using openjdk 11.0.5 under Ubuntu 18.04 and avro-tools
>> 1.9.1.
>> >>>
>> >>> The warnings are somewhat annoying because they can corrupt output of
>> tools that print to the standard output, such as recodec.
>> >>>
>> >>> Aside: is there any documentation for the commands in avro-tools?
>> Some seem to have some command-line help (though unfortunately there
>> doesn't seem to be a standard way of showing it), but often that help often
>> doesn't describe what the command actually does.
>> >>>
>> >>> Here's the output that I see:
>> >>>
>> >>> WARNING: An illegal reflective access operation has occurred
>> >>> WARNING: Illegal reflective access by
>> org.apache.hadoop.security.authentication.util.KerberosUtil
>> (file:/home/rog/other/avro-tools-1.9.1.jar) to method
>> sun.security.krb5.Config.getInstance()
>> >>> WARNING: Please consider reporting this to the maintainers of
>> org.apache.hadoop.security.authentication.util.KerberosUtil
>> >>> WARNING: Use --illegal-access=warn to enable warnings of further
>> illegal reflective access operations
>> >>> WARNING: All illegal access operations will be denied in a future
>> release
>> >>> 20/01/16 08:12:39 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> >>>
>> >>> cheers,
>> >>> rog.
>> >>>
>>
>
