On unixish systems it probably depends on the locale, as in LANG and LC_COLLATE. In my experience, the least surprising behavior comes with LANG=C, except when you're dealing with file names containing a lot of non-ascii text.
On Thu, Nov 28, 2019 at 04:29 Ryan Skraba <r...@skraba.com> wrote: > Effectively, the schemas are added in the order that the file system > lists files: > https://github.com/apache/avro/blob/f310ac8db5ab962a49d448f41b7b953488cdb033/lang/java/tools/src/main/java/org/apache/avro/tool/SpecificCompilerTool.java#L149 > > As you observed, this depends on the operating system and/or > filesystem... I've experienced this in the past (with an unrelated > tool that generated a classpath from a list of JARS, and seeing an > unreliable order on Windows vs. linux). > > Just reading the code, it should be deterministic if you explicitly > list the avsc files (or at least the "problem" file) with the > required order: > > java -jar avro-tools-1.9.1.jar compile schemas/Component.avsc > /schemas/Parent.avsc out-dir/ > > or > > java -jar avro-tools-1.9.1.jar compile schemas/Component.avsc schemas/ > out-dir/ > > Would it be possible to give this workaround a try? > > I took a quick look at the avro-maven-plugin; it doesn't use > listFiles() directly to discover files, but uses FileSetManager from > the maven project. I'm hoping they've taken this into account! > > Thanks for the well-described, well-defined email! It would make an > excellent bug report :D https://issues.apache.org/jira/browse/AVRO > > Ryan > > > On Thu, Nov 28, 2019 at 12:05 AM Austin Cawley-Edwards > <austin.caw...@gmail.com> wrote: > > > > Hi, > > > > We're trying to use the `compile {src dir} {output dir}` command in > > `avro-tools` and finding that there are some non-deterministic > > behaviors between systems, depending on how the OS sorts files. > > > > Example: > > schemas/Component.avsc > > - defines Component record type in the namespace `com.test` > > > > schemas/Parent.avsc > > - defines a Parent record, in the same `com.test` namespace, with a > > field of type `com.test.Component` > > > > > > With the command, `java -jar avro-tools-1.9.1.jar compile schemas/ > > out-dir/`, some systems compile the directory in the order Component, > > Parent while others compile in the order Parent, Component. The latter > > fails as Component has not been defined when it is referenced by > > Parent. > > > > We have also tried using the IDL and importing the dependency types, > > and then converting them to avsc, and finally compiling the entire > > directory, but that fails as the generated avsc files embed/ duplicate > > the "Component" types each time it is used. > > > > > > Is there a way to deterministically compile a directory? Or compile > > directly from IDL to java? > > > > > > OS: > > Linux 857aaf92e059 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 > > 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux > > > > Avro: > > version 1.9.1 > > > > > > > > Thank you! > > Austin Cawley-Edwards >
