Good to know, Reuven! I just hit it again with `--console=verbose`, and have more info: https://gist.github.com/ryan-williams/252597d9ecd6973378d5cc2e9dabceb1
It seems that the generated files are correctly in place when the failing compileJava is invoked. Perhaps something is wrong (and nondeterministic) with the classpath of the `:beam-sdks-java-extensions-sql:compileJava` task? I'll see if I can run with more verbose/DEBUG logging and get any more info. On Sun, Feb 17, 2019 at 1:42 PM Reuven Lax <[email protected]> wrote: > I also see this almost every other run. > > On Sun, Feb 17, 2019 at 8:24 AM Ryan Williams <[email protected]> > wrote: > >> I still see this about 20% of the time I run `./gradlew compileTestJava >> spotlessApply checkstyleMain checkstyleTest` (with 8-way parallelism), >> which I've taken to running as a kind of "pre-PreCommit" (for Java). >> >> In total I'd estimate I've seen this error 20 times (across multiple >> `./gradlew clean`s and spanning several weeks), but never twice in a row: I >> always immediately re-run, and the error goes away. >> >> That seems consistent with the idea that gradle is racing the codegen >> task against the compile task that requires the codegen's outputs. >> >> However, `./gradlew -p sdks/java/extensions/sql -m compileJava | grep >> compileJavacc` implies that Gradle knows that compilation depends on the >> codegen having run. >> >> Here are three successive invocations of `./gradlew compileTestJava >> spotlessApply checkstyleMain checkstyleTest` >> <https://gist.github.com/ryan-williams/6d2735b2484e9459be656abf379533c5>; >> >> - The first one succeeds (and includes output showing that the >> codegen task, `:beam-sdks-java-extensions-sql:compileJavacc`, was run). >> - Then, I rebased on one new upstream commit and ran again, and saw >> the failure. >> - Immediately re-running saw the failure go away. >> >> I guess in that output, tasks that are run but produce no stdout/stderr >> are elided from the output, so we can't completely tell when the codegen >> and compile tasks in question are being run. I'll start running with >> --console=verbose >> <https://stackoverflow.com/questions/45883963/gradle-4-0-does-not-display-executed-tasks-in-command-line#comment92914258_45889966>, >> which I think will give a better sense of how this is happening. >> >> >> On Fri, Feb 8, 2019 at 12:41 AM Ryan Williams <[email protected]> >> wrote: >> >>> After your last message, Alex, I saw the issue on a branch with minimal >>> unrelated changes on top of b4b5495307 >>> <https://github.com/apache/beam/commit/b4b549530730d9e0283ad0cdb203d384a107dfbf> >>> (January >>> 28). >>> >>> I ran `./gradlew clean` for the first time in a while at that time, and >>> worked happily for 11 days, but just saw the issue again on a branch >>> rebased on top of 381ab55b59 >>> <https://github.com/apache/beam/commit/381ab55b59> (today): >>> >>> > Task :beam-sdks-java-extensions-sql:compileJava FAILED >>> …/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29: >>> error: cannot find symbol >>> import >>> org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl; >>> ^ >>> symbol: class BeamSqlParserImpl >>> location: package org.apache.beam.sdk.extensions.sql.impl.parser.impl >>> 1 error >>> >>> Every time I've seen it, I immediately re-run the compile, and it >>> succeeds. >>> >>> Perhaps something is still wrong with my environment, but otherwise it >>> would seem that something is still flaky here. >>> >>> I'm compiling on an 8-core macOS machine, fwiw, and usually running >>> `./gradlew compileTestJava` which compiles many projects concurrently. >>> >>> On Mon, Jan 28, 2019 at 4:12 PM Alex Amato <[email protected]> wrote: >>> >>>> If it continues to occur, maybe it is an environmental issue, be sure >>>> to try to clean as well. >>>> ./gradlew clean >>>> >>>> On Mon, Jan 28, 2019 at 11:12 AM Ryan Williams <[email protected]> >>>> wrote: >>>> >>>>> Yea I was rebased on top of a more recent master than your previous >>>>> message, when I saw it again. Perhaps I was mistaken. I'll ping here if I >>>>> see it again, thanks. >>>>> >>>>> On Mon, Jan 28, 2019 at 1:52 PM Alex Amato <[email protected]> wrote: >>>>> >>>>>> After I did a rebase, it went away for me. So I think that this >>>>>> should work. Are you saying that you did rebase ontop of master and it >>>>>> still occurred? Strange. >>>>>> >>>>>> On Sat, Jan 26, 2019 at 3:48 PM Ryan Williams <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hm, I just encountered this again on a branch that based on >>>>>>> 5b46b02b49 (top of trunk from this afternoon). Is it definitely >>>>>>> supposed to >>>>>>> be fixed? >>>>>>> >>>>>>> >>>>>>> On Thu, Jan 24, 2019 at 9:19 PM Alex Amato <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Please try rebasing from master, I believe this issue has been >>>>>>>> resolved. >>>>>>>> >>>>>>>> On Thu, Jan 24, 2019 at 3:29 PM Ryan Williams <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I'm seeing every ≈third `./gradlew compileJava` fail locally due >>>>>>>>> to this; re-running the commit has always succeeded, so far. >>>>>>>>> >>>>>>>>> Sounds like there is not an immediate fix in the works / no one >>>>>>>>> assigned on the JIRA? >>>>>>>>> >>>>>>>>> On Wed, Jan 23, 2019 at 3:17 PM Kenneth Knowles <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> This might connect to vendoring Calcite. It will be easiest, and >>>>>>>>>> have the best incremental build, if we separate the generated code >>>>>>>>>> into its >>>>>>>>>> own module that has relocation to match the vendored Calcite. >>>>>>>>>> >>>>>>>>>> Kenn >>>>>>>>>> >>>>>>>>>> On Wed, Jan 23, 2019 at 11:29 AM Anton Kedin <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> We don't pre-generate the code as a separate step. Code gen from >>>>>>>>>>> the SQL parser syntax spec and its compilation happens both during >>>>>>>>>>> the Beam >>>>>>>>>>> SQL build task. Splitting the code generation and compilation might >>>>>>>>>>> not be >>>>>>>>>>> trivial. We definitely should look into fixing this though. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Anton >>>>>>>>>>> >>>>>>>>>>> On Wed, Jan 23, 2019 at 11:13 AM Alex Amato <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Okay, make sense perhaps we can somehow make it fail when it >>>>>>>>>>>> fails to generate the dep, rather than when compiling the java >>>>>>>>>>>> code later on >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jan 23, 2019 at 11:12 AM Anton Kedin <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> ParserImpl is autogenerated by Calcite at build time. It seems >>>>>>>>>>>>> that there's a race condition there and it sometimes fails. >>>>>>>>>>>>> Rerunning the >>>>>>>>>>>>> build works for me. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Anton >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jan 23, 2019, 11:06 AM Alex Amato <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> https://jira.apache.org/jira/browse/BEAM-6495?filter=-2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any ideas, how this got through the precommit? >>>>>>>>>>>>>> >>>>>>>>>>>>>> > Task :beam-sdks-java-extensions-sql:compileJava FAILED >>>>>>>>>>>>>> >>>>>>>>>>>>>> /usr/local/google/home/ajamato/go/src/ >>>>>>>>>>>>>> github.com/apache/beam/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29: >>>>>>>>>>>>>> error: cannot find symbol >>>>>>>>>>>>>> >>>>>>>>>>>>>> import >>>>>>>>>>>>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl; >>>>>>>>>>>>>> >>>>>>>>>>>>>> ^ >>>>>>>>>>>>>> >>>>>>>>>>>>>> symbol: class BeamSqlParserImpl >>>>>>>>>>>>>> >>>>>>>>>>>>>> location: package >>>>>>>>>>>>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1 error >>>>>>>>>>>>>> >>>>>>>>>>>>>>
