I don't know much about these specific tests, but could we simplify things and get rid of the whole "Needs Runner" designation by just making the direct runner a dependency for the tests of each module (including core)?
On Fri, Dec 28, 2018 at 6:20 PM Gleb Kanterov <[email protected]> wrote: > > I looked into failing tests, we can put them into 5 categories: > - CastTest, invalid test case, fixed in apache/beam#7372 > - UsesMetricsPusher tests running with direct-runner, attempted to fix in > apache/beam#7374 > - hanging tests using test stream: > BoundedReadFromUnboundedSourceTest#testTimeBound and > SplittableDoFnTest#testLateData, needs more investigation > - test using Watch transform: WatchTest, FileIOTest, AvroIOTest. Watch > transform needs refactoring after apache/beam#6467 > - TFRecordIOTest, needs more investigation > > Not sure what would be the best way to proceed while tests are failing. > > On Fri, Dec 28, 2018 at 5:30 PM Reuven Lax <[email protected]> wrote: >> >> I think they run in postcommit. Seems like this was an oversight maybe. >> >> On Fri, Dec 28, 2018, 5:21 PM Ismaël Mejía <[email protected] wrote: >>> >>> Big +1 >>> >>> I am in awe that they are not active anymore, they used to be in the >>> good old maven days, when IDE integration worked and most JIRAs were >>> not about build or test issues. Sigh. >>> >>> On Fri, Dec 28, 2018 at 3:24 PM Reuven Lax <[email protected]> wrote: >>> > >>> > Kenn and I both noticed that some needsRunner tests time out, and we were >>> > both wondering why our PreCommit was still green. This tests are meant to >>> > be quick, and IMO should definitely be part of Java PreCommit. >>> > >>> > On Fri, Dec 28, 2018 at 2:59 PM Gleb Kanterov <[email protected]> wrote: >>> >> >>> >> After reading Beam Testing I had an impression that NeedsRunner tests >>> >> are executed as a part of Java PreCommit using Direct runner. However, >>> >> it doesn't seem to be the case. I've tried running these tests locally, >>> >> and few of them are failing or timeout. >>> >> >>> >> My proposal is adding ":beam-runners-direct-java:needsRunnerTests" to >>> >> "Run Java PreCommit" Jenkins job, or running them in parallel as a part >>> >> of a new task, not to increase build time. Running needsRunnerTests on >>> >> my laptop took 3 minutes, so I just lean towards adding them to >>> >> javaPreCommit. Thoughts? >>> >> >>> >> > Task :beam-runners-direct-java:needsRunnerTests >>> >> >>> >> org.apache.beam.runners.core.metrics.MetricsPusherTest > test FAILED >>> >> java.lang.AssertionError at MetricsPusherTest.java:71 >>> >> >>> >> org.apache.beam.sdk.io.TFRecordIOTest > testReadInvalidDataMask FAILED >>> >> java.lang.AssertionError >>> >> >>> >> org.apache.beam.sdk.io.TFRecordIOTest > testReadInvalidLengthMask FAILED >>> >> java.lang.AssertionError >>> >> >>> >> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests > >>> >> testContinuouslyWriteAndReadMultipleFilepatterns[0: true] FAILED >>> >> java.lang.IllegalArgumentException at AvroIOTest.java:557 >>> >> >>> >> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests > >>> >> testWriteThenReadJavaClass[0: true] FAILED >>> >> java.lang.AssertionError at AvroIOTest.java:306 >>> >> >>> >> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests > >>> >> testContinuouslyWriteAndReadMultipleFilepatterns[1: false] FAILED >>> >> java.lang.IllegalArgumentException at AvroIOTest.java:557 >>> >> >>> >> org.apache.beam.sdk.io.AvroIOTest$NeedsRunnerTests > >>> >> testWriteThenReadJavaClass[1: false] FAILED >>> >> java.lang.AssertionError at AvroIOTest.java:306 >>> >> >>> >> org.apache.beam.sdk.io.FileIOTest > testMatchWatchForNewFiles FAILED >>> >> java.lang.IllegalArgumentException at FileIOTest.java:192 >>> >> >>> >> org.apache.beam.sdk.io.TextIOReadTest$BasicIOTest > >>> >> testReadWatchForNewFiles FAILED >>> >> java.lang.IllegalArgumentException at TextIOReadTest.java:846 >>> >> >>> >> org.apache.beam.sdk.schemas.transforms.CastTest > testTypeNarrowFail >>> >> FAILED >>> >> java.lang.AssertionError >>> >> >>> >> org.apache.beam.sdk.schemas.transforms.CastTest > testWeakedNullableFail >>> >> FAILED >>> >> java.lang.AssertionError >>> >> >>> >> org.apache.beam.sdk.transforms.SplittableDoFnTest > testBoundedness >>> >> FAILED >>> >> org.apache.beam.sdk.testing.TestPipeline$PipelineRunMissingException >>> >> >>> >> org.apache.beam.sdk.transforms.SplittableDoFnTest > testLateData FAILED >>> >> org.junit.runners.model.TestTimedOutException at >>> >> SplittableDoFnTest.java:683 >>> >> >>> >> org.apache.beam.sdk.transforms.WatchTest > >>> >> testSinglePollMultipleInputsWithSideInput FAILED >>> >> java.lang.IllegalArgumentException at WatchTest.java:112 >>> >> >>> >> org.apache.beam.sdk.transforms.WatchTest > >>> >> testMultiplePollsWithKeyExtractor FAILED >>> >> java.lang.IllegalArgumentException at WatchTest.java:203 >>> >> >>> >> org.apache.beam.sdk.transforms.WatchTest > testSinglePollMultipleInputs >>> >> FAILED >>> >> java.lang.IllegalArgumentException at WatchTest.java:82 >>> >> >>> >> org.apache.beam.sdk.transforms.WatchTest > >>> >> testMultiplePollsWithTerminationDueToTerminationCondition FAILED >>> >> java.lang.IllegalArgumentException at WatchTest.java:150 >>> >> >>> >> org.apache.beam.sdk.transforms.WatchTest > >>> >> testMultiplePollsWithManyResults FAILED >>> >> java.lang.IllegalArgumentException at WatchTest.java:308 >>> >> >>> >> org.apache.beam.sdk.transforms.WatchTest > testSinglePollWithManyResults >>> >> FAILED >>> >> java.lang.IllegalArgumentException at WatchTest.java:258 >>> >> >>> >> org.apache.beam.sdk.transforms.WatchTest > >>> >> testMultiplePollsStopAfterTimeSinceNewOutput FAILED >>> >> java.lang.IllegalArgumentException at WatchTest.java:232 >>> >> >>> >> org.apache.beam.sdk.transforms.WatchTest > >>> >> testMultiplePollsWithTerminationBecauseOutputIsFinal FAILED >>> >> java.lang.IllegalArgumentException at WatchTest.java:150 >>> >> >>> >> 647 tests completed, 21 failed, 4 skipped > > > > -- > Cheers, > Gleb
