+1. I strongly vote for freezing commits and getting our testing coverage in acceptable state. We have been struggling to stabilize branch-3 due to test failures and releasing Hive 3.0 in current state would be unacceptable.
Currently there are quite a few test suites which are not even running and are being timed out. We have been committing patches (to both branch-3 and master) without test coverage for these tests. We should immediately figure out what’s going on before we proceed with commits. For reference following test suites are timing out on master: (https://issues.apache.org/jira/browse/HIVE-19506) TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed out) TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed out) TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed out) TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed out) TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) Vineet On May 11, 2018, at 1:46 PM, Vihang Karajgaonkar <vih...@cloudera.com<mailto:vih...@cloudera.com>> wrote: +1 There are many problems with the test infrastructure and in my opinion it has not become number one bottleneck for the project. I was looking at the infrastructure yesterday and I think the current infrastructure (even its own set of problems) is still under-utilized. I am planning to increase the number of threads to process the parallel test batches to start with. It needs a restart on the server side. I can do it now, it folks are okay with it. Else I can do it over weekend when the queue is small. I listed the improvements which I thought would be useful under https://issues.apache.org/jira/browse/HIVE-19425 but frankly speaking I am not able to devote as much time as I would like to on it. I would appreciate if folks who have some more time if they can help out. I think to start with https://issues.apache.org/jira/browse/HIVE-19429 will help a lot. We need to pack more test runs in parallel and containers provide good isolation. For the short term green runs, I think we should @Ignore the tests which are known to be failing since many runs. They are anyways not being addressed as such. If people think they are important to be run we should fix them and only then re-enable them. Also, I feel we need light-weight test run which we can run locally before submitting it for the full-suite. That way minor issues with the patch can be handled locally. May be create a profile which runs a subset of important tests which are consistent. We can apply some label that pre-checkin-local tests are runs successful and only then we submit for the full-suite. More thoughts are welcome. Thanks for starting this conversation. On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez < jcama...@apache.org<mailto:jcama...@apache.org>> wrote: I believe we have reached a state (maybe we did reach it a while ago) that is not sustainable anymore, as there are so many tests failing / timing out that it is not possible to verify whether a patch is breaking some critical parts of the system or not. It also seems to me that due to the timeouts (maybe due to infra, maybe not), ptest runs are taking even longer than usual, which in turn creates even longer queue of patches. There is an ongoing effort to improve ptests usability ( https://issues.apache.org/jira/browse/HIVE-19425), but apart from that, we need to make an effort to stabilize existing tests and bring that failure count to zero. Hence, I am suggesting *we stop committing any patch before we get a green run*. If someone thinks this proposal is too radical, please come up with an alternative, because I do not think it is OK to have the ptest runs in their current state. Other projects of certain size (e.g., Hadoop, Spark) are always green, we should be able to do the same. Finally, once we get to zero failures, I suggest we are less tolerant with committing without getting a clean ptests run. If there is a failure, we need to fix it or revert the patch that caused it, then we continue developing. Please, let’s all work together as a community to fix this issue, that is the only way to get to zero quickly. Thanks, Jesús PS. I assume the flaky tests will come into the discussion. Let´s see first how many of those we have, then we can work to find a fix.
