On Thu, Feb 8, 2018 at 5:47 PM, 郑锴(铁杰) <zhengkai...@alibaba-inc.com> wrote: >>>I'm looking at you, TestDFSStripedOutputStreamWithFailure ... > AFAIK and IMO, it's pretty hard to get all the test cases stably running > given the limitation of MiniDFSCluster, and if we'd agree on that, we could > remove these cases as unit tests and cover them in integration tests instead > using a true cluster, like based on k8s infra.
Since its inception, the Mini*Clusters have made it much easier to write unit tests (good!), but these tests take more time and resources than fine-grained tests (unfortunate) and they're often unreliable (bad). As we see in issues like HDFS-12711, the resource problem seems to harm reliability across the entire suite of tests (dire). Running integration tests with our CI is helpful, but perhaps we should separate the Mini*Cluster tests and/or press contributors to write lighter unit tests for new functionality. -C > We're lacking basic facility > infra env and tools to get most of the complicated functionalities well > tested and covered, so let's avoid too much complicated tests. Fixing of > such tests should definitely help and be appreciated. > > Regards, > Kai > > ------------------------------------------------------------------ > 发件人：Chris Douglas <cdoug...@apache.org> > 发送时间：2018年2月8日(星期四) 08:39 > 收件人：Hdfs-dev <hdfs-dev@hadoop.apache.org> > 主 题：Re: [DISCUSS] Meetup for HDFS tests and build infra > > Created a poll [1] to inform scheduling. -C > > [1]: https://doodle.com/poll/r22znitzae9apfbf > > On Tue, Feb 6, 2018 at 3:09 PM, Chris Douglas <cdoug...@apache.org> wrote: >> The HDFS build is not healthy. Many of the unit tests aren't actually >> run in Jenkins due to resource exhaustion, haven't been updated since >> build/test/data was the test temp dir, or are chronically unstable >> (I'm looking at you, TestDFSStripedOutputStreamWithFailure). The >> situation has deteriorated slowly, but we can't confidently merge >> patches, let alone significant features, when our CI infra is in this >> state. >> >> How would folks feel about a half to full-day meetup to work through >> patches improving this, specifically? We can improve tests, >> troubleshoot the build, and rev/commit existing patches. It would >> require some preparation, so the simultaneous attention is productive >> and not a coordination bottleneck. I started a wiki page for this [1], >> please add to it. >> >> If enough people can make time for this, say in 2-3 weeks, the project >> would certainly benefit. -C >> >> [1]: https://s.apache.org/ng3C > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
