Hi Clement. The problem we are trying to solve by this testing tool is: Running all the exists HTML test cases[1] automatically and compare current developing version to the previous stable release to test if any function is broken. It's more like regression testing.
Jest and jest-image-snapshot is my first option. But jest-image-snap is designed to compare with the pre-generated baseline snapshots. For us, there are some problems: 1. Not like normal tests, the pre-generated snapshots are depend on puppeteer versions, operating systems, or even hardwares. For example the default system font on macOS and windows are different. These are all unnecessary variables in a testing system, which we should eliminate. 2. Baseline snapshots for nearly 400 tests may have a huge size. 3. There is also a small problem, we need to update the baseline snapshots when the test cases are changed. There are some workarounds on this issue, like developers generate his own baseline snapshots, instead of uploading the baseline snapshots. But I think a better solution for us is picking out the underlying pixelmatch[2] and rewrite the testing module, which will do: 1. Download the latest stable ECharts release. 2. Run test cases with both stable release and developing version to get snapshots, in a consistent puppeeter, OS, and hardware. 3. Compare the snapshots and check if there is any diff. At last, here are some downsides about this testing tool: 1. It can't test new add functions. But it's mainly for regression testing so I think it's not a problem. 2. Running each test case twice may be time time consuming. [1] https://github.com/apache/incubator-echarts/tree/master/test [2] https://github.com/mapbox/pixelmatch On Thu, Sep 5, 2019 at 8:48 AM Clement Ho <clem...@gitlab.com.invalid> wrote: > Curious as to the decision not to use something like jest image snapshots. > Could you share your thought process? > > On Tue, Sep 3, 2019 at 7:23 AM Yi Shen <shenyi....@gmail.com> wrote: > > > Thanks Ovilia. > > > > The progress and todo is in the pull request description. The next big > > thing is supporting interaction testing in the tool. > > > > On Tue, Sep 3, 2019 at 8:17 PM Ovilia <oviliazh...@gmail.com> wrote: > > > > > Hi Shen, > > > > > > Very nice work! > > > Can we know more about the next plan about the test framework? > > > > > > Wenli > > > > > > > > > On Tue, Sep 3, 2019 at 8:05 PM SHUANG SU <sushuang0...@gmail.com> > wrote: > > > > > > > Wonderful! > > > > > > > > ------------------------------ > > > > Su Shuang (100pah) > > > > ------------------------------ > > > > > > > > > > > > > > > > On Tue, 3 Sep 2019 at 18:51, Yi Shen <shenyi....@gmail.com> wrote: > > > > > > > > > Hi all, > > > > > > > > > > I opened a pull request about adding a tool for running all > > > > > tests automatically > > > > > > > > > > https://github.com/apache/incubator-echarts/pull/11154 > > > > > > > > > > Please leave your comments if you have any suggestions. Thanks > > > > > > > > > > -- > > > > > Yi Shen > > > > > Senior Developer > > > > > Baidu, Inc. > > > > > > > > > > > > > > > > > > -- > > Yi Shen > > Senior Developer > > Baidu, Inc. > > > -- Yi Shen Senior Developer Baidu, Inc.
