Thanks for the in-depth explanation! On Wed, Sep 4, 2019 at 10:35 PM Yi Shen <shenyi....@gmail.com> wrote:
> Hi Clement. > > The problem we are trying to solve by this testing tool is: > Running all the exists HTML test cases[1] automatically and compare current > developing version to the previous stable release to test if any function > is broken. > It's more like regression testing. > > Jest and jest-image-snapshot is my first option. > But jest-image-snap is designed to compare with the pre-generated baseline > snapshots. For us, there are some problems: > > 1. Not like normal tests, the pre-generated snapshots are depend on > puppeteer versions, operating systems, or even hardwares. > For example the default system font on macOS and windows are different. > These are all unnecessary variables in a testing system, which we should > eliminate. > 2. Baseline snapshots for nearly 400 tests may have a huge size. > 3. There is also a small problem, we need to update the baseline snapshots > when the test cases are changed. > > There are some workarounds on this issue, like developers generate his own > baseline snapshots, instead of uploading the baseline snapshots. > But I think a better solution for us is picking out the underlying > pixelmatch[2] and rewrite the testing module, which will do: > 1. Download the latest stable ECharts release. > 2. Run test cases with both stable release and developing version to get > snapshots, in a consistent puppeeter, OS, and hardware. > 3. Compare the snapshots and check if there is any diff. > > At last, here are some downsides about this testing tool: > 1. It can't test new add functions. But it's mainly for regression testing > so I think it's not a problem. > 2. Running each test case twice may be time time consuming. > > [1] https://github.com/apache/incubator-echarts/tree/master/test > [2] https://github.com/mapbox/pixelmatch > > On Thu, Sep 5, 2019 at 8:48 AM Clement Ho <clem...@gitlab.com.invalid> > wrote: > > > Curious as to the decision not to use something like jest image > snapshots. > > Could you share your thought process? > > > > On Tue, Sep 3, 2019 at 7:23 AM Yi Shen <shenyi....@gmail.com> wrote: > > > > > Thanks Ovilia. > > > > > > The progress and todo is in the pull request description. The next big > > > thing is supporting interaction testing in the tool. > > > > > > On Tue, Sep 3, 2019 at 8:17 PM Ovilia <oviliazh...@gmail.com> wrote: > > > > > > > Hi Shen, > > > > > > > > Very nice work! > > > > Can we know more about the next plan about the test framework? > > > > > > > > Wenli > > > > > > > > > > > > On Tue, Sep 3, 2019 at 8:05 PM SHUANG SU <sushuang0...@gmail.com> > > wrote: > > > > > > > > > Wonderful! > > > > > > > > > > ------------------------------ > > > > > Su Shuang (100pah) > > > > > ------------------------------ > > > > > > > > > > > > > > > > > > > > On Tue, 3 Sep 2019 at 18:51, Yi Shen <shenyi....@gmail.com> wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I opened a pull request about adding a tool for running all > > > > > > tests automatically > > > > > > > > > > > > https://github.com/apache/incubator-echarts/pull/11154 > > > > > > > > > > > > Please leave your comments if you have any suggestions. Thanks > > > > > > > > > > > > -- > > > > > > Yi Shen > > > > > > Senior Developer > > > > > > Baidu, Inc. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Yi Shen > > > Senior Developer > > > Baidu, Inc. > > > > > > > > -- > Yi Shen > Senior Developer > Baidu, Inc. >
