Hi, following up on Werner's query a while back where he was dissatisfied with how the "image diff" metric we use was catching or not problems in test artifacts, I put together a simple python script to address the same problem.
Now that I have a script that does something that is of some use (see below) I could use the help of someone that is familiar with the results of the test suite to bounce with them a few iterations of this until we arrive at a product that makes sense to show to everyone. In my mind I could simply pass them the script when we have something to look at and we could discuss next steps and iterate. My problem is that the EPS filenames for the tests don't really tell me much, so I am having some trouble assessing how well what I have is doing. If you're curious about the details, and for the record: The script compares two directories containing EPS artifacts, meant to be the baseline directory and the test/check directory: $build/input/regression/out-test and $build/input/regression/out-test-baseline It first weeds out all pairs of files with the same name and EPS contents Then it converts the rest into PDF "catalogs" using ghostscript, renders them with pymupdf (I'm particular about this happens exactly), and then compares the pixels, weeding out a few more tests. On what remains it uses a feature detector (at the moment ORB, SIFT and a Harris corner detector are implemented) creating 2D pointclouds of features which are then compared for similarity, and it reports the average distance of the points that don't have an exact match. At the moment it doesn't have a scoring system for points only present in one of the two images but these are already set aside for some upcoming use. At the moment it produces this output whe comparing this 23b3a475aaa964ca9be3cf5af8bdc2390dce12e7 (baseline) to this 23a3778f2ed6a12b2e41026d3e6646f7177f0a29 (test) ------- Loaded 5446 EPS artifacts 2 artifacts are only in baseline 5433 artifacts have identical EPS sources 4 artifacts render to identical pixels 9 artifacts render to different pixels Test 'beam-feather-1.eps' diff measure 2.32 Test 'beam-feather.eps' diff measure 2.32 Test 'beam-french-1.eps' diff measure 2.13 Test 'beam-french.eps' diff measure 2.13 Test 'tablature-tie-spanner-1.eps' diff measure 1.45 Test 'tablature-tie-spanner-2.eps' diff measure 1.17 Test 'tablature-tie-spanner.eps' diff measure 1.41 ------- Cheers, L -- Luca Fascione
