Re: Improving tracking/processing of buildfarm test failures

[Alexander Lakhin]

Hello hackers,

Please take a look at the February report on buildfarm failures:
# SELECT br, count(*) FROM failures WHERE dt >= '2025-02-01' AND
 dt < '2025-03-01' GROUP BY br;
REL_13_STABLE: 17
REL_14_STABLE: 11
REL_15_STABLE: 7
REL_16_STABLE: 15
REL_17_STABLE: 13
master: 186
-- Total: 249
(Counting test failures only, excluding indent-check, Configure, Build
errors.)


# SELECT COUNT(*) FROM (SELECT DISTINCT issue_link FROM failures WHERE
 dt >= '2025-02-01' AND dt < '2025-03-01');
32


 # SELECT issue_link, count(*) FROM failures WHERE dt >= '2025-02-01' AND
 dt < '2025-03-01' GROUP BY issue_link ORDER BY 2 DESC LIMIT 7;
https://www.postgresql.org/message-id/816167.1740278884%40sss.pgh.pa.us : 32
-- Fixed

https://www.postgresql.org/message-id/976dcc37-b629-490e-a052-a057477d062f%40dunslane.net
 : 26
--Fixed

https://www.postgresql.org/message-id/57988cee-5cc8-44f9-981e-8d62e556f5c8%40gmail.com
 : 21
-- Fixed

https://www.postgresql.org/message-id/1834446.1740588...@sss.pgh.pa.us : 21
-- Fixed

https://www.postgresql.org/message-id/CA%2BTgmoaob_3aNF5S49WFoJs6Rb37rNnd5WH_e9nXAn_jtWWCLQ%40mail.gmail.com
 : 16
-- Fixed

https://www.postgresql.org/message-id/559462.1737760111%40sss.pgh.pa.us : 14
-- Fixed

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=5bf12323b : 14
-- Fixed


# SELECT count(*) FROM failures WHERE dt >= '2025-02-01' AND
 dt < '2025-03-01' AND issue_link IS NULL; -- Unsorted/unhelpful failures
13

Short-lived failures:190

(I've also updated my script to exclude "*-build" failures.)

Best regards,
Alexander Lakhin
Neon (https://neon.tech)
#!/bin/bash

TMP=${TMPDIR:-/tmp}
wget "https://buildfarm.postgresql.org/cgi-bin/show_failures.pl?max_days=31"; -O 
"$TMP/failures.html"
wget "https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures"; -O 
"$TMP/known-test-failures.html"
sed -E 's/\&max_days/\&amp;max_days/; s/(hours|mins|secs| i) < /\1 \&lt; /; 
s/\&nbsp;/ /g' -i "$TMP/failures.html"
cat << 'EOF' > "$TMP/t.xsl"
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform";
  xmlns="http://www.w3.org/1999/xhtml";
  xmlns:xhtml="http://www.w3.org/1999/xhtml";>

<xsl:output method="text"/>

<xsl:template match="/">
    <xsl:for-each select=".//xhtml:tr[@class='fail' or @class='warnx' or 
@class='warn']//xhtml:td[@class='status']
    [not(contains(., 'Configure')) and not(contains(., 'Build')) and 
not(contains(., 'Make')) and
     not(contains(., 'Install&#09;')) and not(contains(., 'Doc&#09;')) and 
not(contains(., '-build')) and
     not(contains(., 'Contrib&#09;'))  and not(contains(., 'indent-check'))]">
        <xsl:value-of select="xhtml:a/@href" />
        <xsl:text>
        </xsl:text>
    </xsl:for-each>
</xsl:template>

</xsl:stylesheet>
EOF

for fl in $(xsltproc "$TMP/t.xsl" "$TMP/failures.html"); do
#echo $fl
    if [[ $fl == show_log* ]]; then
        sfl=${fl/\&/\&amp;}
        grep -q "$sfl" "$TMP/known-test-failures.html" && continue
        echo "An unknown failure found: 
https://buildfarm.postgresql.org/cgi-bin/$fl";
        wget "https://buildfarm.postgresql.org/cgi-bin/$fl"; -O 
"$TMP/failure-$fl.log"

        il=""
        if grep -q -Pzo \
'(?s)Details for system "[^"]+" failure at stage pg_amcheckCheck,.*'\
'postgresql:pg_amcheck / pg_amcheck/005_opclass_damage\s+TIMEOUT'\
            "$TMP/failure-$fl.log"; then
            
il="https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures#005_opclass_damage.pl_fails_on_Windows_animals_due_to_timeout";

        elif grep -q -Pzo \
'(?s)pgsql.build/src/test/isolation/output_iso/regression.diffs<.*'\
'\+isolationtester: canceling step d2a1 after (300|360) seconds\s*\n'\
' step d2a1: &lt;... completed&gt;\s*\n'\
'-  sum\s*\n'\
'------\s*\n'\
'-10000\s*\n.*'\
'\+ERROR:  canceling statement due to user request\s*\n'\
' step e1c: COMMIT;'\
            "$TMP/failure-$fl.log"; then
            
il="https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures#deadlock-parallel.spec_fails_due_to_timeout_on_jit-enabled_animals";
        fi
        
        if [ -n "$il" ]; then
            echo "        The corresponding issue: $il"
            echo
        fi
    else
        echo "Invalid link: $fl"
    fi
done