Is there a way, in postfix, to run a script when the authentication fails, please ? I would like to use nftables sets, with the timeout option, to ban IP addresses. I know fail2ban exists, but I am considering other options. nftables sets, implemented in the kernel, with the timeout option, seem to be a great and very light option.
No, postfix itself can not. You can use a milter to read the headers looking for an authentication fail. The milter can run a shell command for the firewall. The way I do it, is I use omprog of rsyslog to process postfix logs with a script. The script can watch for log lines from opendmarc and run a shell command for the firewall when it finds a fail. Both of these ways requires the ability to do script coding.