I want to be able to run my own "idp" type script when someone tries to connect to my mailserver. Basically I want to refuse them even a tcp connection to smtpd if the connecting ip is in our internal blacklist.
is there a reason you want to involve postfix at all? fail2ban scans logs, and then creates firewall rules. since your already have your IP lists, skip the scan & just create the fw rule in front of postfix. populate the lists into an iptables firewall ipset. then create a fw rule to reject traffic from those IPs to your smtpd IP/port.