It seems that the issue has something to do with the fluidsynth package, when I remove it the issue goes away. So maybe this is a fluidsynth issue as it was updated along with pipewire.
- Bug#1100943: pipewire: Sporadic audio crackling... Alejandro Rosso
- Processed: Re: Bug#1100943: Narrow down th... Debian Bug Tracking System
- Bug#1100943: Narrow down the problem Alejandro Rosso
- Bug#1100943: Narrow down the problem Dylan Aïssi
