What audio backend are you using ? I know Pulse Audio / Alsa can just drop / duplicate samples. They try to reduce latency by adjusting some buffer size dynamically but those adjustments will cause discontinuities. You can for instance run pulse audio in debug mode and you'll see if / when it does those and if it matches with what you see.
Cheers,
Sylvain
