For the sake of completeness, I seem to have solved it after some more research and it turned out to be systemd, as following bug that was reported for Centos7 seems to be applying to Debian Buster as well:
https://github.com/systemd/systemd/issues/9276 Luckily the workaround mentioned in: https://github.com/systemd/systemd/issues/9276#issuecomment-442514543 applies here as well and by setting this to the desired swappiness value and rebooting the system so far it seems to work as before and swapping out is only done if it can't be avoided. Martin
