(Aha, updating just xterm fixes the problem). When locales don't work, I can't enter umlauts at all - they get replaced by a "generic" character that isn't an umlaut, even when checked using a hexdump tool.
So not only couldn't I see the umlauts, I couldn't type them blindly either.
pgp1De7YUg0b5.pgp
Description: OpenPGP digital signature
