Ok, I found out that xfce4-taskmanager's GUI is not frozen if I set the refresh rate to 2000ms in the configuration file ($HOME/.config/xfce4/xfce4-taskmanager.rc, option RefreshRate=2000).
xfce4-taskmanager still uses 40% of my CPU and the GUI reacts slow with this refresh rate. If I set the refresh rate to 5s in the GUI xfce4-taskmanager shows me a CPU usage of 20%. The xfce CPU graph widget shows peaks in CPU usage every 5 seconds. It seems that xfce4-taskmanager produces a high load on ARM CPUs when it updates the process list.
signature.asc
Description: OpenPGP digital signature