It turns out that another module is needed to make it work. Adding these to debian/config/armhf/config did the trick for me:
CONFIG_SND_SUN4I_SPDIF=m CONFIG_SND_SOC_SPDIF=m Since adding modules is unlikely to break anything: Is there a chance that this can still be enabled for stretch? Cheers, Stefan