I keep experiencing this issue even with a brand new board (same model, newer hw revision) and on linux-image-4.10.0-rc6-armmp.
I have been informed that this issue might have been fixed by: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers?id=2154d94b40ea2a5de05245521371d0461bb0d669 This change was not present in 4.10-0-rc6, and I will now try with linux-image-4.11.0-trunk, which does include this patch.