I'll see if I can adapt the dramboot config for the SAMA5D27. I'll probably have to set up SWD debugging... for that I need an adapter for my JLink, and I'll need to solder some wires onto the GiantBoard SWD pads. ... You might be able to use J-link with U-boot too. Put a breakpoint at 0x20008040 and see what happens.
Try the above first.