... I can fatload nuttx.bin. But when I run (go 0x20008040) nuttx doesn't boot. Doing the same sequence on the SAMA5D2-XULT results in nuttx booting...
When I worked with the SAMA5D4-EK, I used a tiny bootloader that I wrote called dramboot. It make debug a lot easier. See boards/arm/sama5d4-ek/README.txt
You might be able to use J-link with U-boot too. Put a breakpoint at 0x20008040 and see what happens.
NOTE: I see that the Giant board in now in stock at Mouser US for $62.50
