No custom board, it's the tm4c1294-launchpad.
Seems no general problem with heap allocation, so I guess it's not in
the linker script, it's rather directly linked to the usage of the elf-
binary loader, and there are only two configs in the nuttx repo which
use the CONFIG_ELF stuff.
That part has 256Kb of SRAM. That is more that many, but it could be
that you are using too much SRAM and just cannot run ELF reliably. It
does want a lot of SRAM.
Most of the other boards that use CONFIG_ELF=y in a configuration have a
much larger internal ram (exceptions in red):
$ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep RAM_SIZE
./arm/cxd56xx/spresense/configs/elf/defconfig:CONFIG_RAM_SIZE=1572864
./arm/cxd56xx/spresense/configs/module/defconfig:CONFIG_RAM_SIZE=1572864
./arm/cxd56xx/spresense/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=1572864
./arm/cxd56xx/spresense/configs/rndis/defconfig:CONFIG_RAM_SIZE=1572864
./arm/cxd56xx/spresense/configs/wifi/defconfig:CONFIG_RAM_SIZE=1572864
./arm/lc823450/lc823450-xgevk/configs/elf/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lc823450/lc823450-xgevk/configs/krndis/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lc823450/lc823450-xgevk/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lc823450/lc823450-xgevk/configs/rndis/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lc823450/lc823450-xgevk/configs/usb/defconfig:CONFIG_RAM_SIZE=1044480
./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_RAM_SIZE=65536
./arm/sama5/sama5d4-ek/configs/elf/defconfig:CONFIG_RAM_SIZE=268435456
./arm/sama5/sama5d4-ek/configs/knsh/defconfig:CONFIG_RAM_SIZE=268435456
./arm/stm32/olimex-stm32-p407/configs/kelf/defconfig:CONFIG_RAM_SIZE=114688
./arm/stm32/stm32f4discovery/configs/elf/defconfig:CONFIG_RAM_SIZE=114688
./arm/stm32/stm32f4discovery/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=114688
./arm/stm32/stm32f4discovery/configs/rndis/defconfig:CONFIG_RAM_SIZE=114688
./arm/tms570/tms570ls31x-usb-kit/configs/nsh/defconfig:CONFIG_RAM_SIZE=262143
./risc-v/k210/maix-bit/configs/elf/defconfig:CONFIG_RAM_SIZE=2097152
./risc-v/k210/maix-bit/configs/module/defconfig:CONFIG_RAM_SIZE=2097152
./risc-v/k210/maix-bit/configs/posix_spawn/defconfig:CONFIG_RAM_SIZE=2097152
And one of these exceptions has a large external DRAM:
$ find . -name defconfig | xargs grep -l CONFIG_ELF=y | xargs grep EXTDRAM
./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXTDRAM=y
./arm/lpc17xx_40xx/lx_cpu/configs/nsh/defconfig:CONFIG_LPC17_40_EXTDRAMSIZE=33554432
