Maybe LittleFS or SmartFS could be extended to handle NAND.
I believe that LittleFS has been used with NAND with a NAND FLASH
translation layer. I am not sure of the state of that work, but I know
that people have tried it. Google "LittleFS on NAND flash" and see the
hits.
If a NAND flash translation layer (NFTL) were ported to NuttX, then you
should be able to use almost any file system, although some would be
extremely inefficient with NAND. The NAND flash layer would need to do
sparing, wear-leveling, ECC calculations, etc. as needed by NAND.
I have used NXFFS with NAND and it /almost /works.
The basic filesystem issue is that all available filesystems need to do
read-modify-write operations on flash sections. And they assume that
you can always write a '1' bit to a '0' bit. SmartFS and NXFFS both
assume this behavior explicitly. It is a good assumption with NOR
flash. The problem is that NAND can only be written a whole sector at a
time. This is because the error correction coding (ECC): If you change
even one bit in the NAND sector, you have to re-write the whole sector
with new ECC (because you can't change the '0' bits in the ECC back into
'1''s without erasing the sector).
Since SmartFS and NXFFS both do bit twiddling in the sectors they would
be very inefficient with an NFTL: Each bit twiddle would cause a whole
sector re-write. The same is probably true for LittleFS. Other file
systems like FAT could also be used if there were an NFTL, however, FAT
would also be inefficient (each directory update or FAT update and each
file data size change would cause another sector write).
So the only real solution to support NAND efficiently is use a file
system that is designed specifically around the peculiarities of NAND.
That would require some research (Alan has suggested a couple of places
to start).