Good Morning Everyone, A little update regarding my initial request. Meanwhile we know a little bit more about one or the other thing.
#1 statfs reporting wrong sizes: The current nuttx code doesn't follow the fat32 white paper [1] recommendation of recomputing the number of free clusters at every mount time. Instead, nuttx takes the information from the FSInfo.sector as granted. A pull request is pending [2]. As it seems does Windows stop updating the FSInfo sector once it recognises any sort of corruption (e.g. indicated number of free cluster conflicts with the FAT table, corrupted dir entries ...). If then are files moved around outside of nuttx, nuttx won't recognise it since that value didn't get updated. It might look like as the card is full while the card is empty. In the case the fs_fsifreecountis is low enough and a file / dir is allocated before a statfs call, it will re-trigger due to the extension of the chain and therefore reduction of that count. [1] https://www.win.tue.nl/~aeb/linux/fs/fat/fatgen103.pdf [2] https://github.com/apache/incubator-nuttx/pull/3760 #2 corrupted directory entries and file content. No update here, this is still the big question. It might help if others have recognised it at some point. A bit more information of the directory entries corruption, this might start in the middle of a file entry, which is 32 Bytes long, but seems to end with the end of a logical block. Therefore, you can have files which at once have a totally wrong file size, create time and so on. A fsck.msdos of the SD-card once in a while might reveal such behaviour. @Jukka, your fix might have an impact here, not sure. But hard to check. While looking into the issue we discovered one more thing. We might make a follow up request once we have time to look into it. If one iterates over a directory with readdir, then the last entry in a directory cluster won't be returned if that one is allocated on the last slot (1024) of the last cluster! If there is another partially filled cluster following for that dir all will be shown. Anyone aware what other projects do use FAT32 partitions which will fill up 4GB regularly and create lots of files (>1024) in a folder? Anyone aware of a tool like fsck.msdos for nuttx or ported to c and a quickformat routine? Best Regards, Reto On 2021/05/18 13:10:45, Reto Gähwiler <gret.hexa...@gmail.com> wrote: > Dear All, > > First of all, in case a similar thread pops-up authored by myself, please > ignore. > > Recently we discovered some issues with the FAT32 partition on the SD-Card > used in our device running on nuttx. There are actually a bunch of issues > we are strugle to understand related to the filesystem. > The issue discovered recently is, that a statfs call won't return the true > number of free clusters. It rather returns what ever is in the FS INFO > section of the FAT32 partition. Inserting such a "corrupted" card into an > SD-Card reader and mounting it to Windows shows following: > > #1 Windows file explorer reports the correct free space. > #2 Comparing the free space in file explorer and looging at FS INFO section > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any > longer the FS INFO section on that card. Even if all the files/dires are > wiped from the card. > #3 Whenever files are added / removed under nuttx the FS INFO section would > be updated by nuttx, but to the wrong number since the base is wrong. > #4 Mounting the card to linux and properly unmount it might actually fix > the issue. But not always! > #5 Quick format the card resolved the issue too. Windows would once again > write the FS INFO section and nuttx would shows the correct free space. > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the > issue too. Reporting a broken file in a root folder on that card. > > We indeed had issues with corrupted files in the past. Usually in the > logging directory, which is accessed most. How they corrupt we do not know. > The corruption is recognised in the following ways: > > #A cryptic and too long file names, we only make use of short file names > (8.3) > #B files whcih became folders > #C sizes which are not possible > #D combinations of the above > > Our application supervises the SD-Card with statfs and removes files once > we hit a quota of 0.85. Therefore, the card is not filling up. However, the > most recent recognition of this behvaiour was due to an application bug > which allowed the card to fill up under some circumstances. > Playing around with a quickformated SD-Card, filled up with data and trying > to retrigger the issue revealed that I never get the ENOSPC but only EIO > once the card is full! But never managed to get the statfs issue. > For the logging we use the fstream family (fopen, fwrite, fread, fflush, > fsync) commands. For config files and other things also the low level > read/write are used. > > One more thing, among all the logging files we once in a while have > corrupted file content. Typically that is a skipped byte or doubled byte > (we use protobuf and can see that encoding the binary). This seems to > happen during reading but also writing the file. > > At the moment we didn't investigate the fat driver of nuttx but rather > tried to analyse the problem. And now we have a bunch of question marks. > > #1 Has anyone recongised similar issues? > #2 Does anyone know a tool which can be used in nuttx equivalent to chkdsk > under windows? Or do quick format? > #3 Why is there no ENOSPC but EIO once the card is full? > #4 What triggers the corrupted files? How robust is the nuttx FAT32 > considering stackdumps and powerlosses? > > For the purpose of investigation, I have an image of the SD-Card when it > was in the state of reporting wrong sizes in nuttx. > > Looking forward to your answers and ideas. Appreciate your help, > > Reto >
