Hi Jean-Pierre, That one worked! I also tried it with 1200 and 12000 renamed files, both with no errors.
Thanks for your info about ldd. There was indeed a /x86_64-linux-gnu/libntfs-3g.so.833.0.0 (which I renamed just to be sure), but I see that the test version is statically linked ("not a dynamic executable"). I also just discovered that ntfsresize -fiv does a filesystem consistency check. Do you think it is good enough to trust to detect any issues, e.g. after a power-failure? (This backup drive is likely only ever to be used with Linux and rsync.) Thanks, -- Chris PS. The use case for this drive is a daily Linux backup but also a "grab-and-go" backup for fleeing a forest-fire where we will likely have to rely on borrowed computers for a couple weeks. NTFS is the only Linux-writable filesystem I found that can preserve POSIX ownerships and permissions and is readable without any additional software on Mac/PC/Linux. Thank you for your dedicated work on ntfs-3g! On Tue Aug 4 2020, at 5:33 AM, Jean-Pierre André <jean-pierre.an...@wanadoo.fr> wrote: > Hi, > > Chris Roehrig wrote on 8/4/20 2:46 AM: >> Hi Jean-Pierre, >> I tried it, but it made no difference: baddir still gives exactly the >> same CHKDSK error. > > Well, I found out my previous tries do not fix all cases, > as the behavior depends on the order of the renamings. > > Please find attached a third try (discard the previous > variants before applying this one). > >> I'm not 100% sure if I'm using the patched version: To mount the >> partition I'm running the patched ntfs-3g directly from its build directory: >> ./ntfs-3g_ntfsprogs-2017.3.23AR.5/src/ntfs-3g /dev/sdb1 /mnt >> Will that be completely stand-alone and isolated from Ubuntu's built-in >> ntfs-3g or could it e.g. pull in an old AR.3 shared lib or something? I >> couldn't find any existing ntfs-3g stuff in /lib or /usr/lib. I'm not >> familiar with how FUSE stuff works. > > This a fifteen year old bug in a file unchanged for ten years, > so the patch applies to any non-archaic version. > The bug has been left unnoticed because it only occurs in very > specific conditions, and it does not harm. It is also probably > self-healing under some condition. > > You have apparently correctly applied the recipe for > testing without installing (and without disturbing your > installed configuration). However, if you do "make install" > the generated code will not be put according to the rules > for your distribution. > > You will have to declare the target directories as arguments > to configure, something like (not tested, depends on your > distribution). > > ./configure --bindir=/bin --sbindir=/sbin --libdir=/lib \ > --mandir=/usr/share/man > > You can determine where libntfs-3g is installed by executing > ldd $(which ntfs-3g) > > Jean-Pierre > >> Thanks, >> -- Chris >> >> >> On Mon Aug 3 2020, at 12:16 AM, Jean-Pierre André >> <jean-pierre.an...@wanadoo.fr> wrote: >> >>> Hi, >>> >>> Please find attached a hopefully better variant of the fix. >>> >>> Jean-Pierre >>> >>> Jean-Pierre André wrote on 8/1/20 12:35 PM: >>>> Hi, >>>> >>>> Thank you for your report including a test pointing out >>>> the issue. >>>> >>>> Attached is a patch expected to fix it. >>>> >>>> Please test and report. >>>> >>>> Jean-Pierre >>>> >>>> Chris Roehrig wrote on 7/30/20 1:01 AM: >>>>> I'm trying to get my Linux-based NTFS backup drive to pass a CHKDSK and >>>>> came upon this curious situation where CHKDSK finds errors. >>>>> >>>>> It seems to be some issue with how ntfs-3g modifies a directory index >>>>> when renaming many files. >>>>> >>>>> The CHKDSK error always seems to be of the form: >>>>> Stage 2: Examining file name linkage ... >>>>> The first free byte, 0xc0, and bytes available, 0x150, for root index >>>>> $I30 in file 0x40 are not equal. >>>>> >>>>> I've attached a python script (mkbaddir.py) that creates two (apparently) >>>>> identical directories, one of which reliably causes this CHKDSK error; >>>>> the other doesn't. >>>>> >>>>> How to demonstrate: >>>>> - Format an NTFS partition or thumbdrive using Windows or mkfs.ntfs. >>>>> - Mount the partition on a Linux system. >>>>> I used Mint 20 with ntfs-3g 2017.3.23AR.3 integrated FUSE 28 and >>>>> python 3.8.2. >>>>> - Chdir to the new NTFS partition and run the script: >>>>> /tmp/mkbaddir.py # creates 'baddir' in current dir. >>>>> /tmp/mkbaddir.py -G # creates 'gooddir' in current dir. >>>>> diff -r baddir gooddir # no difference >>>>> du -sB1 baddir gooddir # same size (128K) >>>>> - Boot into Windows (10 v1903) and run (from a terminal) chkdsk X: >>>>> (where X: is the NTFS drive). >>>>> - This will say: >>>>> "Errors found. CHKDSK cannot continue in read-only mode." >>>>> - Delete baddir (I used cygwin's rm -rf), and run chkdsk X: again. >>>>> - This will now have no errors. >>>>> >>>>> My guess at what's happening: >>>>> The script creates a directory of 410 empty files and then renames them >>>>> with slightly larger names, which as I understand leaves a bunch of >>>>> unused nodes in the b-tree. The -G option just renames the 410 known >>>>> files; without the -G option, it uses os.walk() to traverse the directory >>>>> which I'm guessing leaves the b-tree in a slightly different state with >>>>> even more unused nodes. >>>>> >>>>> The 410 was chosen by trial-and-error so that some internal threshhold is >>>>> just exceeded by the baddir but not by the gooddir. With more than 410 >>>>> (using the -c option; say -c 500), both baddir and gooddir will cause >>>>> CHKDSK errors. >>>>> >>>>> If I run the script on Windows/cygwin (Python 3.6.9) to create the >>>>> folders, it does not give any CHKDSK errors even with many more files. >>>>> >>>>> So there seems to be some issue with how ntfs-3g modifies the b-tree when >>>>> renaming many files that is causing CHKDSK to complain. >>>>> >>>>> >>>>> I encountered this issue when trying to get my Linux-based NTFS backup >>>>> drive to consistently pass a CHKDSK. I use a script to first rename >>>>> POSIX names to valid windows names, replacing '?' with '@@3F', etc so I >>>>> can reverse the renaming afterwards. I have some website mirror folders >>>>> with many files of the form: >>>>> details.asp?id=xxxxx&key=val >>>>> which gave rise to this issue. (In the mkbaddir script I use only >>>>> alphanumeric names to be clear this is not an illegal char issue). >>>>> >>>>> >>> <shorten_index-v2.patch> >> > > <shorten_index-v3.patch> _______________________________________________ ntfs-3g-devel mailing list ntfs-3g-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel