Hi Jean-Pierre,
        That one worked!    I also tried it with 1200 and 12000 renamed files, 
both with no errors.

Thanks for your info about ldd.  There was indeed a 
/x86_64-linux-gnu/libntfs-3g.so.833.0.0 (which I renamed just to be sure), but 
I see that the test version is statically linked ("not a dynamic executable").

I also just discovered that ntfsresize -fiv does a filesystem consistency 
check.     Do you think it is good enough to trust to detect any issues, e.g. 
after a power-failure?   (This backup drive is likely only ever to be used with 
Linux and rsync.)

Thanks,
        -- Chris

PS.  The use case for this drive is a daily Linux backup but also a 
"grab-and-go" backup for fleeing a forest-fire where we will likely have to 
rely on borrowed computers for a couple weeks.   NTFS is the only 
Linux-writable filesystem I found that can preserve POSIX ownerships and 
permissions and is readable without any additional software on Mac/PC/Linux.    
Thank you for your dedicated work on ntfs-3g!


On Tue Aug 4 2020, at 5:33 AM, Jean-Pierre André <jean-pierre.an...@wanadoo.fr> 
wrote:

> Hi,
> 
> Chris Roehrig wrote on 8/4/20 2:46 AM:
>> Hi Jean-Pierre,
>>      I tried it, but it made no difference:   baddir still gives exactly the 
>> same CHKDSK error.
> 
> Well, I found out my previous tries do not fix all cases,
> as the behavior depends on the order of the renamings.
> 
> Please find attached a third try (discard the previous
> variants before applying this one).
> 
>> I'm not 100% sure if I'm using the patched version:    To mount the 
>> partition I'm running the patched ntfs-3g directly from its build directory:
>>      ./ntfs-3g_ntfsprogs-2017.3.23AR.5/src/ntfs-3g  /dev/sdb1  /mnt
>> Will that be completely stand-alone and isolated from Ubuntu's built-in 
>> ntfs-3g or could it e.g. pull in an old AR.3 shared lib or something?   I 
>> couldn't find any existing ntfs-3g stuff in /lib or /usr/lib.    I'm not 
>> familiar with how FUSE stuff works.
> 
> This a fifteen year old bug in a file unchanged for ten years,
> so the patch applies to any non-archaic version.
> The bug has been left unnoticed because it only occurs in very
> specific conditions, and it does not harm. It is also probably
> self-healing under some condition.
> 
> You have apparently correctly applied the recipe for
> testing without installing (and without disturbing your
> installed configuration). However, if you do "make install"
> the generated code will not be put according to the rules
> for your distribution.
> 
> You will have to declare the target directories as arguments
> to configure, something like (not tested, depends on your
> distribution).
> 
> ./configure --bindir=/bin --sbindir=/sbin --libdir=/lib \
>    --mandir=/usr/share/man
> 
> You can determine where libntfs-3g is installed by executing
> ldd $(which ntfs-3g)
> 
> Jean-Pierre
> 
>> Thanks,
>>      -- Chris
>> 
>> 
>> On Mon Aug 3 2020, at 12:16 AM, Jean-Pierre André 
>> <jean-pierre.an...@wanadoo.fr> wrote:
>> 
>>> Hi,
>>> 
>>> Please find attached a hopefully better variant of the fix.
>>> 
>>> Jean-Pierre
>>> 
>>> Jean-Pierre André wrote on 8/1/20 12:35 PM:
>>>> Hi,
>>>> 
>>>> Thank you for your report including a test pointing out
>>>> the issue.
>>>> 
>>>> Attached is a patch expected to fix it.
>>>> 
>>>> Please test and report.
>>>> 
>>>> Jean-Pierre
>>>> 
>>>> Chris Roehrig wrote on 7/30/20 1:01 AM:
>>>>> I'm trying to get my Linux-based NTFS backup drive to pass a CHKDSK and 
>>>>> came upon this curious situation where CHKDSK finds errors.
>>>>> 
>>>>> It seems to be some issue with how ntfs-3g modifies a directory index 
>>>>> when renaming many files.
>>>>> 
>>>>> The CHKDSK error always seems to be of the form:
>>>>> Stage 2: Examining file name linkage ...
>>>>> The first free byte, 0xc0, and bytes available, 0x150, for root index 
>>>>> $I30 in file 0x40 are not equal.
>>>>> 
>>>>> I've attached a python script (mkbaddir.py) that creates two (apparently) 
>>>>> identical directories, one of which reliably causes this CHKDSK error; 
>>>>> the other doesn't.
>>>>> 
>>>>> How to demonstrate:
>>>>>     - Format an NTFS partition or thumbdrive using Windows or mkfs.ntfs.
>>>>>     - Mount the partition on a Linux system.
>>>>>         I used Mint 20 with ntfs-3g 2017.3.23AR.3 integrated FUSE 28 and
>>>>>         python 3.8.2.
>>>>>     - Chdir to the new NTFS partition and run the script:
>>>>>         /tmp/mkbaddir.py        # creates 'baddir' in current dir.
>>>>>         /tmp/mkbaddir.py -G    # creates 'gooddir' in current dir.
>>>>>         diff -r baddir gooddir        # no difference
>>>>>         du -sB1 baddir gooddir    # same size (128K)
>>>>>     - Boot into Windows (10 v1903) and run (from a terminal) chkdsk X:  
>>>>> (where X: is the NTFS drive).
>>>>>         - This will say:
>>>>>         "Errors found.  CHKDSK cannot continue in read-only mode."
>>>>>     - Delete baddir (I used cygwin's rm -rf), and run chkdsk X: again.
>>>>>         - This will now have no errors.
>>>>> 
>>>>> My guess at what's happening:
>>>>> The script creates a directory of 410 empty files and then renames them 
>>>>> with slightly larger names, which as I understand leaves a bunch of 
>>>>> unused nodes in the b-tree.  The -G option just renames the 410 known 
>>>>> files; without the -G option, it uses os.walk() to traverse the directory 
>>>>> which I'm guessing leaves the b-tree in a slightly different state with 
>>>>> even more unused nodes.
>>>>> 
>>>>> The 410 was chosen by trial-and-error so that some internal threshhold is 
>>>>> just exceeded by the baddir but not by the gooddir.   With more than 410 
>>>>> (using the -c option; say -c 500), both baddir and gooddir will cause 
>>>>> CHKDSK errors.
>>>>> 
>>>>> If I run the script on Windows/cygwin (Python 3.6.9) to create the 
>>>>> folders, it does not give any CHKDSK errors even with many more files.
>>>>> 
>>>>> So there seems to be some issue with how ntfs-3g modifies the b-tree when 
>>>>> renaming many files that is causing CHKDSK to complain.
>>>>> 
>>>>> 
>>>>> I encountered this issue when trying to get my Linux-based NTFS backup 
>>>>> drive to consistently pass a CHKDSK.  I use a script to first rename 
>>>>> POSIX names to valid windows names, replacing '?' with '@@3F', etc so I 
>>>>> can reverse the renaming afterwards.  I have some website mirror folders 
>>>>> with many files of the form:
>>>>>     details.asp?id=xxxxx&key=val
>>>>> which gave rise to this issue.   (In the mkbaddir script I use only 
>>>>> alphanumeric names to be clear this is not an illegal char issue).
>>>>> 
>>>>> 
>>> <shorten_index-v2.patch>
>> 
> 
> <shorten_index-v3.patch>



_______________________________________________
ntfs-3g-devel mailing list
ntfs-3g-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel

Reply via email to