Hi,

Chris Roehrig wrote on 8/4/20 2:46 AM:
Hi Jean-Pierre,
        I tried it, but it made no difference:   baddir still gives exactly the 
same CHKDSK error.

Well, I found out my previous tries do not fix all cases,
as the behavior depends on the order of the renamings.

Please find attached a third try (discard the previous
variants before applying this one).

I'm not 100% sure if I'm using the patched version:    To mount the partition 
I'm running the patched ntfs-3g directly from its build directory:
        ./ntfs-3g_ntfsprogs-2017.3.23AR.5/src/ntfs-3g  /dev/sdb1  /mnt
Will that be completely stand-alone and isolated from Ubuntu's built-in ntfs-3g 
or could it e.g. pull in an old AR.3 shared lib or something?   I couldn't find 
any existing ntfs-3g stuff in /lib or /usr/lib.    I'm not familiar with how 
FUSE stuff works.

This a fifteen year old bug in a file unchanged for ten years,
so the patch applies to any non-archaic version.
The bug has been left unnoticed because it only occurs in very
specific conditions, and it does not harm. It is also probably
self-healing under some condition.

You have apparently correctly applied the recipe for
testing without installing (and without disturbing your
installed configuration). However, if you do "make install"
the generated code will not be put according to the rules
for your distribution.

You will have to declare the target directories as arguments
to configure, something like (not tested, depends on your
distribution).

./configure --bindir=/bin --sbindir=/sbin --libdir=/lib \
   --mandir=/usr/share/man

You can determine where libntfs-3g is installed by executing
ldd $(which ntfs-3g)

Jean-Pierre

Thanks,
        -- Chris


On Mon Aug 3 2020, at 12:16 AM, Jean-Pierre André 
<jean-pierre.an...@wanadoo.fr> wrote:

Hi,

Please find attached a hopefully better variant of the fix.

Jean-Pierre

Jean-Pierre André wrote on 8/1/20 12:35 PM:
Hi,

Thank you for your report including a test pointing out
the issue.

Attached is a patch expected to fix it.

Please test and report.

Jean-Pierre

Chris Roehrig wrote on 7/30/20 1:01 AM:
I'm trying to get my Linux-based NTFS backup drive to pass a CHKDSK and came 
upon this curious situation where CHKDSK finds errors.

It seems to be some issue with how ntfs-3g modifies a directory index when 
renaming many files.

The CHKDSK error always seems to be of the form:
Stage 2: Examining file name linkage ...
The first free byte, 0xc0, and bytes available, 0x150, for root index $I30 in 
file 0x40 are not equal.

I've attached a python script (mkbaddir.py) that creates two (apparently) 
identical directories, one of which reliably causes this CHKDSK error; the 
other doesn't.

How to demonstrate:
     - Format an NTFS partition or thumbdrive using Windows or mkfs.ntfs.
     - Mount the partition on a Linux system.
         I used Mint 20 with ntfs-3g 2017.3.23AR.3 integrated FUSE 28 and
         python 3.8.2.
     - Chdir to the new NTFS partition and run the script:
         /tmp/mkbaddir.py        # creates 'baddir' in current dir.
         /tmp/mkbaddir.py -G    # creates 'gooddir' in current dir.
         diff -r baddir gooddir        # no difference
         du -sB1 baddir gooddir    # same size (128K)
     - Boot into Windows (10 v1903) and run (from a terminal) chkdsk X:  (where 
X: is the NTFS drive).
         - This will say:
         "Errors found.  CHKDSK cannot continue in read-only mode."
     - Delete baddir (I used cygwin's rm -rf), and run chkdsk X: again.
         - This will now have no errors.

My guess at what's happening:
The script creates a directory of 410 empty files and then renames them with 
slightly larger names, which as I understand leaves a bunch of unused nodes in 
the b-tree.  The -G option just renames the 410 known files; without the -G 
option, it uses os.walk() to traverse the directory which I'm guessing leaves 
the b-tree in a slightly different state with even more unused nodes.

The 410 was chosen by trial-and-error so that some internal threshhold is just 
exceeded by the baddir but not by the gooddir.   With more than 410 (using the 
-c option; say -c 500), both baddir and gooddir will cause CHKDSK errors.

If I run the script on Windows/cygwin (Python 3.6.9) to create the folders, it 
does not give any CHKDSK errors even with many more files.

So there seems to be some issue with how ntfs-3g modifies the b-tree when 
renaming many files that is causing CHKDSK to complain.


I encountered this issue when trying to get my Linux-based NTFS backup drive to 
consistently pass a CHKDSK.  I use a script to first rename POSIX names to 
valid windows names, replacing '?' with '@@3F', etc so I can reverse the 
renaming afterwards.  I have some website mirror folders with many files of the 
form:
     details.asp?id=xxxxx&key=val
which gave rise to this issue.   (In the mkbaddir script I use only 
alphanumeric names to be clear this is not an illegal char issue).


<shorten_index-v2.patch>


--- libntfs-3g/index.c.ref	2017-03-23 10:42:44.000000000 +0100
+++ libntfs-3g/index.c	2020-08-04 12:48:46.067922900 +0200
@@ -1563,19 +1563,31 @@
 			   INDEX_ENTRY *ie, INDEX_BLOCK *ib)
 {
 	INDEX_ENTRY *ie_roam;
+	int freed_space;
+	BOOL full;
 	int ret = STATUS_ERROR;
 	
 	ntfs_log_trace("Entering\n");
 	
+	full = ih->index_length == ih->allocated_size;
 	ie_roam = ntfs_ie_dup_novcn(ie);
 	if (!ie_roam)
 		return STATUS_ERROR;
 
 	ntfs_ie_delete(ih, ie);
 
-	if (ntfs_icx_parent_vcn(icx) == VCN_INDEX_ROOT_PARENT)
+	if (ntfs_icx_parent_vcn(icx) == VCN_INDEX_ROOT_PARENT) {
+		/*
+		 * Recover the space which may have been freed
+		 * while deleting an entry from root index
+		 */
+		freed_space = le32_to_cpu(ih->allocated_size)
+				- le32_to_cpu(ih->index_length);
+		if (full && (freed_space > 0) && !(freed_space & 7)) {
+			ntfs_ir_truncate(icx, cpu_to_le32(ih->index_length));
+		}
 		ntfs_inode_mark_dirty(icx->actx->ntfs_ino);
-	else
+	} else
 		if (ntfs_ib_write(icx, ib))
 			goto out;
 	
_______________________________________________
ntfs-3g-devel mailing list
ntfs-3g-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel

Reply via email to