Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-11 Thread David Newall
provide your exact kernel version. I apologise.  I included that in bugzilla but forgot to include it in my message to this list.  It's 4.4.0-159.

[PATCH RFC] btrfs: Introduce btrfs child tree block verification system

2019-09-11 Thread Qu Wenruo
Although we have btrfs_verify_level_key() function to check the first key and level at tree block read time, it has its limitation due to tree lock context, it's not reliable handling new tree blocks. So btrfs_verify_level_key() is good as a pre-check, but it can't ensure new tree blocks are still

Re: [mainline][BUG][PPC][btrfs][bisected 00801a] kernel BUG at fs/btrfs/locking.c:71!

2019-09-11 Thread Abdul Haleem
On Tue, 2019-09-03 at 13:39 +0300, Nikolay Borisov wrote: > > On 3.09.19 г. 11:55 ч., Abdul Haleem wrote: > > Greeting's > > > > Mainline kernel panics with LTP/fs_fill-dir tests for btrfs file system on > > my P9 box running mainline kernel 5.3.0-rc5 > > > > BUG_ON was first introduced by belo

possible circular locking dependency detected (sb_internal/fs_reclaim)

2019-09-11 Thread Zdenek Sojka
Hello, this is my fourth attempt to post this message to the mailing list; this time, without any attached kernel config (because it has over 100KiB). I also tried contacting the kernel btrfs maintainers directly by email, but they probably also didn't receive the message... I am running kerne

Re: [mainline][BUG][PPC][btrfs][bisected 00801a] kernel BUG at fs/btrfs/locking.c:71!

2019-09-11 Thread Nikolay Borisov
On 11.09.19 г. 11:00 ч., Abdul Haleem wrote: > On Tue, 2019-09-03 at 13:39 +0300, Nikolay Borisov wrote: >> >> corresponds to? > > btrfs_search_slot+0x8e8/0xb80 maps to fs/btrfs/ctree.c:2751 > write_lock_level = BTRFS_MAX_LEVEL; That doesn't make sense, presumably btrfs_sear

Re: possible circular locking dependency detected (sb_internal/fs_reclaim)

2019-09-11 Thread Nikolay Borisov
On 11.09.19 г. 10:54 ч., Zdenek Sojka wrote: > Hello, > > this is my fourth attempt to post this message to the mailing list; this > time, without any attached kernel config (because it has over 100KiB). I also > tried contacting the kernel btrfs maintainers directly by email, but they > p

Re: possible circular locking dependency detected (sb_internal/fs_reclaim)

2019-09-11 Thread Zdenek Sojka
Hello, -- Původní e-mail -- Od: Nikolay Borisov Komu: Zdenek Sojka , linux-btrfs@vger.kernel.org Datum: 11. 9. 2019 10:16:49 Předmět: Re: possible circular locking dependency detected (sb_internal/fs_reclaim) On 11.09.19 г. 10:54 ч., Zdenek Sojka wrote: > Hello, > >

Re: [mainline][BUG][PPC][btrfs][bisected 00801a] kernel BUG at fs/btrfs/locking.c:71!

2019-09-11 Thread Abdul Haleem
On Wed, 2019-09-11 at 11:09 +0300, Nikolay Borisov wrote: > > On 11.09.19 г. 11:00 ч., Abdul Haleem wrote: > > On Tue, 2019-09-03 at 13:39 +0300, Nikolay Borisov wrote: > >> > > > > >> corresponds to? > > > > btrfs_search_slot+0x8e8/0xb80 maps to fs/btrfs/ctree.c:2751 > > write

Re: [PATCH] btrfs: nofs inode allocations

2019-09-11 Thread Filipe Manana
On Tue, Sep 10, 2019 at 7:22 PM Josef Bacik wrote: > > A user reported a lockdep splat > > == > WARNING: possible circular locking dependency detected > 5.2.11-gentoo #2 Not tainted > -- >

Re: Odd locking pattern introduced as part of "nowait aio support"

2019-09-11 Thread Andres Freund
Hi, On 2019-09-11 14:04:20 +1000, Dave Chinner wrote: > On Tue, Sep 10, 2019 at 03:33:27PM -0700, Andres Freund wrote: > > Hi, > > > > Especially with buffered io it's fairly easy to hit contention on the > > inode lock, during writes. With something like io_uring, it's even > > easier, because i

Re: Odd locking pattern introduced as part of "nowait aio support"

2019-09-11 Thread Christoph Hellwig
On Wed, Sep 11, 2019 at 02:39:26AM -0700, Andres Freund wrote: > I do really wish buffered NOWAIT was supported... Send a patch..

Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-11 Thread David Newall
echo w > /proc/sysrq-trigger Interesting. One material point which I failed to mention is that the btrfs volume is on an encrypted volume (cryptsetup luksOpen /dev/vdc backups). The first step, "mount -r /dev/vg/ext2fs-snapshot /btrfs-backup-volume/local-snapshot", seemed to trigger the pro

Re: Odd locking pattern introduced as part of "nowait aio support"

2019-09-11 Thread Ritesh Harjani
Hi, On 9/11/19 3:09 PM, Andres Freund wrote: Hi, On 2019-09-11 14:04:20 +1000, Dave Chinner wrote: On Tue, Sep 10, 2019 at 03:33:27PM -0700, Andres Freund wrote: Hi, Especially with buffered io it's fairly easy to hit contention on the inode lock, during writes. With something like io_uring,

Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-11 Thread Nikolay Borisov
On 11.09.19 г. 13:21 ч., David Newall wrote: >> echo w > /proc/sysrq-trigger > > Interesting. > > One material point which I failed to mention is that the btrfs volume is > on an encrypted volume (cryptsetup luksOpen /dev/vdc backups). > > The first step, "mount -r /dev/vg/ext2fs-snapshot > /

Re: Odd locking pattern introduced as part of "nowait aio support"

2019-09-11 Thread Goldwyn Rodrigues
On 2:39 11/09, Andres Freund wrote: > > Ok. Goldwyn, do you want to write a patch, or do you want me to write > one up? I'll post one shortly. Thanks for bringing this up. -- Goldwyn

task btrfs-transacti:1752 blocked for more than 122 seconds.

2019-09-11 Thread Zdenek Sojka
Hello, I am getting task hangups from time to time, mostly when doing fstrim or dedupe while there is other IO ongoing. The following error triggered while I was doing a fstrim, but it might be unrelated. When this hangup happens, I am unable to reboot the computer normally; tasks are stuck, a

Re: [mainline][BUG][PPC][btrfs][bisected 00801a] kernel BUG at fs/btrfs/locking.c:71!

2019-09-11 Thread Nikolay Borisov
On 11.09.19 г. 11:00 ч., Abdul Haleem wrote: > On Tue, 2019-09-03 at 13:39 +0300, Nikolay Borisov wrote: >> >> On 3.09.19 г. 11:55 ч., Abdul Haleem wrote: >>> Greeting's >>> >>> Mainline kernel panics with LTP/fs_fill-dir tests for btrfs file system on >>> my P9 box running mainline kernel 5.3.

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread Austin S. Hemmelgarn
On 2019-09-10 19:32, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : Defrag may break up extents. Defrag may fuse extents. But it shouln't ever unshare extents. Actually, spitting or merging extents will unshare them in a large majority of cases. Ok, this point seems to be re

Re: [PATCH v2 3/6] btrfs-progs: check/common: Make repair_imode_common() to handle inodes in subvolume trees

2019-09-11 Thread Nikolay Borisov
On 11.09.19 г. 3:39 ч., Qu Wenruo wrote: > [...] >>> - Search for DIR_INDEX/DIR_ITEM >>> If above search fails, we falls back to locate the DIR_INDEX/DIR_ITEM >>> just after the INODE_ITEM. >>> If any can be found, it's definitely a directory. >> >> This needs an explicit satement that it

Re: [PATCH v2 3/6] btrfs-progs: check/common: Make repair_imode_common() to handle inodes in subvolume trees

2019-09-11 Thread Qu Wenruo
[...] >>> I have intentionally deleted INODE_REF too see what's happening. Is this >>> intended? >> >> Yes, completely intended. >> >> For this case, you need to iterate through the whole tree to locate the >> DIR_INDEX to fix, which is not really possible with current code base, >> which only sea

[PATCH] btrfs-progs: corrupt-block: Fix description of 'r' option

2019-09-11 Thread Nikolay Borisov
Since commit 04be0e4b1962 ("btrfs-progs: corrupt-block: Correctlyi handle -r when passing -I") the 'r' switch is used with both -I and -d options. So remove the wrong clarificatoin that -r is used only with -d option. Signed-off-by: Nikolay Borisov --- btrfs-corrupt-block.c | 2 +- 1 file change

[PATCH] Btrfs: fix unwritten extent buffers and hangs on future writeback attempts

2019-09-11 Thread fdmanana
From: Filipe Manana The lock_extent_buffer_io() returns 1 to the caller to tell it everything went fine and the callers needs to start writeback for the extent buffer (submit a bio, etc), 0 to tell the caller everything went fine but it does not need to start writeback for the extent buffer, and

Re: task btrfs-transacti:1752 blocked for more than 122 seconds.

2019-09-11 Thread Filipe Manana
On Wed, Sep 11, 2019 at 12:25 PM Zdenek Sojka wrote: > > Hello, > > I am getting task hangups from time to time, mostly when doing fstrim or > dedupe while there is other IO ongoing. The following error triggered while I > was doing a fstrim, but it might be unrelated. > > When this hangup happe

Re: Deadlock between btrfs-transacti and userland on 5.2.1

2019-09-11 Thread Filipe Manana
On Thu, Jul 18, 2019 at 1:25 PM Drazen Kacar wrote: > > Hello, > > I think I've encountered a deadlock between btrfs-transacti and postgres > process(es). This is system information (btrfs fi usage obtained after > poweroff and boot): > > # cat /etc/redhat-release > CentOS Linux release 7.6.1810 (

[PATCH 2/9] btrfs: separate out the extent io init function

2019-09-11 Thread Josef Bacik
We are moving extent_io_tree into it's on file, so separate out the extent_state init stuff from extent_io_tree_init(). Signed-off-by: Josef Bacik --- fs/btrfs/extent_io.c | 18 +++--- fs/btrfs/extent_io.h | 2 ++ fs/btrfs/super.c | 9 - 3 files changed, 21 insertions(+

[PATCH 3/9] btrfs: move extent_io_tree defs to their own header

2019-09-11 Thread Josef Bacik
extent_io.c/h are huge, encompassing a bunch of different things. The extent_io_tree code can live on its own, so separate this out. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 1 + fs/btrfs/extent-io-tree.h | 227 ++ fs/btrfs/extent_io.c

[PATCH 7/9] btrfs: separate out the extent buffer init code

2019-09-11 Thread Josef Bacik
We want to move this into it's own file, so separate out the init/exit code for setting up the extent_buffer cache. Signed-off-by: Josef Bacik --- fs/btrfs/extent_io.c | 36 +++- fs/btrfs/extent_io.h | 2 ++ fs/btrfs/super.c | 9 - 3 files changed, 2

[PATCH 4/9] btrfs: export find_delalloc_range

2019-09-11 Thread Josef Bacik
This utilizes internal stuff to the extent_io_tree, so we need to export it before we move it. Signed-off-by: Josef Bacik --- fs/btrfs/extent-io-tree.h | 2 ++ fs/btrfs/extent_io.c | 5 ++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/extent-io-tree.h b/fs/btrfs

[PATCH 9/9] btrfs: move the extent-buffer code

2019-09-11 Thread Josef Bacik
This moves the extent buffer code into its own file. Signed-off-by: Josef Bacik --- fs/btrfs/Makefile|3 +- fs/btrfs/extent-buffer.c | 1266 ++ fs/btrfs/extent_io.c | 1255 - 3 files changed, 1268 insertions(

[PATCH 8/9] btrfs: migrate the extent_buffer code out of extent-io.h

2019-09-11 Thread Josef Bacik
There's two main things we do with extent buffers, we mess with the buffers themselves, and then we do IO on those buffers. Separate out the actual extent buffer manipulation code from extent_io since they are unrelated. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 2 +- fs/btrfs

[PATCH 0/9] btrfs: break up extent_io.c a little bit

2019-09-11 Thread Josef Bacik
Currently extent_io.c includes all of the extent-io-tree code, the extent buffer code, the code to do IO on extent buffers and data extents, as well as a bunch of other random stuff. The random stuff just needs to be cleaned up, and is probably too invasive for this point in the development cycle.

[PATCH 5/9] btrfs: move the failrec tree stuff into extent-io-tree.h

2019-09-11 Thread Josef Bacik
This needs to be cleaned up in the future, but for now it belongs to the extent-io-tree stuff since it uses the internal tree search code. Needed to export get_state_failrec and set_state_failrec as well since we're not going to move the actual IO part of the failrec stuff out at this point. Signe

[PATCH 1/9] btrfs: separate out the extent leak code

2019-09-11 Thread Josef Bacik
We check both extent buffer and extent state leaks in the same function, separate these two functions out so we can move them around. Signed-off-by: Josef Bacik --- fs/btrfs/extent_io.c | 28 +--- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/fs/btrfs/ex

Re: [PATCH RFC] btrfs: Introduce btrfs child tree block verification system

2019-09-11 Thread Josef Bacik
On Wed, Sep 11, 2019 at 03:46:24PM +0800, Qu Wenruo wrote: > Although we have btrfs_verify_level_key() function to check the first > key and level at tree block read time, it has its limitation due to tree > lock context, it's not reliable handling new tree blocks. > How is it not reliable with n

Re: [PATCH 4/9] btrfs: export find_delalloc_range

2019-09-11 Thread Nikolay Borisov
On 11.09.19 г. 18:26 ч., Josef Bacik wrote: > This utilizes internal stuff to the extent_io_tree, so we need to export > it before we move it. > > Signed-off-by: Josef Bacik > --- > fs/btrfs/extent-io-tree.h | 2 ++ > fs/btrfs/extent_io.c | 5 ++--- > 2 files changed, 4 insertions(+), 3

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-11 Thread Josef Bacik
On Tue, Sep 10, 2019 at 03:26:49PM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > Sometimes when fsync'ing a file we need to log that other inodes exist and > when we need to do that we acquire a reference on the inodes and then drop > that reference using iput() after logging them.

Re: [PATCH] Btrfs: fix unwritten extent buffers and hangs on future writeback attempts

2019-09-11 Thread Chris Mason
On 11 Sep 2019, at 15:55, fdman...@kernel.org wrote: > From: Filipe Manana > > So fix this by not overwriting the return value (ret) with the result > from flush_write_bio(). We also need to clear the > EXTENT_BUFFER_WRITEBACK > bit in case flush_write_bio() returns an error, otherwise it will h

Re: [PATCH 1/3] btrfs: Move backref cache code out of relocation.c

2019-09-11 Thread Josef Bacik
On Fri, Sep 06, 2019 at 10:15:31AM -0700, Mark Fasheh wrote: > From: Mark Fasheh > > No functional changes are made here, we simply move the backref cache out of > relocation.c and into it's own file, backref-cache.c. We also add the > headers relocation.h and backref-cache.h. > > Signed-off-by

Re: [PATCH 2/3] btrfs: move ref finding machinery out of build_backref_tree()

2019-09-11 Thread Josef Bacik
On Fri, Sep 06, 2019 at 10:15:32AM -0700, Mark Fasheh wrote: > From: Mark Fasheh > > build_backref_tree() is walking extent refs in what is an otherwise self > contained chunk of code. We can shrink the total number of lines in > build_backref_tree() *and* make it more readable by moving that wa

Re: [PATCH 3/3] btrfs: move useless node processing out of build_backref_cache

2019-09-11 Thread Josef Bacik
On Fri, Sep 06, 2019 at 10:15:33AM -0700, Mark Fasheh wrote: > From: Mark Fasheh > > The backref cache has to clean up nodes referring to tree leaves. It is > trivial to pull that code into it's own function, which is what I do here. > > Signed-off-by: Mark Fasheh > --- Reviewed-by: Josef Baci

Re: [PATCH] Btrfs: fix unwritten extent buffers and hangs on future writeback attempts

2019-09-11 Thread Filipe Manana
On Wed, Sep 11, 2019 at 5:04 PM Chris Mason wrote: > > On 11 Sep 2019, at 15:55, fdman...@kernel.org wrote: > > > From: Filipe Manana > > > > So fix this by not overwriting the return value (ret) with the result > > from flush_write_bio(). We also need to clear the > > EXTENT_BUFFER_WRITEBACK > >

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread Zygo Blaxell
On Wed, Sep 11, 2019 at 08:02:40AM -0400, Austin S. Hemmelgarn wrote: > On 2019-09-10 19:32, webmas...@zedlx.com wrote: > > > > Quoting "Austin S. Hemmelgarn" : > > > > > > > > Defrag may break up extents. Defrag may fuse extents. But it > > > > shouln't ever unshare extents. > > > > > Actually

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-11 Thread David Sterba
On Mon, Aug 26, 2019 at 05:04:36PM +0800, Anand Jain wrote: > Function call chain __btrfs_map_block()->find_live_mirror() uses > thread pid to determine the %mirror_num when the mirror_num=0. > > This patch introduces a framework so that we can add policies to determine > the %mirror_num. And als

[PATCH v2] Btrfs: fix unwritten extent buffers and hangs on future writeback attempts

2019-09-11 Thread fdmanana
From: Filipe Manana The lock_extent_buffer_io() returns 1 to the caller to tell it everything went fine and the callers needs to start writeback for the extent buffer (submit a bio, etc), 0 to tell the caller everything went fine but it does not need to start writeback for the extent buffer, and

[PATCH] Btrfs: make btrfs_wait_extents() static

2019-09-11 Thread fdmanana
From: Filipe Manana It's not used ouside of transaction.c Signed-off-by: Filipe Manana --- fs/btrfs/transaction.c | 2 +- fs/btrfs/transaction.h | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index e3adb714c04b..84a42e388aa

[PATCH] Btrfs: fix missing error return if writeback for extent buffer never started

2019-09-11 Thread fdmanana
From: Filipe Manana If lock_extent_buffer_for_io() fails, it returns a negative value, but its caller btree_write_cache_pages() ignores such error. This means that a call to flush_write_bio(), from lock_extent_buffer_for_io(), might have failed. We should make btree_write_cache_pages() notice suc

[PATCH 2/3] ext4: fix inode rwsem regression

2019-09-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues This is similar to 942491c9e6d6 ("xfs: fix AIM7 regression") Apparently our current rwsem code doesn't like doing the trylock, then lock for real scheme. So change our read/write methods to just do the trylock for the RWF_NOWAIT case. Fixes: 728fbc0e10b7 ("ext4: nowait a

[PATCH 3/3] f2fs: fix inode rwsem regression

2019-09-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues This is similar to 942491c9e6d6 ("xfs: fix AIM7 regression") Apparently our current rwsem code doesn't like doing the trylock, then lock for real scheme. So change our read/write methods to just do the trylock for the RWF_NOWAIT case. We don't need a check for IOCB_NOWAI

[PATCH 1/3] btrfs: fix inode rwsem regression

2019-09-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues This is similar to 942491c9e6d6 ("xfs: fix AIM7 regression") Apparently our current rwsem code doesn't like doing the trylock, then lock for real scheme. So change our read/write methods to just do the trylock for the RWF_NOWAIT case. Fixes: edf064e7c6fe ("btrfs: nowait

Fix inode sem regression for nowait

2019-09-11 Thread Goldwyn Rodrigues
This changes the way we acquire the inode semaphore when the I/O is marked with IOCB_NOWAIT. The regression was discovered in AIM7 and later by Andres in ext4. This has been fixed in XFS by 942491c9e6d6 ("xfs: fix AIM7 regression") I realized f2fs and btrfs also have the same code and need to be u

Re: [PATCH] Btrfs: fix unwritten extent buffers and hangs on future writeback attempts

2019-09-11 Thread Dennis Zhou
On Wed, Sep 11, 2019 at 05:13:15PM +0100, Filipe Manana wrote: > On Wed, Sep 11, 2019 at 5:04 PM Chris Mason wrote: > > > > On 11 Sep 2019, at 15:55, fdman...@kernel.org wrote: > > > > > From: Filipe Manana > > > > > > So fix this by not overwriting the return value (ret) with the result > > > fr

Re: [PATCH] btrfs-progs: drop unique uuid test for btrfstune -M

2019-09-11 Thread David Sterba
On Tue, Sep 10, 2019 at 01:12:43PM +0800, Anand Jain wrote: > > Nikolay, > > >>> This is intended. Otherwise it's an open avenue for the user to shoot > >>> themselves in the foot. > >> > >>  I don't understand how? > > Again. Any idea how? Is there any test case? > > > > > UUID, by defi

Re: [PATCH] Btrfs: fix unwritten extent buffers and hangs on future writeback attempts

2019-09-11 Thread Filipe Manana
On Wed, Sep 11, 2019 at 5:54 PM Dennis Zhou wrote: > > On Wed, Sep 11, 2019 at 05:13:15PM +0100, Filipe Manana wrote: > > On Wed, Sep 11, 2019 at 5:04 PM Chris Mason wrote: > > > > > > On 11 Sep 2019, at 15:55, fdman...@kernel.org wrote: > > > > > > > From: Filipe Manana > > > > > > > > So fix t

Re: [PATCH] btrfs: volumes: Allow missing devices to be writeable

2019-09-11 Thread David Sterba
On Thu, Aug 29, 2019 at 03:17:31PM +0800, Qu Wenruo wrote: > [BUG] > There is a long existing bug that degraded mounted btrfs can allocate new > SINGLE/DUP chunks on a RAID1 fs: > #!/bin/bash > > dev1=/dev/test/scratch1 > dev2=/dev/test/scratch2 > mnt=/mnt/btrfs > > umount $mnt &> /dev/

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread webmaster
Quoting "Austin S. Hemmelgarn" : On 2019-09-10 19:32, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : === I CHALLENGE you and anyone else on this mailing list: ===  - Show me an exaple where splitting an extent requires unsharing, and this split is needed to defrag. Mak

Re: [PATCH 1/3] btrfs: fix inode rwsem regression

2019-09-11 Thread David Sterba
On Wed, Sep 11, 2019 at 11:45:15AM -0500, Goldwyn Rodrigues wrote: > From: Goldwyn Rodrigues > > This is similar to 942491c9e6d6 ("xfs: fix AIM7 regression") > Apparently our current rwsem code doesn't like doing the trylock, then > lock for real scheme. So change our read/write methods to just

Re: [PATCH] Btrfs: fix unwritten extent buffers and hangs on future writeback attempts

2019-09-11 Thread Dennis Zhou
On Wed, Sep 11, 2019 at 06:02:58PM +0100, Filipe Manana wrote: > On Wed, Sep 11, 2019 at 5:54 PM Dennis Zhou wrote: > > > > On Wed, Sep 11, 2019 at 05:13:15PM +0100, Filipe Manana wrote: > > > On Wed, Sep 11, 2019 at 5:04 PM Chris Mason wrote: > > > > > > > > On 11 Sep 2019, at 15:55, fdman...@ke

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread Austin S. Hemmelgarn
On 2019-09-11 13:20, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-10 19:32, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : === I CHALLENGE you and anyone else on this mailing list: ===  - Show me an exaple where splitting an extent requires unshar

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-11 Thread Josef Bacik
On Mon, Aug 26, 2019 at 05:04:36PM +0800, Anand Jain wrote: > Function call chain __btrfs_map_block()->find_live_mirror() uses > thread pid to determine the %mirror_num when the mirror_num=0. > > This patch introduces a framework so that we can add policies to determine > the %mirror_num. And als

Re: [PATCH] Btrfs: fix missing error return if writeback for extent buffer never started

2019-09-11 Thread Josef Bacik
On Wed, Sep 11, 2019 at 05:42:28PM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > If lock_extent_buffer_for_io() fails, it returns a negative value, but its > caller btree_write_cache_pages() ignores such error. This means that a > call to flush_write_bio(), from lock_extent_buffer_f

Re: [PATCH] Btrfs: make btrfs_wait_extents() static

2019-09-11 Thread Josef Bacik
On Wed, Sep 11, 2019 at 05:42:38PM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > It's not used ouside of transaction.c > > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-11 Thread Eli V
On Wed, Sep 11, 2019 at 2:46 PM Josef Bacik wrote: > > On Mon, Aug 26, 2019 at 05:04:36PM +0800, Anand Jain wrote: > > Function call chain __btrfs_map_block()->find_live_mirror() uses > > thread pid to determine the %mirror_num when the mirror_num=0. > > > > This patch introduces a framework so t

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-11 Thread Josef Bacik
On Wed, Sep 11, 2019 at 03:13:21PM -0400, Eli V wrote: > On Wed, Sep 11, 2019 at 2:46 PM Josef Bacik wrote: > > > > On Mon, Aug 26, 2019 at 05:04:36PM +0800, Anand Jain wrote: > > > Function call chain __btrfs_map_block()->find_live_mirror() uses > > > thread pid to determine the %mirror_num when

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread webmaster
Quoting "Austin S. Hemmelgarn" : On 2019-09-11 13:20, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-10 19:32, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : === I CHALLENGE you and anyone else on this mailing list: ===  - Show me an exaple wher

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread Zygo Blaxell
On Wed, Sep 11, 2019 at 01:20:53PM -0400, webmas...@zedlx.com wrote: > > Quoting "Austin S. Hemmelgarn" : > > > On 2019-09-10 19:32, webmas...@zedlx.com wrote: > > > > > > Quoting "Austin S. Hemmelgarn" : > > > > > > > > > > === I CHALLENGE you and anyone else on this mailing list: === > > >

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread webmaster
Quoting "Austin S. Hemmelgarn" : On 2019-09-11 13:20, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-10 19:32, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : Given this, defrag isn't willfully unsharing anything, it's just a side-effect of how it

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread Zygo Blaxell
On Wed, Sep 11, 2019 at 04:01:01PM -0400, webmas...@zedlx.com wrote: > > Quoting "Austin S. Hemmelgarn" : > > > On 2019-09-11 13:20, webmas...@zedlx.com wrote: > > > > > > Quoting "Austin S. Hemmelgarn" : > > > > > > > On 2019-09-10 19:32, webmas...@zedlx.com wrote: > > > > > > > > > > Quoting

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread webmaster
Quoting Zygo Blaxell : On Wed, Sep 11, 2019 at 01:20:53PM -0400, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : > On 2019-09-10 19:32, webmas...@zedlx.com wrote: > > > > Quoting "Austin S. Hemmelgarn" : > > > > > > === I CHALLENGE you and anyone else on this mailing list: ===

Re: [PATCH RFC] btrfs: Introduce btrfs child tree block verification system

2019-09-11 Thread Qu Wenruo
On 2019/9/12 上午12:02, Josef Bacik wrote: > On Wed, Sep 11, 2019 at 03:46:24PM +0800, Qu Wenruo wrote: >> Although we have btrfs_verify_level_key() function to check the first >> key and level at tree block read time, it has its limitation due to tree >> lock context, it's not reliable handling ne

Re: [PATCH] btrfs: volumes: Allow missing devices to be writeable

2019-09-11 Thread Qu Wenruo
On 2019/9/12 上午1:17, David Sterba wrote: > On Thu, Aug 29, 2019 at 03:17:31PM +0800, Qu Wenruo wrote: >> [BUG] >> There is a long existing bug that degraded mounted btrfs can allocate new >> SINGLE/DUP chunks on a RAID1 fs: >> #!/bin/bash >> >> dev1=/dev/test/scratch1 >> dev2=/dev/test/scra

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread Remi Gauvin
On 2019-09-11 7:21 p.m., webmas...@zedlx.com wrote: > > For example, lets examine the typical home user. If he is using btrfs, > it means he probably wants snapshots of his data. And, after a few > snapshots, his data is fragmented, and the current defrag can't help > because it does a terrible

Re: [PATCH] btrfs-progs: drop unique uuid test for btrfstune -M

2019-09-11 Thread Anand Jain
thanks for the comments, more inline below. - btrfstume -M isn't the place to check if the fsid is a duplicate. Because, libblkid will be unaware of the complete list of fs images with its fsid. I don't understand this part. Blkid tracks the device iformation, like the uuid, and the

[PATCH] btrfs: qgroup: Fix the wrong io_tree when freeing reserved data space

2019-09-11 Thread Qu Wenruo
Commit bc42bda22345 ("btrfs: qgroup: Fix qgroup reserved space underflow by only freeing reserved ranges") is freeing wrong range in BTRFS_I()->io_failure_tree, which should be BTRFS_I()->io_tree. Just fix it. Reported-by: Josef Bacik Fixes: bc42bda22345 ("btrfs: qgroup: Fix qgroup reserved spac

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread webmaster
Quoting Remi Gauvin : On 2019-09-11 7:21 p.m., webmas...@zedlx.com wrote: For example, lets examine the typical home user. If he is using btrfs, it means he probably wants snapshots of his data. And, after a few snapshots, his data is fragmented, and the current defrag can't help because it

[PATCH v2 0/6] btrfs-progs: check: Repair invalid inode mode in subvolume trees

2019-09-11 Thread Qu Wenruo
Before this patch, btrfs check can only repair bad free space cache inode mode (as it was the first case detected by tree-checker and reported) But Murphy is always right, what may happen will finally happen, we have users reporting bad inode mode in subvolume trees. According to the creation time

[PATCH v3 4/6] btrfs-progs: check/lowmem: Repair bad imode early

2019-09-11 Thread Qu Wenruo
For lowmem mode, if we hit a bad inode mode, normally it is reported when we checking the DIR_INDEX/DIR_ITEM of the parent inode. If we didn't repair at that timing, the error will be recorded even we fixed it later. So this patch will check for INODE_ITEM_MISMATCH error type, and if it's really

[PATCH v3 5/6] btrfs-progs: check/original: Fix inode mode in subvolume trees

2019-09-11 Thread Qu Wenruo
To make original mode to repair imode error in subvolume trees, this patch will do: - Remove the show-stopper checks for root->objectid. Now repair_imode_original() will accept inodes in subvolume trees. - Export detect_imode() for original mode Due to the call requirement, original mode must

[PATCH v3 6/6] btrfs-progs: tests/fsck: Add new images for inode mode repair functionality

2019-09-11 Thread Qu Wenruo
Add new test image for imode repair in subvolume trees. The new test cases including the following cases: - Regular file with bad imode It still has the valid INODE_REF and parent dir has correct DIR_INDEX and DIR_ITEM. In this case, no matter if the file is empty or not, it should be repa

[PATCH v3 3/6] btrfs-progs: check/common: Make repair_imode_common() to handle inodes in subvolume trees

2019-09-11 Thread Qu Wenruo
[[PROBLEM]] Before this patch, repair_imode_common() can only handle two types of inodes: - Free space cache inodes - ROOT DIR inodes For inodes in subvolume trees, the core complexity is how to determine the correct imode, thus it was not implemented. However there are more reports of incorrect

[PATCH v3 2/6] btrfs-progs: check/common: Introduce a function to find imode using info from INODE_REF item

2019-09-11 Thread Qu Wenruo
Introduce a function, find_file_type(), to find filetype using info from INODE_REF, including dir_id from key index/name from inode_ref_item. This function will: - Search DIR_INDEX first DIR_INDEX is easier since there is only one item in it. - Validate the DIR_INDEX item If the DIR_INDEX is

[PATCH v3 1/6] btrfs-progs: check: Export btrfs_type_to_imode

2019-09-11 Thread Qu Wenruo
This function will be later used by common mode code, so export it. Signed-off-by: Qu Wenruo --- check/main.c| 15 --- check/mode-common.h | 15 +++ 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/check/main.c b/check/main.c index 2e16b4e6f05b..90

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread Remi Gauvin
On 2019-09-11 11:05 p.m., webmas...@zedlx.com wrote: > > Close, but essentially: yes. I'm implying that snapshots induce future > fragmentation. The mere act of snapshoting won't create fragments > immediately, but if there are any future writes to previously snapshoted > files, those writes are

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread Remi Gauvin
On 2019-09-11 11:30 p.m., Remi Gauvin wrote: > > This statement makes me wonder if you really belong on a Linux > Development list. > > This is why I should avoid getting into debates,, ha.. ext4 does now have defrag.. sorry :) signature.asc Description: OpenPGP digital signature

Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-11 Thread David Newall
On 11/9/19 8:22 pm, Nikolay Borisov wrote: I saved it tohttps://davidnewall.com/kern.1 Nothing useful in that log, everything seems normal. Thank you, again, for your help.  I am grateful. If I understand the output, both df and mount are waiting for show_mountinfo which is waiting for btrfs

Re: Feature requests: online backup - defrag - change RAID level

2019-09-11 Thread Zygo Blaxell
On Wed, Sep 11, 2019 at 07:21:31PM -0400, webmas...@zedlx.com wrote: > > Quoting Zygo Blaxell : > > > On Wed, Sep 11, 2019 at 01:20:53PM -0400, webmas...@zedlx.com wrote: > > > > > > Quoting "Austin S. Hemmelgarn" : > > > > > > > On 2019-09-10 19:32, webmas...@zedlx.com wrote: > > > > > > > > >

Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-11 Thread Nikolay Borisov
On 12.09.19 г. 7:38 ч., David Newall wrote: > On 11/9/19 8:22 pm, Nikolay Borisov wrote: >>> I saved it tohttps://davidnewall.com/kern.1 >> Nothing useful in that log, everything seems normal. > > Thank you, again, for your help.  I am grateful. > > If I understand the output, both df and moun

Re: [PATCH 3/3] f2fs: fix inode rwsem regression

2019-09-11 Thread Chao Yu
On 2019/9/12 0:45, Goldwyn Rodrigues wrote: > From: Goldwyn Rodrigues > > This is similar to 942491c9e6d6 ("xfs: fix AIM7 regression") > Apparently our current rwsem code doesn't like doing the trylock, then > lock for real scheme. So change our read/write methods to just do the > trylock for th

Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-11 Thread Nikolay Borisov
On 12.09.19 г. 9:11 ч., Nikolay Borisov wrote: > > > On 12.09.19 г. 7:38 ч., David Newall wrote: >> On 11/9/19 8:22 pm, Nikolay Borisov wrote: I saved it tohttps://davidnewall.com/kern.1 >>> Nothing useful in that log, everything seems normal. >> >> Thank you, again, for your help.  I am