[PATCH] btrfs: balance: use redundancy instead of integrity

2019-09-24 Thread Anand Jain
When balance reduces the number of copies of data it reduces the redundancy, use the term redundancy instead of integrity. Signed-off-by: Anand Jain --- fs/btrfs/volumes.c | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c ind

Re: Btrfs filesystem trashed after OOM scenario

2019-09-24 Thread Chris Murphy
On Tue, Sep 24, 2019 at 10:25 PM Nick Bowler wrote: > > On Tue, Sep 24, 2019, 18:34 Chris Murphy, wrote: > > On Tue, Sep 24, 2019 at 4:04 PM Nick Bowler wrote: > > > - Running Linux 5.2.14, I pushed this system to OOM; the oom killer > > > ran and killed some userspace tasks. At this point many

Re: [PATCH] btrfs: Fix a regression which we can't convert to SINGLE profile

2019-09-24 Thread Anand Jain
On 9/25/19 10:13 AM, Qu Wenruo wrote: [BUG] With v5.3 kernel, we just can't convert to SINGLE profile by all means: # btrfs balance start -f -dconvert=single $mnt ERROR: error during balancing '/mnt/btrfs': Invalid argument # dmesg -t | tail validate_convert_profile: data profile=0x10

Re: error: invalid convert data profile single

2019-09-24 Thread Qu Wenruo
On 2019/9/25 下午12:15, Chris Murphy wrote: > $ sudo btrfs balance start -dconvert=single,soft / > > It's definitely a 5.3.0 regression, it works without -f on 5.2.15. > Super slow. The media should be able to write at 20M/s. Already pinned down and send the fix (CCed to you) https://patchwork.ke

Re: Btrfs filesystem trashed after OOM scenario

2019-09-24 Thread Nick Bowler
On Tue, Sep 24, 2019, 18:34 Chris Murphy, wrote: > On Tue, Sep 24, 2019 at 4:04 PM Nick Bowler wrote: > > - Running Linux 5.2.14, I pushed this system to OOM; the oom killer > > ran and killed some userspace tasks. At this point many of the > > remaining tasks were stuck in uninterruptible sleep

Re: error: invalid convert data profile single

2019-09-24 Thread Chris Murphy
$ sudo btrfs balance start -dconvert=single,soft / It's definitely a 5.3.0 regression, it works without -f on 5.2.15. Super slow. The media should be able to write at 20M/s. Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE:

[PATCH] btrfs: Fix a regression which we can't convert to SINGLE profile

2019-09-24 Thread Qu Wenruo
[BUG] With v5.3 kernel, we just can't convert to SINGLE profile by all means: # btrfs balance start -f -dconvert=single $mnt ERROR: error during balancing '/mnt/btrfs': Invalid argument # dmesg -t | tail validate_convert_profile: data profile=0x1 allowed=0x20 is_valid=1 final=0

Re: error: invalid convert data profile single

2019-09-24 Thread Qu Wenruo
On 2019/9/25 上午9:26, Chris Murphy wrote: > kernel 5.3.0-1.fc31.x86_64 > btrfs-progs-5.2.1-1.fc31.x86_64 > > sudo btrfs balance start -dconvert=single,soft / > ERROR: error during balancing '/': Invalid argument > There may be more info in syslog - try dmesg | tail > $ sudo btrfs balance start -

Re: Tree checker

2019-09-24 Thread Charles Wright
Ok, I'll check it out. I think I'v tracked down all the files with bad "atime" and cleared all the BTRFS errors by cp -r , deleting the originals and moving the copies in . thanks a bunch . Charles Wright On Tue, Sep 24, 2019 at 12:08 AM Chris Murphy wrote: > > On Mon, Sep 23, 2019 at 6:35 PM

error: invalid convert data profile single

2019-09-24 Thread Chris Murphy
kernel 5.3.0-1.fc31.x86_64 btrfs-progs-5.2.1-1.fc31.x86_64 sudo btrfs balance start -dconvert=single,soft / ERROR: error during balancing '/': Invalid argument There may be more info in syslog - try dmesg | tail $ sudo btrfs balance start -dconvert=single / ERROR: error during balancing '/': Inva

Re: Btrfs filesystem trashed after OOM scenario

2019-09-24 Thread Chris Murphy
On Tue, Sep 24, 2019 at 4:04 PM Nick Bowler wrote: > > Hi folks, > > So I had an interesting scenario that I thought I'd share in case > anyone wants to investigate before I blow away this filesystem... > > Timeline: > - Running Linux 5.2.14, I pushed this system to OOM; the oom killer > ran and k

Re: BTRFS checksum mismatch - false positives

2019-09-24 Thread Chris Murphy
On Tue, Sep 24, 2019 at 7:42 AM wrote: > > Sorry forgot root when issuing commands: > > ash-4.3# btrfs fi show > Label: '2016.05.06-09:13:52 v7321' uuid: 63121c18-2bed-4c81-a514-77be2fba7ab8 > Total devices 1 FS bytes used 4.31TiB > devid1 size 9.97TiB used 4.55TiB path /dev/mapper/vg1-volume

Btrfs filesystem trashed after OOM scenario

2019-09-24 Thread Nick Bowler
Hi folks, So I had an interesting scenario that I thought I'd share in case anyone wants to investigate before I blow away this filesystem... Timeline: - Running Linux 5.2.14, I pushed this system to OOM; the oom killer ran and killed some userspace tasks. At this point many of the remaining tas

Re: BTRFS checksum mismatch - false positives

2019-09-24 Thread Chris Murphy
On Tue, Sep 24, 2019 at 3:29 AM wrote: > > # btrfs fi show > gives no result - not when adding path either > > # btrfs fi df /volume1 > Data, single: total=4.38TiB, used=4.30TiB > System, DUP: total=8.00MiB, used=96.00KiB > System, single: total=4.00MiB, used=0.00B > Metadata, DUP: total=89.50GiB,

Re: [PATCH 0/4] Minor cleanups in locking helpers

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 07:33:16PM +0200, David Sterba wrote: > Reorganizing code wrt locking helpers, minor text and stack savings. > Reviewed-by: Josef Bacik Thanks, Josef

[PATCH 2/2] btrfs: use btrfs_block_group_cache_done in update_block_group

2019-09-24 Thread Josef Bacik
When free'ing extents in a block group we check to see if the block group is not cached, and then cache it if we need to. However we'll just carry on as long as we're loading the cache. This is problematic because we are dirtying the block group here. If we are fast enough we could do a transact

Re: [RFC PATCH 2/3] fs: add RWF_ENCODED for writing compressed data

2019-09-24 Thread Matthew Wilcox
On Tue, Sep 24, 2019 at 10:22:29PM +0200, Christian Brauner wrote: > On Tue, Sep 24, 2019 at 10:01:41PM +0200, Jann Horn wrote: > > Mmh... but if the file descriptor has been passed through a privilege > > boundary, it isn't really clear whether the original opener of the > > file intended for this

[PATCH 1/2] btrfs: check page->mapping when loading free space cache

2019-09-24 Thread Josef Bacik
While testing 5.2 we ran into the following panic [52238.017028] BUG: kernel NULL pointer dereference, address: 0001 [52238.105608] RIP: 0010:drop_buffers+0x3d/0x150 [52238.304051] Call Trace: [52238.308958] try_to_free_buffers+0x15b/0x1b0 [52238.317503] shrink_page_list+0x1164/0x178

Re: [RFC PATCH 2/3] fs: add RWF_ENCODED for writing compressed data

2019-09-24 Thread Omar Sandoval
On Tue, Sep 24, 2019 at 10:01:41PM +0200, Jann Horn wrote: > On Tue, Sep 24, 2019 at 9:35 PM Omar Sandoval wrote: > > On Tue, Sep 24, 2019 at 10:15:13AM -0700, Omar Sandoval wrote: > > > On Thu, Sep 19, 2019 at 05:44:12PM +0200, Jann Horn wrote: > > > > On Thu, Sep 19, 2019 at 8:54 AM Omar Sandova

[PATCH 2/4] btrfs: separate out the extent buffer init code

2019-09-24 Thread Josef Bacik
We want to move this into it's own file, so separate out the init/exit code for setting up the extent_buffer cache. Signed-off-by: Josef Bacik --- fs/btrfs/extent_io.c | 36 +++- fs/btrfs/extent_io.h | 2 ++ fs/btrfs/super.c | 9 - 3 files changed, 2

[PATCH 4/4] btrfs: move the extent-buffer code

2019-09-24 Thread Josef Bacik
This moves the extent buffer code into its own file. Signed-off-by: Josef Bacik --- fs/btrfs/Makefile|3 +- fs/btrfs/extent-buffer.c | 1268 ++ fs/btrfs/extent_io.c | 1256 - 3 files changed, 1270 insertions(

[PATCH 3/4] btrfs: migrate the extent_buffer code out of extent-io.h

2019-09-24 Thread Josef Bacik
There's two main things we do with extent buffers, we mess with the buffers themselves, and then we do IO on those buffers. Separate out the actual extent buffer manipulation code from extent_io since they are unrelated. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 2 +- fs/btrfs

[PATCH 0/4] The remaining extent_io.c split code

2019-09-24 Thread Josef Bacik
Hopefully all of it makes it this time, if you want you can pull from git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git \ extent-io-rearranging I've rebased on top of misc-next and fixed up all the conflicts. Thanks, Josef

Re: [RFC PATCH 2/3] fs: add RWF_ENCODED for writing compressed data

2019-09-24 Thread Christian Brauner
On Tue, Sep 24, 2019 at 10:01:41PM +0200, Jann Horn wrote: > On Tue, Sep 24, 2019 at 9:35 PM Omar Sandoval wrote: > > On Tue, Sep 24, 2019 at 10:15:13AM -0700, Omar Sandoval wrote: > > > On Thu, Sep 19, 2019 at 05:44:12PM +0200, Jann Horn wrote: > > > > On Thu, Sep 19, 2019 at 8:54 AM Omar Sandova

Re: [RFC PATCH 2/3] fs: add RWF_ENCODED for writing compressed data

2019-09-24 Thread Jann Horn
On Tue, Sep 24, 2019 at 9:35 PM Omar Sandoval wrote: > On Tue, Sep 24, 2019 at 10:15:13AM -0700, Omar Sandoval wrote: > > On Thu, Sep 19, 2019 at 05:44:12PM +0200, Jann Horn wrote: > > > On Thu, Sep 19, 2019 at 8:54 AM Omar Sandoval wrote: > > > > Btrfs can transparently compress data written by

Re: Balance ENOSPC during balance despite additional storage added

2019-09-24 Thread Pete
On 9/24/19 2:22 PM, Josef Bacik wrote: > > Just popping in to let you know I've been seeing this internally as well, I > plan > to dig into it after we've run down the panic we're chasing currently. > Thanks, No problem. The only issue it seems to be causing is balance to fail. Pete

BTRFS RAID5/6 - ever?

2019-09-24 Thread hoegge
Dear Chris and others, The problem with RAID5/6 in BTRFS, is really a shame, and led, e.g., Synology to use BTRFS on top of their SHR instead of running "pure BTRFS", hence losing some of the benefits of BTRFS being able to add and remove disks to and from a volume on a running system. Is ther

Re: [RFC PATCH 2/3] fs: add RWF_ENCODED for writing compressed data

2019-09-24 Thread Omar Sandoval
On Tue, Sep 24, 2019 at 10:15:13AM -0700, Omar Sandoval wrote: > On Thu, Sep 19, 2019 at 05:44:12PM +0200, Jann Horn wrote: > > On Thu, Sep 19, 2019 at 8:54 AM Omar Sandoval wrote: > > > Btrfs can transparently compress data written by the user. However, we'd > > > like to add an interface to writ

Re: WITH regression patch, still btrfs-transacti blocked for... (forever)

2019-09-24 Thread David Sterba
On Tue, Sep 24, 2019 at 10:58:08AM +0100, Filipe Manana wrote: > However that's a large patch set which depends on a lot of previous > cleanups, some of which landed in the 5.3 merge window, > Backporting all those patches is against the backport policies for > stable release [1], since many of the

[PATCH 4/4] btrfs: move btrfs_unlock_up_safe to other locking functions

2019-09-24 Thread David Sterba
The function belongs to the family of locking functions, so move it there. The 'noinline' keyword is dropped as it's now an exported function that does not need it. Signed-off-by: David Sterba --- fs/btrfs/ctree.c | 26 -- fs/btrfs/ctree.h | 1 - fs/btrfs

[PATCH 2/4] btrfs: make btrfs_assert_tree_locked static inline

2019-09-24 Thread David Sterba
The function btrfs_assert_tree_locked is used outside of the locking code so it is exported, however we can make it static inine as it's fairly trivial. This is the only locking assertion used in release builds, inlining improves the text size by 174 bytes and reduces stack consumption in the call

[PATCH 3/4] btrfs: move btrfs_set_path_blocking to other locking functions

2019-09-24 Thread David Sterba
The function belongs to the family of locking functions, so move it there. The 'noinline' keyword is dropped as it's now an exported function that does not need it. Signed-off-by: David Sterba --- fs/btrfs/ctree.c | 25 - fs/btrfs/locking.c | 26

[PATCH 0/4] Minor cleanups in locking helpers

2019-09-24 Thread David Sterba
Reorganizing code wrt locking helpers, minor text and stack savings. Debugging build ~~~ textdata bss dec hex filename 1513898 146192 27496 1687586 19c022 pre/btrfs.ko 1514206 146768 27496 1688470 19c396 post/btrfs.ko DELTA: +308 ^^^ the increase is

[PATCH 1/4] btrfs: make locking assertion helpers static inline

2019-09-24 Thread David Sterba
I've noticed that none of the btrfs_assert_*lock* debugging helpers is inlined, despite they're short and mostly a value update. Making them inline shaves 67 from the text size, reduces stack consumption and perhaps also slightly improves the performance due to avoiding unnecessary calls. Signed-o

Re: [RFC PATCH 2/3] fs: add RWF_ENCODED for writing compressed data

2019-09-24 Thread Omar Sandoval
On Thu, Sep 19, 2019 at 05:44:12PM +0200, Jann Horn wrote: > On Thu, Sep 19, 2019 at 8:54 AM Omar Sandoval wrote: > > Btrfs can transparently compress data written by the user. However, we'd > > like to add an interface to write pre-compressed data directly to the > > filesystem. This adds support

Re: [PATCH 1/3] btrfs: Don't opencode btrfs_find_name_in_backref in backref_in_log

2019-09-24 Thread David Sterba
On Fri, Aug 30, 2019 at 05:44:47PM +0300, Nikolay Borisov wrote: > Signed-off-by: Nikolay Borisov Reviewed-by: David Sterba I wrote a short changelog.

Re: [PATCH 2/3] btrfs: Properly handle backref_in_log retval

2019-09-24 Thread David Sterba
On Fri, Aug 30, 2019 at 05:44:48PM +0300, Nikolay Borisov wrote: > This function can return a negative error value if btrfs_search_slot > errors for whatever reason or if btrfs_alloc_path runs out of memory. > This is currently problemattic because backref_in_log is treated by its > callers as if i

Re: [PATCH] Re: Possible FS race condition between iterate_dir and d_alloc_parallel

2019-09-24 Thread Al Viro
On Tue, Sep 24, 2019 at 11:26:28AM -0400, Josef Bacik wrote: > On Tue, Sep 24, 2019 at 04:11:07PM +0100, Al Viro wrote: > > On Tue, Sep 24, 2019 at 11:01:45AM -0400, Josef Bacik wrote: > > > > > Sorry I mis-read the code a little bit. This is purely for the subvolume > > > link > > > directories

Re: [PATCH v2 1/2] btrfs: qgroup: Fix the wrong target io_tree when freeing reserved data space

2019-09-24 Thread David Sterba
On Mon, Sep 16, 2019 at 08:02:38PM +0800, Qu Wenruo wrote: [...] 1 and 2 added to misc-next, thanks.

Re: [PATCH v2] btrfs: relocation: Fix KASAN report about use-after-free due to dead reloc tree cleanup race

2019-09-24 Thread David Sterba
On Mon, Sep 23, 2019 at 02:56:14PM +0800, Qu Wenruo wrote: > [BUG] > One user reported a reproduciable KASAN report about use-after-free: > BTRFS info (device sdi1): balance: start > -dvrange=1256811659264..1256811659265 > BTRFS info (device sdi1): relocating block group 1256811659264 flags >

Re: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs

2019-09-24 Thread David Sterba
On Tue, Sep 24, 2019 at 05:34:11PM +0200, Adam Borowski wrote: > On Tue, Sep 24, 2019 at 04:26:53PM +0200, David Sterba wrote: > > On Tue, Sep 03, 2019 at 05:00:42PM +0200, Johannes Thumshirn wrote: > > > Add an option to mkfs to specify which checksum algorithm will be used for > > > the filesyste

Re: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs

2019-09-24 Thread Adam Borowski
On Tue, Sep 24, 2019 at 04:26:53PM +0200, David Sterba wrote: > On Tue, Sep 03, 2019 at 05:00:42PM +0200, Johannes Thumshirn wrote: > > Add an option to mkfs to specify which checksum algorithm will be used for > > the filesystem. > > > > Signed-off-by: Johannes Thumshirn > > I'll change the opt

Re: [PATCH] Re: Possible FS race condition between iterate_dir and d_alloc_parallel

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 04:11:07PM +0100, Al Viro wrote: > On Tue, Sep 24, 2019 at 11:01:45AM -0400, Josef Bacik wrote: > > > Sorry I mis-read the code a little bit. This is purely for the subvolume > > link > > directories. We haven't wandered down into this directory yet. If the > > subvolum

Re: [PATCH] Re: Possible FS race condition between iterate_dir and d_alloc_parallel

2019-09-24 Thread Al Viro
On Tue, Sep 24, 2019 at 11:01:45AM -0400, Josef Bacik wrote: > Sorry I mis-read the code a little bit. This is purely for the subvolume link > directories. We haven't wandered down into this directory yet. If the > subvolume is being deleted and we still have the fake directory entry for it > t

Re: [PATCH] Re: Possible FS race condition between iterate_dir and d_alloc_parallel

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 03:51:04PM +0100, Al Viro wrote: > On Tue, Sep 24, 2019 at 09:30:26AM -0400, Josef Bacik wrote: > > > > We pass next->d_name.name to dir_emit() (i.e. potentially to > > > copy_to_user()). And we have no warranty that it's not a long > > > (== separately allocated) name, th

Re: [PATCH] Re: Possible FS race condition between iterate_dir and d_alloc_parallel

2019-09-24 Thread Al Viro
On Tue, Sep 24, 2019 at 09:30:26AM -0400, Josef Bacik wrote: > > We pass next->d_name.name to dir_emit() (i.e. potentially to > > copy_to_user()). And we have no warranty that it's not a long > > (== separately allocated) name, that will be freed while > > copy_to_user() is in progress. Sure, it

Re: [PATCH 2/3] btrfs: move ref finding machinery out of build_backref_tree()

2019-09-24 Thread David Sterba
On Wed, Sep 11, 2019 at 12:09:29PM -0400, Josef Bacik wrote: > On Fri, Sep 06, 2019 at 10:15:32AM -0700, Mark Fasheh wrote: > > From: Mark Fasheh > > > > build_backref_tree() is walking extent refs in what is an otherwise self > > contained chunk of code. We can shrink the total number of lines

Re: [PATCH 0/7] btrfs: workqueue fixes+cleanups

2019-09-24 Thread David Sterba
On Mon, Sep 16, 2019 at 11:30:51AM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > Hello, > > This series fixes a few issues with Btrfs' use of workqueues. > > The bulk of this series fixes multiple cases of the same bug: a work > item shouldn't be freed while it potentially depends on an

Re: [PATCH] Btrfs: fix race setting up and completing qgroup rescan workers

2019-09-24 Thread David Sterba
On Tue, Sep 24, 2019 at 10:49:54AM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > There is a race between setting up a qgroup rescan worker and completing > a qgroup rescan worker that can lead to callers of the qgroup rescan wait > ioctl to either not wait for the rescan worker to c

Re: [PATCH v4 00/12] btrfs-progs: support xxhash64 checksums

2019-09-24 Thread David Sterba
On Tue, Sep 03, 2019 at 05:00:34PM +0200, Johannes Thumshirn wrote: > Now that Nikolay's XXHASH64 support for the Crypto API has landed and BTRFS is > prepared for an easy addition of new checksums, this patchset implements > XXHASH64 as a second, fast but not cryptographically secure checksum hash

Re: WITH regression patch, still btrfs-transacti blocked for... (forever)

2019-09-24 Thread Filipe Manana
On Tue, Sep 24, 2019 at 3:28 PM Josef Bacik wrote: > > On Tue, Sep 24, 2019 at 03:23:06PM +0100, Filipe Manana wrote: > > On Tue, Sep 24, 2019 at 3:19 PM Josef Bacik wrote: > > > > > > On Tue, Sep 24, 2019 at 03:16:56PM +0100, Filipe Manana wrote: > > > > On Tue, Sep 24, 2019 at 2:21 PM Josef Bac

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-24 Thread Anand Jain
On 9/16/19 4:19 PM, Anand Jain wrote: On 12/9/19 6:13 PM, Josef Bacik wrote: On Thu, Sep 12, 2019 at 06:10:08PM +0800, Anand Jain wrote: On 12 Sep 2019, at 6:03 PM, Josef Bacik wrote: On Thu, Sep 12, 2019 at 06:00:21PM +0800, Anand Jain wrote: On 12 Sep 2019, at 5:50 PM, Josef Bacik w

Re: WITH regression patch, still btrfs-transacti blocked for... (forever)

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 03:23:06PM +0100, Filipe Manana wrote: > On Tue, Sep 24, 2019 at 3:19 PM Josef Bacik wrote: > > > > On Tue, Sep 24, 2019 at 03:16:56PM +0100, Filipe Manana wrote: > > > On Tue, Sep 24, 2019 at 2:21 PM Josef Bacik wrote: > > > > > > > > On Tue, Sep 24, 2019 at 07:07:41AM -0

Re: [PATCH v4 08/12] btrfs-progs: add option for checksum type to mkfs

2019-09-24 Thread David Sterba
On Tue, Sep 03, 2019 at 05:00:42PM +0200, Johannes Thumshirn wrote: > Add an option to mkfs to specify which checksum algorithm will be used for > the filesystem. > > Signed-off-by: Johannes Thumshirn I'll change the option to '-c' so we have the most common options as lowercase letters.

Re: [PATCH v4 12/12] btrfs-progs: add test-case for mkfs with xxhash64

2019-09-24 Thread David Sterba
On Tue, Sep 03, 2019 at 05:00:46PM +0200, Johannes Thumshirn wrote: > Add test-cases for creating a file-system xxhash64 as checksumming > algorithm. > > Signed-off-by: Johannes Thumshirn > Reviewed-by: Nikolay Borisov > --- > tests/mkfs-tests/001-basic-profiles/test.sh | 2 ++ > 1 file changed

Re: [PATCH v4 10/12] btrfs-progs: add xxhash64 as checksum algorithm

2019-09-24 Thread David Sterba
On Tue, Sep 03, 2019 at 05:00:44PM +0200, Johannes Thumshirn wrote: > From: David Sterba > > Add xxhash64 as another checksumming algorithm. > > Signed-off-by: David Sterba > Signed-off-by: Johannes Thumshirn > > --- > Changes to v3: > - Fix usage of is_valid_csum_type() (Nikolay) > - Remove

Re: WITH regression patch, still btrfs-transacti blocked for... (forever)

2019-09-24 Thread Filipe Manana
On Tue, Sep 24, 2019 at 3:19 PM Josef Bacik wrote: > > On Tue, Sep 24, 2019 at 03:16:56PM +0100, Filipe Manana wrote: > > On Tue, Sep 24, 2019 at 2:21 PM Josef Bacik wrote: > > > > > > On Tue, Sep 24, 2019 at 07:07:41AM -0400, James Harvey wrote: > > > > On Tue, Sep 24, 2019 at 5:58 AM Filipe Man

Re: WITH regression patch, still btrfs-transacti blocked for... (forever)

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 03:16:56PM +0100, Filipe Manana wrote: > On Tue, Sep 24, 2019 at 2:21 PM Josef Bacik wrote: > > > > On Tue, Sep 24, 2019 at 07:07:41AM -0400, James Harvey wrote: > > > On Tue, Sep 24, 2019 at 5:58 AM Filipe Manana wrote: > > > > > > > > On Sun, Sep 15, 2019 at 2:55 PM Fili

Re: [PATCH] Btrfs: fix race setting up and completing qgroup rescan workers

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 03:12:56PM +0100, Filipe Manana wrote: > On Tue, Sep 24, 2019 at 2:53 PM Josef Bacik wrote: > > > > On Tue, Sep 24, 2019 at 10:49:54AM +0100, fdman...@kernel.org wrote: > > > From: Filipe Manana > > > > > > There is a race between setting up a qgroup rescan worker and comp

Re: WITH regression patch, still btrfs-transacti blocked for... (forever)

2019-09-24 Thread Filipe Manana
On Tue, Sep 24, 2019 at 2:21 PM Josef Bacik wrote: > > On Tue, Sep 24, 2019 at 07:07:41AM -0400, James Harvey wrote: > > On Tue, Sep 24, 2019 at 5:58 AM Filipe Manana wrote: > > > > > > On Sun, Sep 15, 2019 at 2:55 PM Filipe Manana wrote: > > > > > > > > On Sun, Sep 15, 2019 at 1:46 PM James Har

Re: [PATCH] Btrfs: fix race setting up and completing qgroup rescan workers

2019-09-24 Thread Filipe Manana
On Tue, Sep 24, 2019 at 3:07 PM Nikolay Borisov wrote: > > > > On 24.09.19 г. 12:49 ч., fdman...@kernel.org wrote: > > From: Filipe Manana > > > > There is a race between setting up a qgroup rescan worker and completing > > a qgroup rescan worker that can lead to callers of the qgroup rescan wait

Re: [PATCH] Btrfs: fix race setting up and completing qgroup rescan workers

2019-09-24 Thread Filipe Manana
On Tue, Sep 24, 2019 at 2:53 PM Josef Bacik wrote: > > On Tue, Sep 24, 2019 at 10:49:54AM +0100, fdman...@kernel.org wrote: > > From: Filipe Manana > > > > There is a race between setting up a qgroup rescan worker and completing > > a qgroup rescan worker that can lead to callers of the qgroup re

Re: [PATCH] Btrfs: fix race setting up and completing qgroup rescan workers

2019-09-24 Thread Nikolay Borisov
On 24.09.19 г. 12:49 ч., fdman...@kernel.org wrote: > From: Filipe Manana > > There is a race between setting up a qgroup rescan worker and completing > a qgroup rescan worker that can lead to callers of the qgroup rescan wait > ioctl to either not wait for the rescan worker to complete or to

Re: [PATCH] Btrfs: fix race setting up and completing qgroup rescan workers

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 10:49:54AM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > There is a race between setting up a qgroup rescan worker and completing > a qgroup rescan worker that can lead to callers of the qgroup rescan wait > ioctl to either not wait for the rescan worker to c

RE: BTRFS checksum mismatch - false positives

2019-09-24 Thread hoegge
Sorry forgot root when issuing commands: ash-4.3# btrfs fi show Label: '2016.05.06-09:13:52 v7321' uuid: 63121c18-2bed-4c81-a514-77be2fba7ab8 Total devices 1 FS bytes used 4.31TiB devid1 siz

Re: [PATCH] Re: Possible FS race condition between iterate_dir and d_alloc_parallel

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 03:52:15AM +0100, Al Viro wrote: > [btrfs and cifs folks Cc'd] > > On Sat, Sep 21, 2019 at 03:07:31PM +0100, Al Viro wrote: > > > No "take cursors out of the list" parts yet. > > Argh... The things turned interesting. The tricky part is > where do we handle switching cu

Re: Balance ENOSPC during balance despite additional storage added

2019-09-24 Thread Josef Bacik
On Fri, Sep 20, 2019 at 03:29:32PM +0100, Pete wrote: > I have a btrfs that is on top of an lvm logical volume on top of dm-crypt on > a single nvme drive (Samsung 870 Pro 512GB). > > I added a second logical volume to give more space to get rid of ENOSPC > errors during balance, but to no avail.

Re: WITH regression patch, still btrfs-transacti blocked for... (forever)

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 07:07:41AM -0400, James Harvey wrote: > On Tue, Sep 24, 2019 at 5:58 AM Filipe Manana wrote: > > > > On Sun, Sep 15, 2019 at 2:55 PM Filipe Manana wrote: > > > > > > On Sun, Sep 15, 2019 at 1:46 PM James Harvey > > > wrote: > > > > ... > > > > You'll see they're differen

Re: [PATCH 0/9][V3] btrfs: break up extent_io.c a little bit

2019-09-24 Thread Josef Bacik
On Tue, Sep 24, 2019 at 09:46:15AM +0200, David Sterba wrote: > On Mon, Sep 23, 2019 at 02:21:04PM -0400, Josef Bacik wrote: > > > I got some strange merge conflicts, it turns out patch 6/9 did not make > > > it to the mailinglist (nor patchwork where I could pick it eventually). > > > For that it'

Re: [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()

2019-09-24 Thread David Sterba
On Tue, Sep 03, 2019 at 05:00:35PM +0200, Johannes Thumshirn wrote: > The callers of csum_tree_block_size() blindly assume we're only having > crc32c as a possible checksum and thus pass in > btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of > csum_tree_block_size(). > > Signed-off-

Re: [PATCH v4 01/12] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()

2019-09-24 Thread David Sterba
On Tue, Sep 03, 2019 at 05:00:35PM +0200, Johannes Thumshirn wrote: > The callers of csum_tree_block_size() blindly assume we're only having > crc32c as a possible checksum and thus pass in > btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of > csum_tree_block_size(). > > Signed-off-

Re: [PATCH v2.1 00/10] btrfs-progs: image: Enhancement with new data dump feature

2019-09-24 Thread Qu Wenruo
On 2019/9/24 下午7:31, Anand Jain wrote: > On 9/19/19 3:19 PM, WenRuo Qu wrote: >> Gentle ping? >> >> This feature should be pretty useful for both full fs backup and debug >> purpose. >> >> Any feedback? > >  Did you miss my review? There are bugs in the patch. Oh, my bad. Indeed there is a bug

Re: [PATCH v2.1 00/10] btrfs-progs: image: Enhancement with new data dump feature

2019-09-24 Thread Anand Jain
On 9/19/19 3:19 PM, WenRuo Qu wrote: Gentle ping? This feature should be pretty useful for both full fs backup and debug purpose. Any feedback? Did you miss my review? There are bugs in the patch. Thanks, Anand It can still be applied to latest stable branch without any conflict. Thanks,

Re: [PATCH] btrfs-progs: drop unique uuid test for btrfstune -M

2019-09-24 Thread Anand Jain
David, ping on this patch. On 9/12/19 8:45 AM, Anand Jain wrote: thanks for the comments, more inline below. - btrfstume -M isn't the place to check if the fsid is a     duplicate. Because, libblkid will be unaware of the complete list of     fs images with its fsid. I don't understand

Re: WITH regression patch, still btrfs-transacti blocked for... (forever)

2019-09-24 Thread James Harvey
On Tue, Sep 24, 2019 at 5:58 AM Filipe Manana wrote: > > On Sun, Sep 15, 2019 at 2:55 PM Filipe Manana wrote: > > > > On Sun, Sep 15, 2019 at 1:46 PM James Harvey > > wrote: > > > ... > > > You'll see they're different looking backtraces than without the > > > patch, so I don't actually know if

Re: [PATCH 2/2] btrfs: add ioctl for directly writing compressed data

2019-09-24 Thread Nikolay Borisov
On 19.09.19 г. 10:59 ч., Omar Sandoval wrote: > On Thu, Sep 19, 2019 at 09:46:31AM +0200, Nikolay Borisov wrote: >> >> >> On 19.09.19 г. 9:14 ч., Omar Sandoval wrote: >>> On Thu, Sep 05, 2019 at 01:33:56PM +0300, Nikolay Borisov wrote: >> >> >> Won't btrfs_lock_and_flush_ordered_range

Re: [PATCH v3 1/6] btrfs-progs: check: Export btrfs_type_to_imode

2019-09-24 Thread Qu WenRuo
On 2019/9/24 下午4:41, Nikolay Borisov wrote: > > > On 12.09.19 г. 6:11 ч., Qu Wenruo wrote: >> This function will be later used by common mode code, so export it. >> >> Signed-off-by: Qu Wenruo > > Reviewed-by: Nikolay Borisov but see one nit below. > >> --- >> check/main.c| 15

Re: WITH regression patch, still btrfs-transacti blocked for... (forever)

2019-09-24 Thread Filipe Manana
On Sun, Sep 15, 2019 at 2:55 PM Filipe Manana wrote: > > On Sun, Sep 15, 2019 at 1:46 PM James Harvey wrote: > > > > For several weeks, I've had an enormous amount of frustration when > > under heavy I/O load with a filesystem putting processes using it into > > permanent uninterruptible sleep.

Re: help needed with unmountable btrfs-filesystem

2019-09-24 Thread Felix Koop
This fs is on a RAID 6 md device, where 1 or 2 devices had problems. This should not have had an effect with the fs. Now I think it might have. Mit freundlichen Grüßen/Kind regards Felix Koop > Qu Wenruo hat am 24. September 2019 um 11:23 > geschrieben: > > > > > On 2019/9/24 下午5:18, Fe

[PATCH] Btrfs: fix race setting up and completing qgroup rescan workers

2019-09-24 Thread fdmanana
From: Filipe Manana There is a race between setting up a qgroup rescan worker and completing a qgroup rescan worker that can lead to callers of the qgroup rescan wait ioctl to either not wait for the rescan worker to complete or to hang forever due to missing wake ups. The following diagram shows

[PATCH] btrfs/036: fix sporadic failures when unmounting scratch filesystem

2019-09-24 Thread fdmanana
From: Filipe Manana Often this test can fail on unmount because a 'btrfs subvolume snapshot' command is still running and using the scratch the mount point: btrfs/036 168s ... umount: /home/fdmanana/btrfs-tests/scratch_1: target is busy (In some cases useful info about processes tha

RE: BTRFS checksum mismatch - false positives

2019-09-24 Thread hoegge
# btrfs fi show gives no result - not when adding path either # btrfs fi df /volume1 Data, single: total=4.38TiB, used=4.30TiB System, DUP: total=8.00MiB, used=96.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=89.50GiB, used=6.63GiB Metadata, single: total=8.00MiB, used=0.00B

Re: help needed with unmountable btrfs-filesystem

2019-09-24 Thread Qu Wenruo
On 2019/9/24 下午5:18, Felix Koop wrote: > Hi Qu, > > this is what I got: > > root@tuxedo:~# btrfs rescue super-recover /dev/md/1 > Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are > you sure? [y/N]: y > checksum verify failed on 340721664 found EACFB938 wanted 24037B

Re: help needed with unmountable btrfs-filesystem

2019-09-24 Thread Felix Koop
Hi Qu, this is what I got: root@tuxedo:~# btrfs rescue super-recover /dev/md/1 Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are you sure? [y/N]: y checksum verify failed on 340721664 found EACFB938 wanted 24037BC4 checksum verify failed on 340721664 found EACFB938 wan

Re: [PATCH v2 2/2] btrfs: qgroup: Fix reserved data space leak if we have multiple reserve calls

2019-09-24 Thread Qu Wenruo
On 2019/9/24 下午5:12, Nikolay Borisov wrote: > > > On 16.09.19 г. 15:02 ч., Qu Wenruo wrote: >> [BUG] >> The following script can cause btrfs qgroup data space leak: >> >> mkfs.btrfs -f $dev >> mount $dev -o nospace_cache $mnt >> >> btrfs subv create $mnt/subv >> btrfs quota en $mnt >>

Re: [PATCH v2 2/2] btrfs: qgroup: Fix reserved data space leak if we have multiple reserve calls

2019-09-24 Thread Nikolay Borisov
On 16.09.19 г. 15:02 ч., Qu Wenruo wrote: > [BUG] > The following script can cause btrfs qgroup data space leak: > > mkfs.btrfs -f $dev > mount $dev -o nospace_cache $mnt > > btrfs subv create $mnt/subv > btrfs quota en $mnt > btrfs quota rescan -w $mnt > btrfs qgroup limit 128m $m

Re: help needed with unmountable btrfs-filesystem

2019-09-24 Thread Qu Wenruo
On 2019/9/24 下午4:27, Felix Koop wrote: > Hi Qu, > > unfortunately nothing under dmesg. > > ~# btrfs ins dump-super -fFa /dev/md/1 > superblock: bytenr=65536, device=/dev/md/1 > - > csum_type 48250 (INVALID) > csum_size

Re: [PATCH v2 1/2] btrfs: qgroup: Fix the wrong target io_tree when freeing reserved data space

2019-09-24 Thread Nikolay Borisov
On 16.09.19 г. 15:02 ч., Qu Wenruo wrote: > [BUG] > Under the follow case with qgroup enabled, if some error happened after > we have reserved delalloc space, then in error handling path, we could > cause qgroup data space leakage: > > From btrfs_truncate_block() in inode.c: > > ret = bt

Re: [PATCH v3 1/6] btrfs-progs: check: Export btrfs_type_to_imode

2019-09-24 Thread Nikolay Borisov
On 12.09.19 г. 6:11 ч., Qu Wenruo wrote: > This function will be later used by common mode code, so export it. > > Signed-off-by: Qu Wenruo Reviewed-by: Nikolay Borisov but see one nit below. > --- > check/main.c| 15 --- > check/mode-common.h | 15 +++ > 2

Re: help needed with unmountable btrfs-filesystem

2019-09-24 Thread Felix Koop
Sorry, forgot kernel and btrfs versions: root@tuxedo:~# uname -a Linux tuxedo.fkoop.de 5.0.0-27-generic #28-Ubuntu SMP Tue Aug 20 19:53:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux root@tuxedo:~# btrfs --version btrfs-progs v4.20.2 Mit freundlichen Grüßen/Kind regards Felix Koop > Felix Koop

Re: help needed with unmountable btrfs-filesystem

2019-09-24 Thread Felix Koop
Hi Qu, unfortunately nothing under dmesg. ~# btrfs ins dump-super -fFa /dev/md/1 superblock: bytenr=65536, device=/dev/md/1 - csum_type 48250 (INVALID) csum_size 32 csum 0x8e5542eb70bced2a96808

[PATCH 3/3] btrfs-progs: fsck-tests: Add test image for invalid inode generation repair

2019-09-24 Thread Qu Wenruo
The image contains one inode item with invalid generation. The image can be crafted by "btrfs-corrupt-block -i 257 -f generation". It should emulate the bad inode generation caused by older kernel around 2014. The image is repairable for both original and lowmem mode. Signed-off-by: Qu Wenruo --

[PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation

2019-09-24 Thread Qu Wenruo
There are at least two bug reports of kernel tree-checker complaining about invalid inode generation. All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow. So add such check and repair ability to lowmem mode check first. This involves: - Calculate the

[PATCH 2/3] btrfs-progs: check/original: Add check and repair for invalid inode generation

2019-09-24 Thread Qu Wenruo
There are at least two bug reports of kernel tree-checker complaining about invalid inode generation. All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow. So add such check and repair ability to lowmem mode check first. This involves: - Calculate the

[PATCH 0/3] btrfs-progs: Add check and repair for invalid inode generation

2019-09-24 Thread Qu Wenruo
We have at least two user reports about bad inode generation makes kernel reject the fs. According to the creation time, the inode is created by some 2014 kernel. And the generation member of INODE_ITEM is not updated (unlike the transid member) so the error persists until latest tree-checker dete

Re: [PATCH 0/9][V3] btrfs: break up extent_io.c a little bit

2019-09-24 Thread David Sterba
On Mon, Sep 23, 2019 at 02:21:04PM -0400, Josef Bacik wrote: > > I got some strange merge conflicts, it turns out patch 6/9 did not make > > it to the mailinglist (nor patchwork where I could pick it eventually). > > For that it's useful to have the list of commits too along with the > > diffstat,

Re: [PATCH] btrfs: prevent memory leak in super.c

2019-09-24 Thread Nikolay Borisov
On 24.09.19 г. 1:57 ч., Navid Emamdoost wrote: > In btrfs_mount_root the last error checking was not going to the error > handling path. Fixed it. > > Signed-off-by: Navid Emamdoost NAK deactivate_locked_super actually calls btrfs_kill_super which in turn calls generic_shutdown_super which d

Re: [PATCH] btrfs: prevent memory leak

2019-09-24 Thread Nikolay Borisov
On 24.09.19 г. 1:34 ч., Navid Emamdoost wrote: > In btrfsic_mount if btrfsic_dev_state_alloc fails the allocated state > will be leaked. It needs to be released on error handling path. > > Signed-off-by: Navid Emamdoost NAK. The allocated state could have been added to a btrfsic_dev_state wh

Re: [PATCH] btrfs compression: check string length

2019-09-24 Thread Nikolay Borisov
On 24.09.19 г. 9:14 ч., Pavel Machek wrote: > AFAICT, with current code user could pass something like "lzox" and > still get "lzo" compression. Check string lengths to prevent that. > > Signed-off-by: Pavel Machek > > diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c > index b05b3