Re: fsck: to repair or not to repair

2016-05-12 Thread Duncan
Henk Slager posted on Thu, 12 May 2016 19:02:56 +0200 as excerpted: > Maybe you first want to test it on an overlay > of the device or copy the whole fs with dd. WARNING! Because btrfs can be multi-device, it needs some way to track which devices belong to each filesystem, and it uses filesyst

Re: About in-band dedupe for v4.7

2016-05-12 Thread Zygo Blaxell
On Fri, May 13, 2016 at 11:44:40AM +0800, Qu Wenruo wrote: > Although maybe out of your expectation, inband de-dedupe did exposed some > existing bugs we didn't ever found before. > And they are all reproducible without inband dedupe. > > Some examples: > [...] > 3) Slow backref walk. >Already

Re: Amount of scrubbed data goes from 15.90GiB to 26.66GiB after defragment -r -v -clzo on a fs always mounted with compress=lzo

2016-05-12 Thread Duncan
Niccolò Belli posted on Thu, 12 May 2016 15:56:20 +0200 as excerpted: > Thanks for the detailed explanation, hopefully in the future someone > will be able to make defrag snapshot/reflink aware in a scalable manner. It's still planned, AFAIK, but one of the scaling issues in particular, quotas,

Re: About in-band dedupe for v4.7

2016-05-12 Thread Zygo Blaxell
On Wed, May 11, 2016 at 07:36:59PM +0200, David Sterba wrote: > I like the in-memory dedup backend. It's lightweight, only a heuristic, > does not need any IO or persistent storage. OTOH I consider it a subpart > of the in-band deduplication that does all the persistency etc. So I > treat the ioctl

Re: [PATCH] fstests: generic: Test SHARED flag about fiemap ioctl before and after sync

2016-05-12 Thread Darrick J. Wong
On Fri, May 13, 2016 at 09:52:52AM +0800, Qu Wenruo wrote: > The test case will check SHARED flag returned by fiemap ioctl on > reflinked files before and after sync. > > Normally SHARED flag won't change just due to a normal sync operation. > > But btrfs doesn't handle SHARED flag well, and this

Re: BTRFS Data at Rest File Corruption

2016-05-12 Thread Richard A. Lochner
Chris, See notes inline. On Thu, 2016-05-12 at 19:41 -0600, Chris Murphy wrote: > On Thu, May 12, 2016 at 11:49 AM, Richard A. Lochner com> wrote: > > > > > I suspected, and I still suspect that the error occurred upon a > > metadata update that corrupted the checksum for the file, probably >

[PATCH v2 3/3] btrfs-progs: autogen: Don't show success message on fail

2016-05-12 Thread Zhao Lei
When autogen.sh failed, the success message is still in output: # ./autogen.sh ... configure.ac:131: error: possibly undefined macro: PKG_CHECK_VAR If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. Now type './configure' and 'make

[PATCH v2 2/3] btrfs-progs: autogen: Make build success in CentOS 6 and 7

2016-05-12 Thread Zhao Lei
btrfs-progs build failed in CentOS 6 and 7: #./autogen.sh ... configure.ac:131: error: possibly undefined macro: PKG_CHECK_VAR If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. ... Seems PKG_CHECK_VAR is new in pkgconfig 0.28 (24-

[PATCH v2 0/3] btrfs-progs: autogen: Some compatibility fixs

2016-05-12 Thread Zhao Lei
Changelog v1->v2: 1: btrfs-progs: autogen: Make build success in CentOS 6 and 7 Add AC_SUBST(UDEVDIR), suggested by: Jeff Mahoney Zhao Lei (3): btrfs-progs: autogen: Avoid chdir fail on dirname with blank btrfs-progs: autogen: Make build success in CentOS 6 and 7 btrfs-progs: autogen: Do

[PATCH v2 1/3] btrfs-progs: autogen: Avoid chdir fail on dirname with blank

2016-05-12 Thread Zhao Lei
If source put in dir with blanks, as: /var/lib/jenkins/workspace/btrfs progs autogen will failed: ./autogen.sh: line 95: cd: /var/lib/jenkins/workspace/btrfs: No such file or directory Can be fixed by adding quotes into cd command. Signed-off-by: Zhao Lei --- autogen.sh | 2 +- 1 file chang

Re: About in-band dedupe for v4.7

2016-05-12 Thread Qu Wenruo
Wang Shilong wrote on 2016/05/13 11:13 +0800: Hello Guys, I am commenting not because of Qu's patch, of course, Qu and Mark Fasheh Do a really good thing for Btrfs contributions. Just my two cents! 1) I think Currently, we should really focus on Btrfs stability, there are sti

Re: About in-band dedupe for v4.7

2016-05-12 Thread Wang Shilong
Hello Guys, I am commenting not because of Qu's patch, of course, Qu and Mark Fasheh Do a really good thing for Btrfs contributions. Just my two cents! 1) I think Currently, we should really focus on Btrfs stability, there are still many bugs hidden inside Btrfs, please See Filip

[PATCH] fstests: generic: Test SHARED flag about fiemap ioctl before and after sync

2016-05-12 Thread Qu Wenruo
The test case will check SHARED flag returned by fiemap ioctl on reflinked files before and after sync. Normally SHARED flag won't change just due to a normal sync operation. But btrfs doesn't handle SHARED flag well, and this time it won't check any delayed extent tree(reverse extent searching t

Re: BTRFS Data at Rest File Corruption

2016-05-12 Thread Chris Murphy
On Thu, May 12, 2016 at 11:49 AM, Richard A. Lochner wrote: > I suspected, and I still suspect that the error occurred upon a > metadata update that corrupted the checksum for the file, probably due > to silent memory corruption. If the checksum was silently corrupted, > it would be simply writt

Re: BTRFS Data at Rest File Corruption

2016-05-12 Thread Richard A. Lochner
Austin, Ah, the idea of rewriting the "bad" data block is very interesting. I had not thought of that.  Interestingly, the corrupted file is a raw backup image of a btrfs file system partition. I can mount it as a loop device.  I suppose I could rewrite that data block, mount it and run a scrub on

Re: BTRFS Data at Rest File Corruption

2016-05-12 Thread Goffredo Baroncelli
On 2016-05-12 20:29, Austin S. Hemmelgarn wrote: >> I wonder if there is a way to correct or detect that situation. > The closest we could get is to provide an option to handle this in > scrub, preferably with a big scary warning on it as this same > situation can be easily cause by someone modifyi

Re: [PATCH 0/1] btrfs-progs: Typo review of strings and comments

2016-05-12 Thread Nicholas D Steeves
On 12 May 2016 at 07:29, David Sterba wrote: > On Wed, May 11, 2016 at 07:50:35PM -0400, Nicholas D Steeves wrote: >> There were a couple of instances where I wasn't sure what to do; I've >> annotated them, and they are long lines now. To find them, search or >> grep the diff for 'Steeves'. > > F

Re: About in-band dedupe for v4.7

2016-05-12 Thread Mark Fasheh
On Wed, May 11, 2016 at 07:36:59PM +0200, David Sterba wrote: > On Tue, May 10, 2016 at 07:52:11PM -0700, Mark Fasheh wrote: > > Taking your history with qgroups out of this btw, my opinion does not > > change. > > > > With respect to in-memory only dedupe, it is my honest opinion that such a > >

[PATCH] btrfs-progs: added quiet-option for scripts

2016-05-12 Thread btrfs
From: M G Berberich -q,--quiet to prevent status-messages on stderr --verbose as alternative for -v moved 'Mode NO_FILE_DATA enabled' message to stderr changed default for g_verbose to 1 --- cmds-send.c | 26 ++ 1 file changed, 18 insertions(+), 8 deletions(-) diff --git

Re: Input/output error on newly created file

2016-05-12 Thread Szalma László
2016-05-12 19:41 keltezéssel, Nikolaus Rath írta: On May 12 2016, Diego Calleja wrote: El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió: *ping* Anyone any idea? All I can say is that I've had the same problem in the past. In my case, the problematic files where active torr

Re: BTRFS Data at Rest File Corruption

2016-05-12 Thread Austin S. Hemmelgarn
On 2016-05-12 13:49, Richard A. Lochner wrote: Austin, I rebooted the computer and reran the scrub to no avail. The error is consistent. The reason I brought this question to the mailing list is because it seemed like a situation that might be of interest to the developers. Perhaps, there mig

Re: BTRFS Data at Rest File Corruption

2016-05-12 Thread Richard A. Lochner
Andrew, I agree with your supposition about the metadata and corrupted RAM.   I verified that the data blocks on both devices are equal (see my reply to Austin for the commands I used.  I believe that they correctly prove that the blocks are, in fact, equal. I am not sure I have the skills to "w

Re: fsck: to repair or not to repair

2016-05-12 Thread Ashish Samant
On 05/12/2016 10:35 AM, Nikolaus Rath wrote: On May 12 2016, Henk Slager wrote: On Wed, May 11, 2016 at 11:10 PM, Nikolaus Rath wrote: Hello, I recently ran btrfsck on one of my file systems, and got the following messages: checking extents checking free space cache checking fs roots root

Re: BTRFS Data at Rest File Corruption

2016-05-12 Thread Richard A. Lochner
Austin, I rebooted the computer and reran the scrub to no avail.  The error is consistent. The reason I brought this question to the mailing list is because it seemed like a situation that might be of interest to the developers.  Perhaps, there might be a way to "defend" against this type of corr

Re: Input/output error on newly created file

2016-05-12 Thread Nikolaus Rath
On May 12 2016, Diego Calleja wrote: > El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió: >> *ping* >> >> Anyone any idea? > > All I can say is that I've had the same problem in the past. In my > case, the problematic files where active torrents. The interesting > thing is that

Re: [PATCH 1/2] Btrfs: fix race between fsync and direct IO writes for prealloc extents

2016-05-12 Thread Josef Bacik
On 05/09/2016 07:01 AM, fdman...@kernel.org wrote: From: Filipe Manana When we do a direct IO write against a preallocated extent (fallocate) that does not go beyond the i_size of the inode, we do the write operation without holding the inode's i_mutex (an optimization that landed in commit 388

Re: [PATCH] Btrfs: add semaphore to synchronize direct IO writes with fsync

2016-05-12 Thread Josef Bacik
On 05/12/2016 10:26 AM, fdman...@kernel.org wrote: From: Filipe Manana Due to the optimization of lockless direct IO writes (the inode's i_mutex is not held) introduced in commit 38851cc19adb ("Btrfs: implement unlocked dio write"), we started having races between such writes with concurrent fs

Re: fsck: to repair or not to repair

2016-05-12 Thread Nikolaus Rath
On May 12 2016, Henk Slager wrote: > On Wed, May 11, 2016 at 11:10 PM, Nikolaus Rath wrote: >> Hello, >> >> I recently ran btrfsck on one of my file systems, and got the following >> messages: >> >> checking extents >> checking free space cache >> checking fs roots >> root 5 inode 3149867 errors

[PATCH] Btrfs: add semaphore to synchronize direct IO writes with fsync

2016-05-12 Thread fdmanana
From: Filipe Manana Due to the optimization of lockless direct IO writes (the inode's i_mutex is not held) introduced in commit 38851cc19adb ("Btrfs: implement unlocked dio write"), we started having races between such writes with concurrent fsync operations that use the fast fsync path. These ra

Re: fsck: to repair or not to repair

2016-05-12 Thread Henk Slager
On Wed, May 11, 2016 at 11:10 PM, Nikolaus Rath wrote: > Hello, > > I recently ran btrfsck on one of my file systems, and got the following > messages: > > checking extents > checking free space cache > checking fs roots > root 5 inode 3149867 errors 400, nbytes wrong > root 5 inode 3150237 errors

Re: btrfs ate my data in just two days, after a fresh install. ram and disk are ok. it still mounts, but I cannot repair

2016-05-12 Thread Zygo Blaxell
On Thu, May 12, 2016 at 04:35:24PM +0200, Niccolò Belli wrote: > When doing the btrfs check I also always do a btrfs scrub and it never found > any error. Once it didn't manage to finish the scrub because of: > BTRFS critical (device dm-0): corrupt leaf, slot offset bad: > block=670597120,root=1, s

Re: [PATCH] fstests: test creating a symlink and then fsync its parent directory

2016-05-12 Thread Josef Bacik
On 04/24/2016 09:26 PM, fdman...@kernel.org wrote: From: Filipe Manana Test creating a symlink, fsync its parent directory, power fail and mount again the filesystem. After these steps the symlink should exist and its content must match what we specified when we created it (must not be empty or

Re: Input/output error on newly created file

2016-05-12 Thread Diego Calleja
El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió: > *ping* > > Anyone any idea? All I can say is that I've had the same problem in the past. In my case, the problematic files where active torrents. The interesting thing is that I was able to read them correctly up to a point, t

Re: Amount of scrubbed data goes from 15.90GiB to 26.66GiB after defragment -r -v -clzo on a fs always mounted with compress=lzo

2016-05-12 Thread Niccolò Belli
Thanks for the detailed explanation, hopefully in the future someone will be able to make defrag snapshot/reflink aware in a scalable manner. I will not use use defrag anymore, but what do you suggest me to do to reclaim the lost space? Get rid of my current snapshots or maybe simply running bed

Re: Input/output error on newly created file

2016-05-12 Thread Nikolaus Rath
*ping* Anyone any idea? Best, -Nikolaus On May 09 2016, Nikolaus Rath wrote: > On May 09 2016, Filipe Manana wrote: >> On Sun, May 8, 2016 at 11:18 PM, Nikolaus Rath wrote: >>> Hello, >>> >>> I just created an innocent 10 MB on a btrfs file system, yet my attempt >>> to read it a few seconds

Re: btrfs ate my data in just two days, after a fresh install. ram and disk are ok. it still mounts, but I cannot repair

2016-05-12 Thread Austin S. Hemmelgarn
On 2016-05-12 10:35, Niccolò Belli wrote: On lunedì 9 maggio 2016 18:29:41 CEST, Zygo Blaxell wrote: Did you also check the data matches the backup? btrfs check will only look at the metadata, which is 0.1% of what you've copied. From what you've written, there should be a lot of errors in the

Re: [PATCH 2/3] btrfs-progs: autogen: Make build success in CentOS 6 and 7

2016-05-12 Thread Jeff Mahoney
On 5/12/16 6:42 AM, Zhao Lei wrote: > btrfs-progs build failed in CentOS 6 and 7: > #./autogen.sh > ... > configure.ac:131: error: possibly undefined macro: PKG_CHECK_VAR > If this token and others are legitimate, please use m4_pattern_allow. > See the Autoconf documentation. > ...

Re: btrfs ate my data in just two days, after a fresh install. ram and disk are ok. it still mounts, but I cannot repair

2016-05-12 Thread Niccolò Belli
On lunedì 9 maggio 2016 18:29:41 CEST, Zygo Blaxell wrote: Did you also check the data matches the backup? btrfs check will only look at the metadata, which is 0.1% of what you've copied. From what you've written, there should be a lot of errors in the data too. If you have incorrect data but

Undelete deleted subvolume?

2016-05-12 Thread Andrei Borzenkov
I accidentally deleted wrong snapshot using SUSE snapper. Is it possible to undelete subvolume? I know that it is possible to extract files from old tree (although SLES12 does not seem to offer btrfs-find-root), but is it possible to "reconnect" subvolume back? -- To unsubscribe from this list: sen

Re: [PATHCH] add option to supress "At subvol …" message in btrfs send

2016-05-12 Thread David Sterba
On Sat, May 07, 2016 at 06:29:58PM +0200, M G Berberich wrote: > btrfs send puts a “At subvol …” on stderr, which is very annoying in > scripts, esp. cron-jobs. Piping stderr to /dev/null does suppress this > message, but also error-messages which one would probably want to > see. I added an option

Re: [PATCH 0/1] btrfs-progs: Typo review of strings and comments

2016-05-12 Thread David Sterba
On Wed, May 11, 2016 at 07:50:35PM -0400, Nicholas D Steeves wrote: > Thank you David Sterba for the btrfs-typos.txt which gave me a head > start. Unfortunately I wasn't able finish before btrfs-progs-4.5.3 > was released, because I decided to use emacs' > ispell-comments-and-strings to do a full

[PATCH 2/3] btrfs-progs: autogen: Make build success in CentOS 6 and 7

2016-05-12 Thread Zhao Lei
btrfs-progs build failed in CentOS 6 and 7: #./autogen.sh ... configure.ac:131: error: possibly undefined macro: PKG_CHECK_VAR If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. ... Seems PKG_CHECK_VAR is new in pkgconfig 0.28 (24-

[PATCH 3/3] btrfs-progs: autogen: Don't show success message on fail

2016-05-12 Thread Zhao Lei
When autogen.sh failed, the success message is still in output: # ./autogen.sh ... configure.ac:131: error: possibly undefined macro: PKG_CHECK_VAR If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. Now type './configure' and 'make

[PATCH 1/3] btrfs-progs: autogen: Avoid chdir fail on dirname with blank

2016-05-12 Thread Zhao Lei
If source put in dir with blanks, as: /var/lib/jenkins/workspace/btrfs progs autogen will failed: ./autogen.sh: line 95: cd: /var/lib/jenkins/workspace/btrfs: No such file or directory Can be fixed by adding quotes into cd command. Signed-off-by: Zhao Lei --- autogen.sh | 2 +- 1 file chang

Re: Amount of scrubbed data goes from 15.90GiB to 26.66GiB after defragment -r -v -clzo on a fs always mounted with compress=lzo

2016-05-12 Thread Duncan
Niccolò Belli posted on Wed, 11 May 2016 21:50:43 +0200 as excerpted: > Hi, > Before doing the daily backup I did a btrfs check and btrfs scrub as > usual. > After that this time I also decided to run btrfs filesystem defragment > -r -v -clzo on all subvolumes (from a live distro) and just to be

Re: Btrfs progs release 4.5.3

2016-05-12 Thread David Sterba
On Thu, May 12, 2016 at 09:14:44AM +, Duncan wrote: > David Sterba posted on Wed, 11 May 2016 16:47:39 +0200 as excerpted: > > > btrfs-progs 4.5.2 have been released. A bugfix release. > > So 4.5.3 as stated in the subject line, or 4.5.2 as stated in the message > body? > > Given that I'm o

Re: Idea on compatibility for old distributions

2016-05-12 Thread David Sterba
On Thu, May 12, 2016 at 04:51:22PM +0800, Qu Wenruo wrote: > David Sterba wrote on 2016/05/12 10:37 +0200: > > On Thu, May 12, 2016 at 08:57:47AM +0800, Qu Wenruo wrote: > >> Thanks for the info. > >> > >> I also read the phoronix news yesterday. > >> So for RHEL6 that's meaningless. > >> > >> But

Re: Btrfs progs release 4.5.3

2016-05-12 Thread Duncan
David Sterba posted on Wed, 11 May 2016 16:47:39 +0200 as excerpted: > btrfs-progs 4.5.2 have been released. A bugfix release. So 4.5.3 as stated in the subject line, or 4.5.2 as stated in the message body? Given that I'm on 4.5.2, it must be 4.5.3 that's just released. =:^) -- Duncan - List

Re: [PATCH v4] btrfs: qgroup: Fix qgroup accounting when creating snapshot

2016-05-12 Thread David Sterba
On Wed, May 11, 2016 at 01:30:21PM -0700, Josef Bacik wrote: > > Signed-off-by: Qu Wenruo > > Signed-off-by: Mark Fasheh > Reviewed-by: Josef Bacik Applied to for-next with the following fixup to make it bisectable: --- btrfs: build fixup for qgroup_account_snapshot The macro btrfs_st

Re: Idea on compatibility for old distributions

2016-05-12 Thread Qu Wenruo
David Sterba wrote on 2016/05/12 10:37 +0200: On Thu, May 12, 2016 at 08:57:47AM +0800, Qu Wenruo wrote: Thanks for the info. I also read the phoronix news yesterday. So for RHEL6 that's meaningless. But I'm not sure whether it's still meaningless for OpenSUSE, maybe David has some plan on it?

Re: Idea on compatibility for old distributions

2016-05-12 Thread David Sterba
On Thu, May 12, 2016 at 08:57:47AM +0800, Qu Wenruo wrote: > Thanks for the info. > > I also read the phoronix news yesterday. > So for RHEL6 that's meaningless. > > But I'm not sure whether it's still meaningless for OpenSUSE, maybe > David has some plan on it? No plans. -- To unsubscribe from

[PATCH v2] fstests: generic: Test reserved extent map search routine on deduped file

2016-05-12 Thread Qu Wenruo
For fully deduped file, which means all its file exntents are pointing to the same bytenr, btrfs can cause soft lockup when calling fiemap ioctl on that file, like the following output: -- CPU: 1 PID: 7500 Comm: xfs_io Not tainted 4.5.0-rc6+ #2 Hardware name: innotek GmbH VirtualBox/VirtualBox,