Re: [ceph-users] inconsistent pgs

2015-08-10 Thread Константин Сахинов
Igor, Jan, David, thanks for your help. The problem was in bad memory chips. Tested it with http://www.memtest.org/ and found several red results. пт, 7 авг. 2015 г. в 13:44, Константин Сахинов : > Hi! > > I have a large number of inconsistent pgs 229 of 656, and it's increasin

Re: [ceph-users] inconsistent pgs

2015-08-09 Thread Константин Сахинов
step take default step chooseleaf firstn 0 type host step emit } rule ssd_ruleset { ruleset 1 type replicated min_size 1 max_size 10 step take ssd step chooseleaf firstn 0 type host step emit } # end crush map пт, 7 авг. 2015 г.

Re: [ceph-users] btrfs w/ centos 7.1

2015-08-07 Thread Константин Сахинов
I'v tested it on my home cluster: 8 OSDs (4 nodes by 2x4TB OSDs with Celeron J1900 and 8GB RAM) + 4 cache tier OSDs (2 nodes by 2x250GB SSD OSDs with Atom D2500 and 4GB RAM). HDD OSDs worked v-v-very slow. And SSD OSDs sometimes stopped working because btrfs couldn't rebalance quickly enough and ov

Re: [ceph-users] Flapping OSD's when scrubbing

2015-08-07 Thread Константин Сахинов
Hi! One time I faced such a behavior of my home cluster. At the time my OSDs go down I noticed that node is using swap despite of sufficient memory. Tuning /proc/sys/vm/swappiness to 0 helped to solve the problem. пт, 7 авг. 2015 г. в 20:41, Tuomas Juntunen : > Thanks > > > > We play with the va

Re: [ceph-users] НА: inconsistent pgs

2015-08-07 Thread Константин Сахинов
rm to data. But we dont use tiering, maybe some > things > happens with data while removing cache tier, like not all objects was > written back to > lower lier pool? > > > Megov Igor > CIO, Yuterra > > > -- > *От:* Константин Сахино

Re: [ceph-users] inconsistent pgs

2015-08-07 Thread Константин Сахинов
step emit } пт, 7 авг. 2015 г. в 15:39, Константин Сахинов : > It's hard to say now. I changed one-by-one my 6 OSDs from btrfs to xfs. > During the repair process I added 2 more OSDs. Changed crush map from > root-host-osd to root-*chasis*-host-osd structure... There was SSD cache &

Re: [ceph-users] inconsistent pgs

2015-08-07 Thread Константин Сахинов
It's hard to say now. I changed one-by-one my 6 OSDs from btrfs to xfs. During the repair process I added 2 more OSDs. Changed crush map from root-host-osd to root-*chasis*-host-osd structure... There was SSD cache tiering set, when first inconsistency showed up. Then I removed tiering to confirm t

Re: [ceph-users] inconsistent pgs

2015-08-07 Thread Константин Сахинов
"you use XFS on your OSDs?" This OSD was formatted in BTRFS as a whole block device /dev/sdc (with no partition table). Then I moved from BTRFS to XFS /dev/sdc1 (with partition table), because BTRFS was v-v-very slow. Maybe partprober sees some old signatures from first sectors of that disk... By

Re: [ceph-users] inconsistent pgs

2015-08-07 Thread Константин Сахинов
No, dmesg is clean on both hosts of osd.1 (block0) and osd.7 (block2). There are only boot time messages (listings below). If there is cable or SATA-controller issues, will it be shown in /var/log/dmesg? block0 dmesg: [5.296302] XFS (sdb1): Mounting V4 Filesystem [5.316487] XFS (sda1): Mo

[ceph-users] inconsistent pgs

2015-08-07 Thread Константин Сахинов
Hi! I have a large number of inconsistent pgs 229 of 656, and it's increasing every hour. I'm using ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3). For example, pg 3.d8: # ceph health detail | grep 3.d8 pg 3.d8 is active+clean+scrubbing+deep+inconsistent, acting [1,7] # grep 3.d8