On 29/01/20 9:56 pm, Cox, Jason wrote:
I have glusterfs (v6.6) deployed with 3-way replication used by ovirt
(v4.3).
I recently updated 1 of the nodes (now at gluster v6.7) and rebooted.
When it came back online, glusterfs reported there were entries to be
healed under the 2 nodes that had stayed online.
After 2+ days, the 2 nodes still show entries that need healing, so
I’m trying to determine what the issue is.
The files shown in the heal info output are small so healing should
not take long. Also, ‘Gluster v heal <vol>’ and ‘gluster v heal <vol>
full’ both return successful, but the entries persist.
So first off, I’m a little confused by what gluster volume heal <vol>
info is reporting.
The following is what I see from heal info:
# gluster v heal engine info
Brick repo0:/gluster_bricks/engine/engine
/372501f5-062c-4790-afdb-dd7e761828ac/images/968daf61-6858-454a-9ed4-3d3db2ae1805/4317dd0d-fd35-4176-9353-7ff69e3a8dc3.meta
/372501f5-062c-4790-afdb-dd7e761828ac/images/4e3e8ca5-0edf-42ae-ac7b-e9a51ad85922/ceb42742-eaaa-4867-aa54-da525629aae4.meta
Status: Connected
Number of entries: 2
Brick repo1:/gluster_bricks/engine/engine
/372501f5-062c-4790-afdb-dd7e761828ac/images/968daf61-6858-454a-9ed4-3d3db2ae1805/4317dd0d-fd35-4176-9353-7ff69e3a8dc3.meta
/372501f5-062c-4790-afdb-dd7e761828ac/images/4e3e8ca5-0edf-42ae-ac7b-e9a51ad85922/ceb42742-eaaa-4867-aa54-da525629aae4.meta
Status: Connected
Number of entries: 2
Brick repo2:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Repo0 and repo1 were not rebooted, but repo2 was.
Since repo2 went offline I would expect when it came back online it
would have entries that need healing, but based on the heal info
output that’s not what it looks like, so I’m thinking maybe heal info
isn’t reporting what I think it is reporting.
*When gluster volume heal <vol> info reports entries as above, what is
it saying?
In heal info output, it is usually the nodes that were up that display
the list of files that need heal. So the way to interpret it is while
repo2 was down, repo0 and repo1 witnessed some modification to the files
and therefore capture them as needing heal, whose list is what the CLI
displays
From the above output, I was reading it as repo0 has 2 entries that
need to be healed from the other bricks and repo1 has 2 entries that
need healing from the other bricks. However, that doesn’t make sense
since repo2 was the one that was rebooted and a ‘stat’ on the files in
the bricks show repo2 is the older version (checksums also show repo0
and repo1 match). Trying to access the file through the FUSE mount on
any node gives input/output errors.
Getfattr output:
repo0 glusterfs]# getfattr -d -m. -e hex
/gluster_bricks/engine/engine/372501f5-062c-4790-afdb-dd7e761828ac/images/4e3e8ca5-0edf-42ae-ac7b-e9a51ad85922/ceb42742-eaaa-4867-aa54-da525629aae4.meta
getfattr: Removing leading '/' from absolute path names
# file:
gluster_bricks/engine/engine/372501f5-062c-4790-afdb-dd7e761828ac/images/4e3e8ca5-0edf-42ae-ac7b-e9a51ad85922/ceb42742-eaaa-4867-aa54-da525629aae4.meta
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000020000000200000000
trusted.bit-rot.signature=0x0102000000000000009338ff61a57fcb452b92ae816b8e5ff672be6d340e7da0a0dcfa34e26b26933b
trusted.bit-rot.version=0x02000000000000005e09d54f000ef84f
trusted.gfid=0xb85edc187d594872a594c25419154d05
trusted.gfid2path.ff2d749198341aff=0x32393564303861372d386437352d343638392d393239332d3339336434346362656233342f63656234323734322d656161612d343836372d616135342d6461353235363239616165342e6d657461
trusted.glusterfs.mdata=0x010000000000000000000000005e2f88ce000000002d1fe613000000005e2f88ce000000002d10ae53000000005e2f88ce000000002d067b3b
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000000001ad000000000000000000000000000000010000000000000000
trusted.pgfid.295d08a7-8d75-4689-9293-393d44cbeb34=0x00000001
repo1 glusterfs]# getfattr -d -m. -e hex
/gluster_bricks/engine/engine/372501f5-062c-4790-afdb-dd7e761828ac/images/4e3e8ca5-0edf-42ae-ac7b-e9a51ad85922/ceb42742-eaaa-4867-aa54-da525629aae4.meta
getfattr: Removing leading '/' from absolute path names
# file:
gluster_bricks/engine/engine/372501f5-062c-4790-afdb-dd7e761828ac/images/4e3e8ca5-0edf-42ae-ac7b-e9a51ad85922/ceb42742-eaaa-4867-aa54-da525629aae4.meta
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-2=0x000000020000000200000000
trusted.bit-rot.signature=0x0102000000000000009338ff61a57fcb452b92ae816b8e5ff672be6d340e7da0a0dcfa34e26b26933b
trusted.bit-rot.version=0x02000000000000005e09db580000709b
trusted.gfid=0xb85edc187d594872a594c25419154d05
trusted.gfid2path.ff2d749198341aff=0x32393564303861372d386437352d343638392d393239332d3339336434346362656233342f63656234323734322d656161612d343836372d616135342d6461353235363239616165342e6d657461
trusted.glusterfs.mdata=0x010000000000000000000000005e2f88ce000000002d1fe613000000005e2f88ce000000002d10ae53000000005e2f88ce000000002d067b3b
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000000001ad000000000000000000000000000000010000000000000000
trusted.pgfid.295d08a7-8d75-4689-9293-393d44cbeb34=0x00000001
repo2 glusterfs]# getfattr -d -m. -e hex
/gluster_bricks/engine/engine/372501f5-062c-4790-afdb-dd7e761828ac/images/4e3e8ca5-0edf-42ae-ac7b-e9a51ad85922/ceb42742-eaaa-4867-aa54-da525629aae4.meta
getfattr: Removing leading '/' from absolute path names
# file:
gluster_bricks/engine/engine/372501f5-062c-4790-afdb-dd7e761828ac/images/4e3e8ca5-0edf-42ae-ac7b-e9a51ad85922/ceb42742-eaaa-4867-aa54-da525629aae4.meta
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.signature=0x010200000000000000413e794bfbaf54b98bc00df95ce540fb6affe56ab5f5ddbb1fdb9eec096e0232
trusted.bit-rot.version=0x02000000000000005e09d553000151af
trusted.gfid=0xd36b1a8f63bc4a4bbcd0433882866733
trusted.gfid2path.ff2d749198341aff=0x32393564303861372d386437352d343638392d393239332d3339336434346362656233342f63656234323734322d656161612d343836372d616135342d6461353235363239616165342e6d657461
trusted.glusterfs.mdata=0x010000000000000000000000005e1df4a80000000020b618bf000000005e1df4a80000000020a1633c000000005e1df4a80000000020950ed4
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000000001ad000000000000000000000000000000010000000000000000
trusted.pgfid.295d08a7-8d75-4689-9293-393d44cbeb34=0x00000001
searching the gluster logs for
ceb42742-eaaa-4867-aa54-da525629aae4.meta I see:
repo0:
./rhev-data-center-mnt-glusterSD-repo0:_engine.log:The message "E
[MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-engine-replicate-0: Gfid mismatch detected for
<gfid:295d08a7-8d75-4689-9293-393d44cbeb34>/ceb42742-eaaa-4867-aa54-da525629aae4.meta>,
d36b1a8f-63bc-4a4b-bcd0-433882866733 on engine-client-2 and
b85edc18-7d59-4872-a594-c25419154d05 on engine-client-1." repeated 5
times between [2020-01-28 23:01:22.912513] and [2020-01-28
23:02:10.907716]
./rhev-data-center-mnt-glusterSD-repo0:_engine.log:[2020-01-28
23:24:12.808924] E [MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-engine-replicate-0: Gfid mismatch detected for
<gfid:295d08a7-8d75-4689-9293-393d44cbeb34>/ceb42742-eaaa-4867-aa54-da525629aae4.meta>,
d36b1a8f-63bc-4a4b-bcd0-433882866733 on engine-client-2 and
b85edc18-7d59-4872-a594-c25419154d05 on engine-client-1.
repo1:
nothing
repo2:
./rhev-data-center-mnt-glusterSD-repo0:_engine.log:The message "E
[MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-engine-replicate-0: Gfid mismatch detected for
<gfid:295d08a7-8d75-4689-9293-393d44cbeb34>/ceb42742-eaaa-4867-aa54-da525629aae4.meta>,
d36b1a8f-63bc-4a4b-bcd0-433882866733 on engine-client-2 and
b85edc18-7d59-4872-a594-c25419154d05 on engine-client-1." repeated 23
times between [2020-01-29 15:42:46.201849] and [2020-01-29
15:44:36.873793]
./rhev-data-center-mnt-glusterSD-repo0:_engine.log:[2020-01-29
15:44:47.016466] E [MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-engine-replicate-0: Gfid mismatch detected for
<gfid:295d08a7-8d75-4689-9293-393d44cbeb34>/ceb42742-eaaa-4867-aa54-da525629aae4.meta>,
d36b1a8f-63bc-4a4b-bcd0-433882866733 on engine-client-2 and
b85edc18-7d59-4872-a594-c25419154d05 on engine-client-1.
So it looks like a split brain issue according to the log message.
However,
*Why doesn’t heal info show a split brain condition?
*Why does the logs for repo1 not have anything concerning
ceb42742-eaaa-4867-aa54-da525629aae4.meta?
*If repo0 and repo1 match, why is there a split brain issue?
I think for some reason setting of AFR xattrs on the parent dir did not
happen, which is why the files are stuck in split-brain (instead of
getting recreated on repo2 using the files from repo0 or 1). You can
resolve it using the split-brain CLI, eg: `gluster volume heal $volname
split-brain source-brick repo0:/gluster_bricks/engine/engine
/372501f5-062c-4790-afdb-dd7e761828ac/images/968daf61-6858-454a-9ed4-3d3db2ae1805/4317dd0d-fd35-4176-9353-7ff69e3a8dc3.meta`
Thanks,
Ravi
‘Gluster peer status’ on each node shows connected to each of the
other 2 nodes.
‘Gluster volume heal engine info’ on each node shows each brick is
connected.
‘Gluster status engine’ on each node shows all 3 bricks as online, all
3 self-heal daemons as online, all 3 bitrot daemons online, and all 3
scrubber daemons online.
Thanks,
Jason
CONFIDENTIALITY NOTICE: This email and any attachments are for the
sole use of the intended recipient and may contain material that is
proprietary, confidential, privileged or otherwise legally protected
or restricted under applicable government laws. Any review,
disclosure, distributing or other use without expressed permission of
the sender is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies without
reading, printing, or saving.
________
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users
________
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users
