Greetings Thomas, I'll try to assist in determining RC and resolving the issue. The following will help me assisting you:
1. Please create a an issue on GitHub with all the relevant details, it'll be easier to track 2. Please provide all client-side, brick-side and fix-layout logs With that information I could begin an initial assessment of the situation. Thank you, On Sun, Nov 8, 2020 at 5:35 PM Thomas Bätzler <[email protected]> wrote: > Hi, > > the other day we decided to expand our gluster storage by adding two > bricks, going from a 3x2 to a 4x2 distributed-replicated setup. In order > to get to this point we had done rolling upgrades from 3.something to > 5.13 to 7.8, all without issues. We ran into a spot of trouble during > the fix-layout when both of the new nodes crashed in the space of two > days. We rebooted them and the fix-layout process seemed to cope by > starting itself again on these nodes. > > Now weird things are happening. We noticed that we can't access some > files diretly anymore. Doing a stat on them returns a file not found. > However, if we list the directory containing the files, the file is > shown as present and subsequently it can be accessed on the client that > ran the list, but not on other clients! Also, if we umount and remount, > the file is inaccessible again. > > Does anybody have any ideas what's going on here? Is there any way to > fix the volume without taking it offline for days? We have about 60T of > data online and we need that data to be consistent and available. > > OS is Debian 10 with glusterfs-server 7.8-3. > > Volume configuration: > > Volume Name: hotcache > Type: Distributed-Replicate > Volume ID: 4c006efa-6fd6-4809-93b0-28dd33fee2d2 > Status: Started > Snapshot Count: 0 > Number of Bricks: 4 x 2 = 8 > Transport-type: tcp > Bricks: > Brick1: hotcache1:/data/glusterfs/drive1/hotcache > Brick2: hotcache2:/data/glusterfs/drive1/hotcache > Brick3: hotcache3:/data/glusterfs/drive1/hotcache > Brick4: hotcache4:/data/glusterfs/drive1/hotcache > Brick5: hotcache5:/data/glusterfs/drive1/hotcache > Brick6: hotcache6:/data/glusterfs/drive1/hotcache > Brick7: hotcache7:/data/glusterfs/drive1/hotcache > Brick8: hotcache8:/data/glusterfs/drive1/hotcache > Options Reconfigured: > performance.readdir-ahead: off > diagnostics.client-log-level: INFO > diagnostics.brick-log-level: INFO > diagnostics.count-fop-hits: on > diagnostics.latency-measurement: on > server.statedump-path: /var/tmp > diagnostics.brick-sys-log-level: ERROR > storage.linux-aio: on > performance.read-ahead: off > performance.write-behind-window-size: 4MB > performance.cache-max-file-size: 200kb > nfs.disable: on > performance.cache-refresh-timeout: 1 > performance.io-cache: on > performance.stat-prefetch: off > performance.quick-read: on > performance.io-thread-count: 16 > auth.allow: * > cluster.readdir-optimize: on > performance.flush-behind: off > transport.address-family: inet > cluster.self-heal-daemon: enable > > TIA! > > Best regards, > Thomas Bätzler > -- > BRINGE Informationstechnik GmbH > Zur Seeplatte 12 > D-76228 Karlsruhe > Germany > > Fon: +49 721 94246-0 > Fon: +49 171 5438457 > Fax: +49 721 94246-66 > Web: http://www.bringe.de/ > > Geschäftsführer: Dipl.-Ing. (FH) Martin Bringe > Ust.Id: DE812936645, HRB 108943 Mannheim > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > [email protected] > https://lists.gluster.org/mailman/listinfo/gluster-users > -- *Barak Sason Rofman* Gluster Storage Development Red Hat Israel <https://www.redhat.com/> 34 Jerusalem rd. Ra'anana, 43501 [email protected] <[email protected]> T: *+972-9-7692304* M: *+972-52-4326355* @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/redhat.il/> <https://red.ht/sig>
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
