Justin, I see this repeatedly in the log while in DEBUG:
… 2025-05-01 19:58:29,084 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] trying to lock position: 1 2025-05-01 19:58:29,086 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] failed to lock position: 1 2025-05-01 19:58:29,086 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] lock: null 2025-05-01 19:58:29,587 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] trying to lock position: 1 2025-05-01 19:58:29,588 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] failed to lock position: 1 2025-05-01 19:58:29,588 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] lock: null 2025-05-01 19:58:30,089 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] trying to lock position: 1 2025-05-01 19:58:30,090 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] failed to lock position: 1 2025-05-01 19:58:30,091 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] lock: null 2025-05-01 19:58:30,591 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] trying to lock position: 1 2025-05-01 19:58:30,593 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] failed to lock position: 1 2025-05-01 19:58:30,593 DEBUG [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] lock: null … Regards, William Crowell From: Justin Bertram <jbert...@apache.org> Date: Thursday, May 1, 2025 at 3:54 PM To: users@activemq.apache.org <users@activemq.apache.org> Subject: Re: Apache Artemis 2.40.0: Strange File Locking Behavior On NFSv4 > Is there a way to set the lock check interval of org.apache.activemq.artemis.core.server.impl.FileLockNodeManager? No. > It seems like it checks several times a second according to the debug logs and the source code. It's hard-coded to run every 2 seconds. See here [1]. Where do you see in the source code that it will check "several times a second"? Justin [1] https://github.com/apache/activemq-artemis/blob/bfb408f5905ad9ef5f0375d466894c276b09cca5/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/impl/FileLockNodeManager.java#L60 On Thu, May 1, 2025 at 2:26 PM William Crowell <wcrow...@perforce.com.invalid> wrote: > Justin, > > AWS is using NFSv4.2 (and also tested 4.1). The mount options I am using > on AWS which work fine: > > 1.1.1.2:/data on /data type nfs4 > (rw,relatime,sync,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,soft,noac,proto=tcp,timeo=50,retrans=3,sec=sys,clientaddr=1.1.1.3,lookupcache=none,local_lock=none,addr=1.1.1.2) > > The NFS mount options on environment that I am having the problem with: > > > vers=4.1,defaults,lazytime,noatime,nodiratime,rsize=1048576,wsize=1048576,sync,intr,noac > > Is there a way to set the lock check interval of > org.apache.activemq.artemis.core.server.impl.FileLockNodeManager? It seems > like it checks several times a second according to the debug logs and the > source code. > > This is some configuration issue or bug with NFS or network related issue > as you stated. > > Regards, > > William Crowell > > From: Justin Bertram <jbert...@apache.org> > Date: Thursday, May 1, 2025 at 2:27 PM > To: users@activemq.apache.org <users@activemq.apache.org> > Subject: Re: Apache Artemis 2.40.0: Strange File Locking Behavior On NFSv4 > > Everything works fine on AWS, and I cannot reproduce the issue. > > This part is clear as you explained it previously. What isn't clear is > whether AWS is using NFS and if so what mount options it is using and how > those mount options differ from what you're using in the environment with > the problem. > > It's also not clear exactly what mount options you've tried in the > environment with the problem. > > > The issue is that the backup is taking away the lock from the primary > when it is not supposed to. It’s like the backup host gets impatient and > just grabs the lock. > > I'm not sure this is an accurate way to describe what's happening. Here's > what's happening behind the scenes with the broker... > > The primary and the backup rely on an exclusive file lock from the > underlying filesystem implementation to prevent running at the same time. > When the primary activates it acquires this lock and when the backup > activates it attempts to acquire it as well. Since the primary already has > it the backup can't acquire it. The backup then pauses activation and polls > the lock attempting to acquire it. Once it acquires the lock it completes > its activation process which involves updating the timestamp of the lock > file. Meanwhile the primary who originally acquired the lock monitors the > integrity of the lock to ensure it hasn't lost it. When the backup acquires > the lock updates the timestamp of the lock file the primary notices and > realizes that it has lost the lock and shuts itself down to avoid > split-brain. > > The issue here isn't that the backup got impatient and stole the lock from > the primary. The issue is that the file-system allowed the backup to > acquire the lock, and that reason is not clear. Perhaps there was some kind > of network failure between the primary and the NFS mount that caused the > lock to fail. Perhaps there's an NFS misconfiguration or even an NFS bug > that caused it. Perhaps the issue is in a virtualization layer (assuming > that applies). Perhaps it's because all the relevant clocks aren't > synchronized. > > > Sent you the logs directly. > > Thanks for that. I reviewed the logs and the broker appears to be working > as designed. There's no smoking gun. For some reason the filesystem simply > allows the backup to acquire the lock after a few minutes of denying it. > > At this point I don't see any evidence that the broker could be configured > any differently to prevent this issue. > > > Justin > > On Thu, May 1, 2025 at 12:04 PM William Crowell > <wcrow...@perforce.com.invalid> wrote: > > > Justin, > > > > Everything works fine on AWS, and I cannot reproduce the issue. I am > > working with someone who has on-premise hosts. > > > > The issue is that the backup is taking away the lock from the primary > when > > it is not supposed to. It’s like the backup host gets impatient and just > > grabs the lock. Sent you the logs directly. > > > > All options are valid in that link except for intr which was removed. I > > am verifying the Linux kernel build version, but it should be very > recent. > > > > Regards, > > > > William Crowell > > > > From: Justin Bertram <jbert...@apache.org> > > Date: Thursday, May 1, 2025 at 12:57 PM > > To: users@activemq.apache.org <users@activemq.apache.org> > > Subject: Re: Apache Artemis 2.40.0: Strange File Locking Behavior On > NFSv4 > > > Those option listed in the link regarding NFS that will work is sync. > > > > I'm not following you here. Which exact options are you referring to? > > > > > intr was removed and ignored in later versions of the Linux kernel. > > > > That's correct. It remains in the recommended settings because we don't > > know what version of the Linux kernel folks are using. > > > > > I did try mounting the 2 brokers are mounting using these options (on > > AWS)...I was not able to reproduce the issue on AWS. > > > > I'm lost here. Can you clarify exactly what is and isn't working on AWS > and > > in your problematic environment? > > > > > > Justin > > > > On Thu, May 1, 2025 at 11:36 AM William Crowell > > <wcrow...@perforce.com.invalid> wrote: > > > > > Justin, > > > > > > Yes, already ahead of you on this. > > > > > > We are also checking if the clocks are in sync between the brokers and > > the > > > NFS share. I know you can have some drift in clock between the > servers, > > > but it cannot be very large. > > > > > > Those option listed in the link regarding NFS that will work is sync. > > > intr was removed and ignored in later versions of the Linux kernel. > > > > > > I did try mounting the 2 brokers are mounting using these options (on > > AWS): > > > > > > > > > sudo mount -o > > > vers=4.1,soft,sync,intr,noac,lookupcache=none,timeo=50,retrans=3 -t > nfs > > > 1.1.1.4:/data /data > > > > > > I was not able to reproduce the issue on AWS. > > > > > > If you want the full logs, then I can upload those to gist or > something. > > > Working on this. > > > > > > This is what I am seeing on the other system having the problem. Here > is > > > the primary: > > > > > > … > > > 2025-05-01 15:07:15,441 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] Lock > > > appears to be valid; double check by reading status > > > 2025-05-01 15:07:15,441 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > > getting > > > state... > > > 2025-05-01 15:07:15,441 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > trying > > > to lock position: 0 > > > 2025-05-01 15:07:15,441 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > locked > > > position: 0 > > > 2025-05-01 15:07:15,441 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > lock: > > > sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid] > > > 2025-05-01 15:07:15,442 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > > state: L > > > (THIS MEANS ACTIVE) > > > 2025-05-01 15:07:15,442 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] Lock > > > appears to be valid; triple check by comparing timestamp > > > 2025-05-01 15:07:15,444 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] Lock > > > file /data/server.lock originally locked at > 2025-05-01T15:06:16.787+0000 > > > was modified at 2025-05-01T15:07:05.184+0000 > > > 2025-05-01 15:07:15,445 WARN > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] Lost > > the > > > lock according to the monitor, notifying listeners > > > 2025-05-01 15:07:15,445 ERROR [org.apache.activemq.artemis.core.server] > > > AMQ222010: Critical IO Error, shutting down the server. file=Lost > > > NodeManager lock, message=NULL > > > java.io.IOException: lost lock > > > … > > > > > > On the backup it tries once a second to get the lock and can’t which is > > > good, but something finally happens at 3:07:05pm, and it mistakenly > > thinks > > > it can get the lock: > > > > > > … > > > 2025-05-01 15:07:05,183 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > locked > > > position: 0 > > > 2025-05-01 15:07:05,183 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > lock: > > > sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid] > > > 2025-05-01 15:07:05,183 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > > state: L > > > (THIS MEANS ACTIVE) > > > 2025-05-01 15:07:05,184 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > > acquired > > > primary node lock state = L > > > 2025-05-01 15:07:05,186 DEBUG > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > > touched > > > /data/server.lock; new time: 1746112025184 > > > … > > > > > > Regards, > > > > > > William Crowell > > > > > > From: Justin Bertram <jbert...@apache.org> > > > Date: Thursday, May 1, 2025 at 11:44 AM > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > Subject: Re: Apache Artemis 2.40.0: Strange File Locking Behavior On > > NFSv4 > > > Would you mind turning on DEBUG logging for > > > org.apache.activemq.artemis.core.server.impl.FileLockNodeManager [1], > > > reproducing, and uploading the full logs someplace accessible? > > > > > > Regarding your NFS mount options...I was perusing the NFS man page [2] > > and > > > saw this note about noatime & nodiratime: > > > > > > In particular, the atime/noatime, diratime/nodiratime, > > > relatime/norelatime, and strictatime/nostrictatime mount options have > no > > > effect on NFS mounts. > > > > > > I could not find any reference to lazytime, and I also noticed that you > > > weren't using all the recommendations from the ActiveMQ Artemis > > > documentation. I wonder if you might try something like this: > > > > > > vers=4.1,soft,sync,intr,noac,lookupcache=none,timeo=50,retrans=3 > > > > > > Is NFS being used in your AWS use-case? If so, what mount options are > > being > > > used? > > > > > > To be clear, I'm not an NFS expert by any means. Usually this stuff > just > > > works with the recommended settings. > > > > > > > > > Justin > > > > > > [1] > > > > > > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Flogging.html%23configuring-a-specific-level-for-a-logger&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762583175%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Zc5xRwWItteckduPiKucA47xHavuvxc00Ct7qAD8UJI%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/logging.html#configuring-a-specific-level-for-a-logger> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Flogging.html%23configuring-a-specific-level-for-a-logger&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762608826%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=HHyQLAUSppMn9HobappD%2BmC7CgvvhJLfEitj1G2mxww%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/logging.html#configuring-a-specific-level-for-a-logger> > > > > < > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Flogging.html%23configuring-a-specific-level-for-a-logger&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762618125%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=299AeYMe6QeV1%2BcgcUucdIbDgWCNQTFTbbBYYHz371c%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/logging.html#configuring-a-specific-level-for-a-logger> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Flogging.html%23configuring-a-specific-level-for-a-logger&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762627134%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=sbJ6doSsv05SzYmw6vl3rr5eL81wWEqpZdRhcNzR8O0%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/logging.html#configuring-a-specific-level-for-a-logger> > > > > > > > > < > > > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Flogging.html%23configuring-a-specific-level-for-a-logger&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762635822%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=SkDL6DNO0MfP%2FXndgNAsMf%2B4TRtuVPZJNtfFwhC4LM4%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/logging.html#configuring-a-specific-level-for-a-logger> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Flogging.html%23configuring-a-specific-level-for-a-logger&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762644601%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2PqoYO0BN%2FCef3o96KH%2BFLoBvhVkqN061RoNgMe5tFU%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/logging.html#configuring-a-specific-level-for-a-logger> > > > > < > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Flogging.html%23configuring-a-specific-level-for-a-logger&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762653343%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Fj1gX%2BYpbE8k%2FYswgs3XLte8QfuS4jte40gF9d3Bll0%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/logging.html#configuring-a-specific-level-for-a-logger> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Flogging.html%23configuring-a-specific-level-for-a-logger&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762662107%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0eyQogeUg70F7g9Zy5F42r455RAa1r4bpAag9hoZ9KE%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/logging.html#configuring-a-specific-level-for-a-logger> > > > > > > > > > > > > [2] > > > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.man7.org%2Flinux%2Fman-pages%2Fman5%2Fnfs.5.html&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762670856%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=j3tZIoaZwpqLXNm5SbC6bGrAFuin8A3TpnHTFc9DvhQ%3D&reserved=0<https://www.man7.org/linux/man-pages/man5/nfs.5.html> > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.man7.org%2Flinux%2Fman-pages%2Fman5%2Fnfs.5.html&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762679611%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EEiMvDq2I8OyL%2FO%2F7EID084Qfvdt9nl4FjFzJdfYg3Q%3D&reserved=0<https://www.man7.org/linux/man-pages/man5/nfs.5.html>> > > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.man7.org%2Flinux%2Fman-pages%2Fman5%2Fnfs.5.html&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762688485%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=709JU%2FDO8gqnQU2hCZ44q5Jio0xTiIZnRlpyMUey0ag%3D&reserved=0<https://www.man7.org/linux/man-pages/man5/nfs.5.html> > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.man7.org%2Flinux%2Fman-pages%2Fman5%2Fnfs.5.html&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762697199%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=tfDo6p1X8nG2lpPToPFFgXQUx1XlRsct2WVHpR8yhrE%3D&reserved=0<https://www.man7.org/linux/man-pages/man5/nfs.5.html>>> > > > < > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.man7.org%2Flinux%2Fman-pages%2Fman5%2Fnfs.5.html&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762705849%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=l%2BXMKR6E0OLUSJxeZHXKhD97XiNBAnZgf2H5aZR0Xng%3D&reserved=0<https://www.man7.org/linux/man-pages/man5/nfs.5.html> > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.man7.org%2Flinux%2Fman-pages%2Fman5%2Fnfs.5.html&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762714652%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EDRGzJI8hAWAuXrrtB2o7ed6zsE9ehC7eTo58MNSLIA%3D&reserved=0<https://www.man7.org/linux/man-pages/man5/nfs.5.html>> > > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.man7.org%2Flinux%2Fman-pages%2Fman5%2Fnfs.5.html&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762723386%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JjXPPNUfVXm0m7KlxJqiYj4r%2F4yGjOWTOp4nt0dMxFI%3D&reserved=0<https://www.man7.org/linux/man-pages/man5/nfs.5.html> > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.man7.org%2Flinux%2Fman-pages%2Fman5%2Fnfs.5.html&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762732049%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MJoSWS2Sw2TYQ2YOwBA3wW%2BUKb%2B7mojGrLzjpu3ph4w%3D&reserved=0<https://www.man7.org/linux/man-pages/man5/nfs.5.html>>>> > > > > > > On Thu, May 1, 2025 at 9:02 AM William Crowell > > > <wcrow...@perforce.com.invalid> wrote: > > > > > > > Configuration and logs… > > > > > > > > The relevant broker.xml configuration from the primary (broker 1) is: > > > > > > > > … > > > > <connectors> > > > > <connector name="broker1">tcp://1.1.1.2:61616</connector> > > > > <connector name="broker2">tcp://1.1.1.3:61616</connector> > > > > </connectors> > > > > > > > > <cluster-connections> > > > > <cluster-connection name="my-cluster"> > > > > <connector-ref>broker1</connector-ref> > > > > <static-connectors> > > > > <connector-ref>broker2</connector-ref> > > > > </static-connectors> > > > > </cluster-connection> > > > > </cluster-connections> > > > > > > > > <ha-policy> > > > > <shared-store> > > > > <primary> > > > > <failover-on-shutdown>true</failover-on-shutdown> > > > > </primary> > > > > </shared-store> > > > > </ha-policy> > > > > … > > > > > > > > The relevant broker.xml configuration from the backup (broker 2) is: > > > > > > > > … > > > > <connectors> > > > > <connector name="broker1">tcp://1.1.1.2:61616</connector> > > > > <connector name="broker2">tcp://1.1.1.3:61616</connector> > > > > </connectors> > > > > > > > > <cluster-connections> > > > > <cluster-connection name="my-cluster"> > > > > <connector-ref>broker2</connector-ref> > > > > <static-connectors> > > > > <connector-ref>broker1</connector-ref> > > > > </static-connectors> > > > > </cluster-connection> > > > > </cluster-connections> > > > > > > > > <ha-policy> > > > > <shared-store> > > > > <backup> > > > > <allow-failback>false</allow-failback> > > > > </backup> > > > > </shared-store> > > > > </ha-policy> > > > > … > > > > > > > > Startup on the primary: > > > > > > > > … > > > > 2025-05-01 12:51:23,567 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221006: Waiting to obtain primary lock > > > > … > > > > 2025-05-01 12:51:23,747 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221034: Waiting indefinitely to obtain primary lock > > > > 2025-05-01 12:51:23,748 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221035: Primary Server Obtained primary lock > > > > … > > > > 2025-05-01 12:51:24,289 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221007: Server is now active > > > > … > > > > > > > > Backup is started and somehow becomes primary: > > > > > > > > … > > > > 2025-05-01 12:51:48,473 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221032: Waiting to become backup node > > > > 2025-05-01 12:51:48,474 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221033: ** got backup lock > > > > … > > > > 2025-05-01 12:51:48,659 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.40.0 > > > > [339308e1-25f3-11f0-996a-0200ec1b9c8e] started; waiting for primary > to > > > fail > > > > before activating > > > > 2025-05-01 12:51:48,809 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221031: backup announced > > > > … > > > > 2025-05-01 12:52:06,129 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221010: Backup Server is now active > > > > … > > > > 2025-05-01 12:52:07,158 INFO > [org.apache.activemq.artemis.core.client] > > > > AMQ214036: Connection closure to 1.1.1.2/1.1.1.2:61616 has been > > > detected: > > > > AMQ219015: The connection was disconnected because of server shutdown > > > > [code=DISCONNECTED] > > > > … > > > > > > > > Primary loses the lock unexpectedly and shuts down: > > > > > > > > … > > > > 2025-05-01 12:52:16,352 WARN > > > > [org.apache.activemq.artemis.core.server.impl.FileLockNodeManager] > Lost > > > the > > > > lock according to the monitor, notifying listeners > > > > 2025-05-01 12:52:16,353 ERROR > [org.apache.activemq.artemis.core.server] > > > > AMQ222010: Critical IO Error, shutting down the server. file=Lost > > > > NodeManager lock, message=NULL > > > > java.io.IOException: lost lock > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.lambda$registerActiveLockListener$0(SharedStorePrimaryActivation.java:124) > > > > ~[artemis-server-2.40.0.jar:2.40.0] > > > > at > > > > > > > > > > org.apache.activemq.artemis.core.server.NodeManager.lambda$notifyLostLock$0(NodeManager.java:167) > > > > ~[artemis-server-2.40.0.jar:2.40.0] > > > > … > > > > 2025-05-01 12:52:16,528 INFO > [org.apache.activemq.artemis.core.server] > > > > AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.40.0 > > > > [339308e1-25f3-11f0-996a-0200ec1b9c8e] stopped, uptime 52.978 seconds > > > > … > > > > > > > > Regards, > > > > > > > > William Crowell > > > > > > > > From: William Crowell <wcrow...@perforce.com.INVALID> > > > > Date: Thursday, May 1, 2025 at 9:20 AM > > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > > Subject: Apache Artemis 2.40.0: Strange File Locking Behavior On > NFSv4 > > > > Good morning, > > > > > > > > Disclaimer: This is not a bug, but a configuration issue. > > > > > > > > We are using Apache Artemis 2.40.0 on Rocky Linux 9. We are > > configuring > > > a > > > > primary/backup pair on separate hosts and putting the data directory > on > > > an > > > > NSFv4 mount, and we are experiencing problems with the locking > > mechanism. > > > > I do know that NFS is not recommended for production use, but that is > > > what > > > > we are limited to. > > > > > > > > We are following this documentation: > > > > > > > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762743654%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d0OIBbjRr%2BS1V%2FTJGI5g%2B7iQ3oC9zfe8l%2BAbysIh3xE%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762753810%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mQwxWz52shG1nUL7qQCEOppWRvmWRKiXKSRqGt94WIY%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > > > > < > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762762529%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=sFmjO0%2Fk2EKb0T60rD1WO3oDE0wrrFcl1xL0y7q3d7k%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762771112%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k7zmQ2NHbKnMze66Z9LyTWJfqarL6SBTO%2B79q1eFtpk%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > > > > > > > > < > > > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260762779604%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=BpLPdRbvPdq1pdN2qLdTe9IYYEJK0XAnLQ4HhTS6RvA%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763029032%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=oeAeq%2F83o4ZV4rlyh3o5JWlxd5k5e3jg6ZnOkjLBQms%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > > > > < > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763039351%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=R9OkooxJZB4cj0voZwdmKaWWYfrJ5C%2FTNu3ZXvjRKeo%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763048341%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=pL4mzlv8CDLwjsZZFgQXwiVH78mfLuHJtv6wok9uFKA%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > > > > > > > > > > > > > < > > > > > > > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763057450%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MiN%2FlJaTspapd%2BZnlh5QFTXqbbSFUkOXVyf8AAt%2FgaM%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763068133%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CeDZsWVxhCbLP0jX%2FGj6yYvS0hz<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations>8ZLo4hTv7NuEj5qs%3D&reserved=0 > > > > < > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763077125%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ocwnZVyN4ybPK1BOFhe3Sw27PgEzx0hmAldSxEpEplY%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763085994%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=lsmylIyV1XqibJEE%2BDKc%2BetTNzr6aBST6p115o5MsEI%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > > > > > > > > < > > > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763094575%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kVtQ8jgf6hAgpj6GVJJCOz8q5fQR3PqnbAthMjq5pf8%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763103498%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ge7CjnevDkOdvjDeRtZz3hhQrvCje3aYv6RJjsG1GAg%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > > > > < > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763112289%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=foOaCyaYJvzO0wCbk%2B3%2F49SDhbFUUti77jnpduea8yg%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fha.html%23nfs-mount-recommendations&data=05%7C02%7CWCrowell%40perforce.com%7C2babf73f5bc44d7ac6b708dd88e9fd0f%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638817260763121000%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=5uCxW25KCZ%2F7WR%2BAD9ByJ0%2F3ixJIIO1W4oAZu3iCeXQ%3D&reserved=0<https://activemq.apache.org/components/artemis/documentation/latest/ha.html#nfs-mount-recommendations> > > > > > > > > > > > > > > > > > > > > > > What is happening is that the primary loses the lock and goes down > > after > > > > the backup node was started. The mount options on both brokers we > are > > > > using are: > > > > > > > > > > > > > > > > > > vers=4.1,defaults,lazytime,noatime,nodiratime,rsize=1048576,wsize=1048576,sync,intr,noac > > > > > > > > We then tried to startup the nodes sequentially. The primary lost > the > > > > lock and went down shortly after the backup node was started. > > > > > > > > We also tested file locking on brokers 1 and 2: > > > > > > > > Broker 1: > > > > > > > > $ date; flock -x /data/test.lock -c "sleep 30"; echo $?; date > > > > Thu May 1 12:42:52 PM GMT 2025 > > > > 0 > > > > Thu May 1 12:43:22 PM GMT 2025 > > > > > > > > Broker 2: > > > > > > > > date; flock -n /data/test.lock -c "echo lock acquired"; echo $?; > date > > > > Thu May 1 12:42:46 PM GMT 2025 > > > > 1 > > > > Thu May 1 12:42:47 PM GMT 2025 > > > > > > > > This means that broker 2 was unable to acquire the lock because > broker > > 1 > > > > already had it which is not consistent with the behavior on the > Apache > > > > Artemis brokers. I also tested this on AWS, and the failover works > > fine > > > as > > > > expected. > > > > > > > > What am I missing here? > > > > > > > > Regards, > > > > > > > > William Crowell > > > > > > > > > > > > > > > > This e-mail may contain information that is privileged or > confidential. > > > If > > > > you are not the intended recipient, please delete the e-mail and any > > > > attachments and notify us immediately. > > > > > > > > > > > > This e-mail may contain information that is privileged or > confidential. > > > If > > > > you are not the intended recipient, please delete the e-mail and any > > > > attachments and notify us immediately. > > > > > > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click on links or open attachments unless you recognize the sender and > > know > > > the content is safe. > > > > > > > > > This e-mail may contain information that is privileged or confidential. > > If > > > you are not the intended recipient, please delete the e-mail and any > > > attachments and notify us immediately. > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > click on links or open attachments unless you recognize the sender and > know > > the content is safe. > > > > > > This e-mail may contain information that is privileged or confidential. > If > > you are not the intended recipient, please delete the e-mail and any > > attachments and notify us immediately. > > > > > > > CAUTION: This email originated from outside of the organization. Do not > click on links or open attachments unless you recognize the sender and know > the content is safe. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > > CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
