Justin, I don’t *think* that a single broker which is restarted by K8S will meet our needs. Depending on where the replacement pod is created, this can take longer than our RPC timeouts, which default to one minute. I’m hoping we’ll find a solution where failover happens within a few seconds. We’ll pursue the “2 brokers using shared storage” model, if that’s the recommended approach.
john [rg] <https://www.redpointglobal.com/> John Lilley Data Management Chief Architect, Redpoint Global Inc. 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: Justin Bertram <jbert...@apache.org> Sent: Sunday, January 7, 2024 10:00 PM To: users@activemq.apache.org Subject: Re: Looking for HA/replication boilerplate broker.xml and advice *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** One of the nice things about cloud environments that use K8s (among other things) is that they can restart services automatically, even in the event of a complete pod hardware failure. Many folks rely on this capability in lieu of broker-specific HA. If this suits your needs, then that's what I recommend. This keeps the broker configuration & management extremely simple. If that doesn't suit your needs then the next best approach I'd recommend is 2 brokers using shared storage (e.g. on a persistent volume claim) HA configuration. After that, 2 brokers using replication with ZooKeeper would be my recommendation. For what it's worth, I brought this up previously because you said, "We want to configure Artemis for HA using 'replication' strategy." I'm not sure what you mean by a "triple-replica-set." I'm currently working on a big documentation update that will hopefully make a lot of this stuff more clear. Justin On Sun, Jan 7, 2024 at 5:55 PM John Lilley <john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>> wrote: Sorry, one more thing, you mentioned “single pair of brokers with replication”… does that imply that a triple-replica-set would be immune from the split-brain issue? john [rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,UdAdIlXCSpZhbOZm3MBjb3smRBXJINPoRzR1j6gVPtriZo138Fcw1ImEVaibM79XPDsDXG5SnBrg8K2cOKbzWZXiPKWa4c9DH8EQaNlqDT6nKTYXqR3I_LAV_g,,&typo=1> John Lilley Data Management Chief Architect, Redpoint Global Inc. 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: John Lilley Sent: Sunday, January 7, 2024 4:11 PM To: users@activemq.apache.org<mailto:users@activemq.apache.org> Subject: RE: Looking for HA/replication boilerplate broker.xml and advice Justin, Regarding your zookeeper advice we are on Kubernetes -- is there further configuration doc/advice/example when we add K8S into the mix? This is only Azure AKS for now. And we are not wedded to the replication strategy. I while back I *thought* I read it was preferred to shared-storage, but perhaps I’ve misremembered. Is there a “recommended best practice” for Artemis HA on K8S? We’re not after the very best performance, our goals are more like, ease of configuration and management combined with resilience. Thanks john From: Justin Bertram <jbert...@apache.org<mailto:jbert...@apache.org>> Sent: Friday, December 22, 2023 1:38 PM To: users@activemq.apache.org<mailto:users@activemq.apache.org> Subject: Re: Looking for HA/replication boilerplate broker.xml and advice *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** Which configuration is "better" really depends on your use-case. Failback is a nice feature when you have, for example, invested more money into the hardware of your primary broker since failback ensures the primary will take over as soon as it comes back online. Keep in mind that running a single pair of brokers with replication is not recommended due to the risk of split brain. The recommended approach to mitigate split brain is by using ZooKeeper. See the "zookeeper-single-pair-failback" example for more details on that. Justin On Fri, Dec 22, 2023 at 2:13 PM John Lilley <john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>> wrote: Thanks again for the help! FYI I had to add this line to the master’s broker.xml from the example you mentioned: <check-for-live-server>true</check-for-live-server> Otherwise, a failed master that came back would start to fight with the still-active slave. I think I could have also told the slave to “fail-back”? Is there a recommendation for which is better? John [rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,5XwtFHoe_7j3xickhrC1SclF26aQUzGQ0J29kY8MWbiROJpJYHoK1iBBFbGN7Hq7nM9QaXP7xw27D-CApgn18_NYHaqXexG91GvTMH7omSBqy6Zaxg,,&typo=1> John Lilley Data Management Chief Architect, Redpoint Global Inc. 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: Justin Bertram <jbert...@apache.org<mailto:jbert...@apache.org>> Sent: Thursday, December 21, 2023 3:42 PM To: users@activemq.apache.org<mailto:users@activemq.apache.org> Subject: Re: Looking for HA/replication boilerplate broker.xml and advice *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** > ...is it possible to run both master and slave on the same host, for dev/test? Yes. > Like… configure them with different ports? Yes. This is what the examples do which I mentioned in my previous email. You can also use the --port-offset switch with the create command. Justin On Thu, Dec 21, 2023 at 4:36 PM John Lilley <john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>> wrote: Follow up question: is it possible to run both master and slave on the same host, for dev/test? Like… configure them with different ports? Thanks John [rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,d_k8uFQLvdcbwVXF9dBwVRuLOnnnAIAoM3Jv3uUG5zr_o_vvFSzAuxMjv0GmhSyd2QND75uDoO8klh83_xBW-O68X2u6v9hYK06kHtBxXML-5OTwo3xNxo8,&typo=1> John Lilley Data Management Chief Architect, Redpoint Global Inc. 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
