On 04/30/2016 02:57 AM, bdobre...@mirantis.com wrote:
Hi Roman. That's interesting, although’s hard to believe (there is no slave lag in galera multi master). I can only suggest us to create another jepsen test to verify exactly scenario you describe. As well as other OpenStack specific patterns.
There is definitely slave lag in Galera and it can be controlled using the wsrep_causal_reads_flag.
Demonstration script, whose results I have confirmed separately using Pythons scripts, is at:
https://www.percona.com/blog/2013/03/03/investigating-replication- latency-in-percona-xtradb-cluster/
Regards, Bogdan. *Od:* Roman Podoliaka <mailto:rpodoly...@mirantis.com> *Wysłano:* piątek, 29 kwietnia 2016 21:04 *Do:* OpenStack Development Mailing List (not for usage questions) <mailto:openstack-dev@lists.openstack.org> *DW:* openstack-operat...@lists.openstack.org <mailto:openstack-operat...@lists.openstack.org> Hi Bogdan, Thank you for sharing this! I'll need to familiarize myself with this Jepsen thing, but overall it looks interesting. As it turns out, we already run Galera in multi-writer mode in Fuel unintentionally in the case, when the active MySQL node goes down, HAProxy starts opening connections to a backup, then the active goes up again, HAProxy starts opening connections to the original MySQL node, but OpenStack services may still have connections opened to the backup in their connection pools - so now you may have connections to multiple MySQL nodes at the same time, exactly what you wanted to avoid by using active/backup in the HAProxy configuration. ^ this actually leads to an interesting issue [1], when the DB state committed on one node is not immediately available on another one. Replication lag can be controlled via session variables [2], but that does not always help: e.g. in [1] Nova first goes to Neutron to create a new floating IP, gets 201 (and Neutron actually *commits* the DB transaction) and then makes another REST API request to get a list of floating IPs by address - the latter can be served by another neutron-server, connected to another Galera node, which does not have the latest state applied yet due to 'slave lag' - it can happen that the list will be empty. Unfortunately, 'wsrep_sync_wait' can't help here, as it's two different REST API requests, potentially served by two different neutron-server instances. Basically, you'd need to *always* wait for the latest state to be applied before executing any queries, which Galera is trying to avoid for performance reasons. Thanks, Roman [1] https://bugs.launchpad.net/fuel/+bug/1529937 [2] http://galeracluster.com/2015/06/achieving-read-after-write-semantics-with-galera/ On Fri, Apr 22, 2016 at 10:42 AM, Bogdan Dobrelya <bdobre...@mirantis.com> wrote: > [crossposting to openstack-operat...@lists.openstack.org] > > Hello. > I wrote this paper [0] to demonstrate an approach how we can leverage a > Jepsen framework for QA/CI/CD pipeline for OpenStack projects like Oslo > (DB) or Trove, Tooz DLM and perhaps for any integration projects which > rely on distributed systems. Although all tests are yet to be finished, > results are quite visible, so I better off share early for a review, > discussion and comments. > > I have similar tests done for the RabbitMQ OCF RA clusterers as well, > although have yet wrote a report. > > PS. I'm sorry for so many tags I placed in the topic header, should I've > used just "all" :) ? Have a nice weekends and take care! > > [0] https://goo.gl/VHyIIE > > -- > Best regards, > Bogdan Dobrelya, > Irc #bogdando > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
