Hi, On 9/4/20 2:29 PM, Alexandre DERUMIER wrote: > BTW, > > do you think it could be possible to add an extra optionnal layer of security > check, not related to corosync ? > > I'm still afraid of this corosync bug since years, and still don't use HA. > (or I have tried to enable it 2months ago,and this give me a disaster > yesterday..)
Using corosync to enable HA is a bit scary TBH, not sure how people solve this, but I'd rather have the logic outside the cluster so it can make a solid decision whether a node should be rebooted or not, and choose to reboot via iDRAC if all checks fail. Maybe use the metrics support to facilitate this? > Something like an extra heartbeat between nodes daemons, and check if we also > have quorum with theses heartbeats ? > > > > ----- Mail original ----- > De: "aderumier" <aderum...@odiso.com> > À: "pve-devel" <pve-de...@pve.proxmox.com> > Envoyé: Jeudi 3 Septembre 2020 16:11:56 > Objet: [pve-devel] corosync bug: cluster break after 1 node clean shutdown > > Hi, > > I had a problem this morning with corosync, after shutting down cleanly a > node from the cluster. > This is a 14 nodes cluster, node2 was shutted. (with "halt" command) > > HA was actived, and all nodes have been rebooted. > (I have already see this problem some months ago, without HA enabled, and it > was like half of the nodes didn't see others nodes, like 2 cluster > formations) > > Some users have reported similar problems when adding a new node > https://forum.proxmox.com/threads/quorum-lost-when-adding-new-7th-node.75197/ > > > Here the full logs of the nodes, for each node I'm seeing quorum with 13 > nodes. (so it's seem to be ok). > I didn't have time to look live on server, as HA have restarted them. > So, I don't known if it's a corosync bug or something related to > crm,lrm,pmxcs (but I don't see any special logs) > > Only node7 have survived a little bit longer,and I had stop lrm/crm to avoid > reboot. > > libknet1: 1.15-pve1 > corosync: 3.0.3-pve1 > > > Any ideas before submiting bug to corosync mailing ? (I'm seeing new libknet > && corosync version on their github, I don't have read the changelog yet) > > > > > node1 > ----- > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: members: 1/3527, 3/27373, > 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, > 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm1 corosync[3678]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm1 corosync[3678]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm1 corosync[3678]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [dcdb] notice: cpg_send_message retried 1 > times > Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: received all states > Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 has no active > links > > --> reboot > > node2 : shutdown log > ----- > Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd received signal 15: Terminated > Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_00] > [SAT], state written to > /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302Y9120CGN.ata.state > > Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_01] > [SAT], state written to > /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302MG120CGN.ata.state > > Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd is exiting (exit status 0) > Sep 3 10:39:05 m6kvm2 nrpe[20095]: Caught SIGTERM - shutting down... > Sep 3 10:39:05 m6kvm2 nrpe[20095]: Daemon shutdown > Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> starting task > UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam: > Sep 3 10:39:07 m6kvm2 pve-guests[38552]: all VMs and CTs stopped > Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> end task > UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam: OK > Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: received signal TERM > Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server closing > Sep 3 10:39:07 m6kvm2 spiceproxy[36572]: worker exit > Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: worker 36572 finished > Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server stopped > Sep 3 10:39:07 m6kvm2 pvestatd[3786]: received signal TERM > Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server closing > Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server stopped > Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: received signal TERM > Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: got shutdown request with shutdown > policy 'conditional' > Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: shutdown LRM, stop all services > Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: received terminate request > (signal) > Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: stopping pvefw logger > Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: watchdog closed (disabled) > Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: server stopped > Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: received signal TERM > Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: server received shutdown request > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: received signal TERM > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server closing > Sep 3 10:39:10 m6kvm2 pveproxy[29731]: worker exit > Sep 3 10:39:10 m6kvm2 pveproxy[30873]: worker exit > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 30873 finished > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 31391 finished > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 29731 finished > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server stopped > Sep 3 10:39:14 m6kvm2 pve-ha-crm[12847]: server stopped > > > node3 > ----- > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm3 corosync[30580]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm3 corosync[30580]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm3 corosync[30580]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: cpg_send_message retried > 1 times > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: received all states > Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 has no active > links > > > node4 > ----- > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: members: 1/3527, 3/27373, > 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, > 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm4 corosync[4085]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm4 corosync[4085]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: received all states > Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: all data is up to date > Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 has no active > links > Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:20 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > > node5 > ----- > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: members: 1/3527, 3/27373, > 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, > 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm5 corosync[41830]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm5 corosync[41830]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: received all states > Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 has no active > links > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > > > > > > node6 > ----- > Sep 3 10:39:16 m6kvm6 corosync[36694]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm6 corosync[36694]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 has no active > links > > > > node7 > ----- > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3527, 3/27373, > 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, > 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm7 corosync[15467]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: received all states > Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date > Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 has no active > links > Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:20 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > ---> here the others nodes reboot almost at the same time > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 3 link: 0 is down > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 14 link: 0 is down > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 has no active > links > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 has no active > links > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 13 link: 0 is down > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 8 link: 0 is down > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 6 link: 0 is down > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 10 link: 0 is down > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 has no active > links > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 has no active > links > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 has no active > links > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 has no active > links > Sep 3 10:40:28 m6kvm7 corosync[15467]: [TOTEM ] Token has not been received > in 4505 ms > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 11 link: 0 is down > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 9 link: 0 is down > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 has no active > links > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 has no active > links > Sep 3 10:40:30 m6kvm7 corosync[15467]: [TOTEM ] A processor failed, forming > new configuration. > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 4 link: 0 is down > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 12 link: 0 is down > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 1 link: 0 is down > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 has no active > links > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 has no active > links > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 has no active > links > Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] link: host: 5 link: 0 is down > Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 (passive) best > link: 0 (pri: 1) > Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 has no active > links > Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] A new membership (7.82f) was > formed. Members left: 1 3 4 5 6 8 9 10 11 12 13 14 > Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] Failed to receive the leave > message. failed: 1 3 4 5 6 8 9 10 11 12 13 14 > Sep 3 10:40:41 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 12 received > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892 > Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] This node is within the > non-primary component and will NOT provide any services. > Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] Members[1]: 7 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:40:41 m6kvm7 corosync[15467]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: dfsm_deliver_queue: queue > length 51 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 1/3527 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 1/3527 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 1/3527 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 1/3527 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 4/3903 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 4/3903 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 4/3903 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 4/3903 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 9/22810 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 9/22810 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 9/22810 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 9/22810 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 6/37248 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 6/37248 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 6/37248 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 6/37248 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 11/12983 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 11/12983 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 11/12983 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 11/12983 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 10/41940 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 10/41940 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 10/41940 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 10/41940 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 5/42272 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 5/42272 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 5/42272 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 13/39678 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 13/39678 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 13/39678 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 13/39678 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 8/24790 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 8/24790 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 8/24790 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 8/24790 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 12/44214 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 12/44214 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 12/44214 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 12/44214 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 14/42930 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 14/42930 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 14/42930 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from > non-member 14/42930 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: node lost quorum > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: members: 7/15892 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: received write while not > quorate - trigger resync > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: leaving CPG group > Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from > 'online' => 'unknown' > Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] notice: start cluster connection > Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: cpg_join failed: 14 > Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: can't initialize service > Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892 > Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during > cfs-locked 'domain-ha' operation: no quorum! > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds) > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds) > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs > lock update failed - Permission denied > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - > cfs lock update failed - Permission denied > Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active => > lost_agent_lock > Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master => > lost_manager_lock > Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled) > Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock => > wait_for_quorum > Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] rx: host: 4 link: 0 is up > Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best > link: 0 (pri: 1) > Sep 3 10:43:24 m6kvm7 corosync[15467]: [TOTEM ] A new membership (4.834) was > formed. Members joined: 4 > Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 4/3965, 7/15892 > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data > syncronisation > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: members: 4/3965, > 7/15892 > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: starting data > syncronisation > Sep 3 10:43:24 m6kvm7 corosync[15467]: [QUORUM] Members[2]: 4 7 > Sep 3 10:43:24 m6kvm7 corosync[15467]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request > (epoch 4/3965/00000002) > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received sync request > (epoch 4/3965/00000002) > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 7/15892 > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 7/15892 > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: start sending inode > updates > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: sent all (4) updates > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received all states > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date > Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] rx: host: 3 link: 0 is up > Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best > link: 0 (pri: 1) > Sep 3 10:44:05 m6kvm7 corosync[15467]: [TOTEM ] A new membership (3.838) was > formed. Members joined: 3 > Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 3/3613, 4/3965, > 7/15892 > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data > syncronisation > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: members: 3/3613, > 4/3965, 7/15892 > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: starting data > syncronisation > Sep 3 10:44:05 m6kvm7 corosync[15467]: [QUORUM] Members[3]: 3 4 7 > Sep 3 10:44:05 m6kvm7 corosync[15467]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request > (epoch 3/3613/00000002) > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received sync request > (epoch 3/3613/00000002) > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 4/3965 > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 4/3965, > 7/15892 > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received all states > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: dfsm_deliver_queue: > queue length 1 > Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] rx: host: 1 link: 0 is up > Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best > link: 0 (pri: 1) > Sep 3 10:44:31 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.83c) was > formed. Members joined: 1 > Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3552, 3/3613, > 4/3965, 7/15892 > Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data > syncronisation > Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3552, > 3/3613, 4/3965, 7/15892 > Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: starting data > syncronisation > Sep 3 10:44:31 m6kvm7 corosync[15467]: [QUORUM] Members[4]: 1 3 4 7 > Sep 3 10:44:31 m6kvm7 corosync[15467]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request > (epoch 1/3552/00000002) > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received sync request > (epoch 1/3552/00000002) > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 3/3613 > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 3/3613, > 4/3965, 7/15892 > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received all states > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date > Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:49:18 m6kvm7 pmxcfs[15892]: [status] notice: received log > > > > > node8 > ----- > > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: members: 1/3527, 3/27373, > 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, > 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm8 corosync[24361]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm8 corosync[24361]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: received all states > Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 has no active > links > Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log > Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log > Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log > Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log > > > --> reboot > > > > node9 > ----- > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: members: 1/3527, 3/27373, > 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, > 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm9 corosync[22340]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm9 corosync[22340]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: received all states > Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: all data is up to date > Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 has no active > links > Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:20 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > > > --> reboot > > > > node10 > ------ > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm10 corosync[41458]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm10 corosync[41458]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: received all states > Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: all data is up to date > Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 has no active > links > Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log > Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log > > --> reboot > > > node11 > ------ > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm11 corosync[12455]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm11 corosync[12455]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: received all states > Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 has no active > links > Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log > Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log > Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log > Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log > Sep 3 10:40:20 m6kvm11 pmxcfs[12983]: [status] notice: received log > > > > > node12 > ------ > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm12 corosync[43716]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm12 corosync[43716]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [dcdb] notice: cpg_send_message retried > 1 times > Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: received all states > Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 has no active > links > Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:20 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > > > > > > node13 > ------ > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm13 corosync[39182]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm13 corosync[39182]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: cpg_send_message retried > 1 times > Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: received all states > Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: all data is up to date > Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 has no active > links > Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log > Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log > Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log > Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log > --> reboot > > node14 > ------ > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: members: 1/3527, > 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, > 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: starting data > syncronisation > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: received sync request > (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm14 corosync[42413]: [TOTEM ] A new membership (1.82b) was > formed. Members left: 2 > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 > 10 11 12 13 14 > Sep 3 10:39:16 m6kvm14 corosync[42413]: [MAIN ] Completed service > synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: received all states > Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 (passive) best > link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 has no active > links > > --> reboot > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel -- Med vänliga hälsningar Josef Johansson _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
