Hello,
I just tested again after installing the latest updates, unfortunately the
problem persists:
kernel: Tainted: P O 4.15.18-9 -pve # 1
drbd-dkms 9.0.16-1
drbd-utils 9.6.0-1
drbdtop 0.2.1-1
linstor-client 0.7.2-1
linstor-common 0.7.3-1
linstor-controller 0.7.3-1
linstor-proxmox 3.0.2-2
linstor-satellite 0.7.3-1
python-linstor 0.7.2-1
pve-manager / 5.2-11 / 13c2da63 (running kernel: 4.15.18-9-pve)
Here are the details of the errors as requested:
Thank you,
TASK : VM 100 - Snapshot
SUCCESS:
Description:
New resource definition 'vm-100-state-test_snap_14_08' created.
Details:
Resource definition 'vm-100-state-test_snap_14_08' UUID is:
aa2b8e23-ffeb-4e2e-810c-66c4e252f575
SUCCESS:
Description:
Resource definition 'vm-100-state-test_snap_14_08' modified.
Details:
Resource definition 'vm-100-state-test_snap_14_08' UUID is:
aa2b8e23-ffeb-4e2e-810c-66c4e252f575
SUCCESS:
New volume definition with number '0' of resource definition
'vm-100-state-test_snap_14_08' created.
SUCCESS:
Description:
Resource 'vm-100-state-test_snap_14_08' successfully autoplaced on 2 nodes
Details:
Used storage pool: 'drbdpool'
Used nodes: 'vpx3-2', 'vpx3-1'
SUCCESS:
Created resource 'vm-100-state-test_snap_14_08' on 'vpx3-2'
SUCCESS:
Created resource 'vm-100-state-test_snap_14_08' on 'vpx3-1'
SUCCESS:
Description:
Resource 'vm-100-state-test_snap_14_08' on 'vpx3-2' ready
Details:
Self-placing resource: vm-100-state-test_snap_14_08
SUCCESS:
Description:
Resource 'vm-100-state-test_snap_14_08' on 'vpx3-1' ready
Details:
Self-placing resource: vm-100-state-test_snap_14_08
SUCCESS:
Description:
New snapshot 'snap_vm-100-disk-2_test_snap_14_08' of resource 'vm-100-disk-2'
registered.
Details:
Snapshot 'snap_vm-100-disk-2_test_snap_14_08' of resource 'vm-100-disk-2' UUID
is: 6b944397-6536-4fc7-85a1-05d69f580c30
SUCCESS:
Suspended IO of 'vm-100-disk-2' on 'vpx3-2' for snapshot
SUCCESS:
Suspended IO of 'vm-100-disk-2' on 'vpx3-1' for snapshot
ERROR:
(Node: 'vpx3-2') Storage configuration exception
Show reports:
linstor error-reports show 5BF6D8BA-12EE8-000000
ERROR:
(Node: 'vpx3-1') Storage configuration exception
Show reports:
linstor error-reports show 5BF6D8AF-6B93F-000000
ERROR:
(Node: 'vpx3-2') Storage configuration exception
Show reports:
linstor error-reports show 5BF6D8BA-12EE8-000001
ERROR:
(Node: 'vpx3-1') Storage configuration exception
Show reports:
linstor error-reports show 5BF6D8AF-6B93F-000001
snapshot create failed: starting cleanup
error with cfs lock 'storage-drbdstorage': Could not remove
vm-100-state-test_snap_14_08: got lock timeout - aborting command
TASK ERROR: Could not create cluster wide snapshot for: vm-100-disk-2: exit
code 10
╭────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ SnapshotName ┊ NodeNames ┊ Volumes
┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ vm-100-disk-2 ┊ snap_vm-100-disk-2_test_snap_14_08 ┊ vpx3-1, vpx3-2 ┊ 0: 2.01
GiB ┊ Failed ┊
╰────────────────────────────────────────────────────────────────────────────────────────────╯
ERROR REPORT 5BF6D8BA-12EE8-000000
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.7.3
Build ID: 6e47dd2cb440e65727889f4c2f7423996153bdb3
Build time: 2018-11-22T10:24:37+00:00
Error time: 2018-11-23 14:10:13
Node: vpx3-2
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ensureStorageDriver', Source file
'DrbdDeviceHandler.java', Line #777
Error message: Storage configuration exception
Error context:
Storage configuration exception
Call backtrace:
Method Native Class:Line number
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:777
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Description:
Failed to verify volume group name
Cause:
External command timed out
Additional information:
External command: vgs -o vg_name --noheadings
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'checkVolumeGroupEntry', Source file
'LvmDriver.java', Line #479
Error message: Failed to verify volume group name
Call backtrace:
Method Native Class:Line number
checkVolumeGroupEntry N
com.linbit.linstor.storage.LvmDriver:479
checkConfiguration N
com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N
com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N
com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N
com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N
com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Category: Exception
Class name: ChildProcessTimeoutException
Class canonical name: com.linbit.ChildProcessTimeoutException
Generated at: Method 'waitFor', Source file
'ChildProcessHandler.java', Line #187
Call backtrace:
Method Native Class:Line number
waitFor N
com.linbit.extproc.ChildProcessHandler:187
waitFor N
com.linbit.extproc.ChildProcessHandler:124
syncProcess N com.linbit.extproc.ExtCmd:92
exec N com.linbit.extproc.ExtCmd:56
checkVolumeGroupEntry N
com.linbit.linstor.storage.LvmDriver:446
checkConfiguration N
com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N
com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N
com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N
com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N
com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
END OF ERROR REPORT.
ERROR REPORT 5BF6D8AF-6B93F-000000
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.7.3
Build ID: 6e47dd2cb440e65727889f4c2f7423996153bdb3
Build time: 2018-11-22T10:24:37+00:00
Error time: 2018-11-23 14:10:13
Node: vpx3-1
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ensureStorageDriver', Source file
'DrbdDeviceHandler.java', Line #777
Error message: Storage configuration exception
Error context:
Storage configuration exception
Call backtrace:
Method Native Class:Line number
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:777
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Description:
Failed to verify volume group name
Cause:
External command timed out
Additional information:
External command: vgs -o vg_name --noheadings
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'checkVolumeGroupEntry', Source file
'LvmDriver.java', Line #479
Error message: Failed to verify volume group name
Call backtrace:
Method Native Class:Line number
checkVolumeGroupEntry N
com.linbit.linstor.storage.LvmDriver:479
checkConfiguration N
com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N
com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N
com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N
com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N
com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Category: Exception
Class name: ChildProcessTimeoutException
Class canonical name: com.linbit.ChildProcessTimeoutException
Generated at: Method 'waitFor', Source file
'ChildProcessHandler.java', Line #187
Call backtrace:
Method Native Class:Line number
waitFor N
com.linbit.extproc.ChildProcessHandler:187
waitFor N
com.linbit.extproc.ChildProcessHandler:124
syncProcess N com.linbit.extproc.ExtCmd:92
exec N com.linbit.extproc.ExtCmd:56
checkVolumeGroupEntry N
com.linbit.linstor.storage.LvmDriver:446
checkConfiguration N
com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N
com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N
com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N
com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N
com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
END OF ERROR REPORT.
ERROR REPORT 5BF6D8BA-12EE8-000001
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.7.3
Build ID: 6e47dd2cb440e65727889f4c2f7423996153bdb3
Build time: 2018-11-22T10:24:37+00:00
Error time: 2018-11-23 14:11:19
Node: vpx3-2
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ensureStorageDriver', Source file
'DrbdDeviceHandler.java', Line #777
Error message: Storage configuration exception
Error context:
Storage configuration exception
Call backtrace:
Method Native Class:Line number
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:777
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Description:
Failed to verify volume group name
Cause:
External command timed out
Additional information:
External command: vgs -o vg_name --noheadings
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'checkVolumeGroupEntry', Source file
'LvmDriver.java', Line #479
Error message: Failed to verify volume group name
Call backtrace:
Method Native Class:Line number
checkVolumeGroupEntry N
com.linbit.linstor.storage.LvmDriver:479
checkConfiguration N
com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N
com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N
com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N
com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N
com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Category: Exception
Class name: ChildProcessTimeoutException
Class canonical name: com.linbit.ChildProcessTimeoutException
Generated at: Method 'waitFor', Source file
'ChildProcessHandler.java', Line #187
Call backtrace:
Method Native Class:Line number
waitFor N
com.linbit.extproc.ChildProcessHandler:187
waitFor N
com.linbit.extproc.ChildProcessHandler:124
syncProcess N com.linbit.extproc.ExtCmd:92
exec N com.linbit.extproc.ExtCmd:56
checkVolumeGroupEntry N
com.linbit.linstor.storage.LvmDriver:446
checkConfiguration N
com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N
com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N
com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N
com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N
com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
END OF ERROR REPORT.
ERROR REPORT 5BF6D8AF-6B93F-000001
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.7.3
Build ID: 6e47dd2cb440e65727889f4c2f7423996153bdb3
Build time: 2018-11-22T10:24:37+00:00
Error time: 2018-11-23 14:11:19
Node: vpx3-1
============================================================
Reported error:
===============
Category: Error
Class name: ImplementationError
Class canonical name: com.linbit.ImplementationError
Generated at: Method 'ensureStorageDriver', Source file
'DrbdDeviceHandler.java', Line #777
Error message: Storage configuration exception
Error context:
Storage configuration exception
Call backtrace:
Method Native Class:Line number
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:777
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Description:
Failed to verify volume group name
Cause:
External command timed out
Additional information:
External command: vgs -o vg_name --noheadings
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'checkVolumeGroupEntry', Source file
'LvmDriver.java', Line #479
Error message: Failed to verify volume group name
Call backtrace:
Method Native Class:Line number
checkVolumeGroupEntry N
com.linbit.linstor.storage.LvmDriver:479
checkConfiguration N
com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N
com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N
com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N
com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N
com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
Caused by:
==========
Category: Exception
Class name: ChildProcessTimeoutException
Class canonical name: com.linbit.ChildProcessTimeoutException
Generated at: Method 'waitFor', Source file
'ChildProcessHandler.java', Line #187
Call backtrace:
Method Native Class:Line number
waitFor N
com.linbit.extproc.ChildProcessHandler:187
waitFor N
com.linbit.extproc.ChildProcessHandler:124
syncProcess N com.linbit.extproc.ExtCmd:92
exec N com.linbit.extproc.ExtCmd:56
checkVolumeGroupEntry N
com.linbit.linstor.storage.LvmDriver:446
checkConfiguration N
com.linbit.linstor.storage.LvmDriver:332
checkConfiguration N
com.linbit.linstor.storage.LvmThinDriver:287
setConfiguration N
com.linbit.linstor.storage.AbsStorageDriver:540
reconfigureStorageDriver N
com.linbit.linstor.StorPoolData:217
ensureStorageDriverExc N
com.linbit.linstor.core.DrbdDeviceHandler:755
ensureStorageDriver N
com.linbit.linstor.core.DrbdDeviceHandler:773
evaluateStorageVolume N
com.linbit.linstor.core.DrbdDeviceHandler:800
createResourceStorage N
com.linbit.linstor.core.DrbdDeviceHandler:1165
createResource N
com.linbit.linstor.core.DrbdDeviceHandler:1121
dispatchResource N
com.linbit.linstor.core.DrbdDeviceHandler:364
run N
com.linbit.linstor.core.DeviceManagerImpl$DeviceHandlerInvocation:1225
run N
com.linbit.WorkerPool$WorkerThread:179
END OF ERROR REPORT.
Nov 23 14:08:35 vpx3-1 pvedaemon[17198]: <root@pam> snapshot VM 100:
test_snap_14_08
Nov 23 14:08:35 vpx3-1 pvedaemon[13971]: <root@pam> starting task
UPID:vpx3-1:0000432E:0071C1A2:5BF7FBD3:qmsnapshot:100:root@pam:
Nov 23 14:08:39 vpx3-1 Controller[1009]: 14:08:38.992 [MainWorkerPool-2] INFO
LINSTOR/Controller - New volume definition with number '0' of resource
definition 'vm-100-state-test_snap_14_08' created.
Nov 23 14:08:40 vpx3-1 Satellite[1021]: 14:08:40.502 [MainWorkerPool-2] INFO
LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' created for node
'vpx3-1'.
Nov 23 14:08:40 vpx3-1 Satellite[1021]: 14:08:40.613 [MainWorkerPool-2] INFO
LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' created for node
'vpx3-2'.
Nov 23 14:08:41 vpx3-1 Satellite[1021]: 14:08:41.174 [StltWorkerPool_0001] INFO
LINSTOR/Satellite - Aligning size from 1560948 KiB to 1564672 KiB to be a
multiple of extent size 4096 KiB
Nov 23 14:08:42 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Starting
worker thread (from drbdsetup [17326])
Nov 23 14:08:42 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Starting sender thread (from drbdsetup [17333])
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
disk( Diskless -> Attaching )
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
Maximum number of peer devices = 7
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Method to
ensure write ordering: flush
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
drbd_bm_resize called with capacity == 3128600
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
resync bitmap: bits=391075 words=42777 pages=84
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
size = 1528 MB (1564300 KB)
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
recounting of set bits took additional 0ms
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
disk( Attaching -> UpToDate )
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
attached to current UUID: 3B5634C1B39D3086
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
size = 1528 MB (1564300 KB)
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn(
StandAlone -> Unconnected )
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Starting receiver thread (from drbd_w_vm-100-s [17327])
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn(
Unconnected -> Connecting )
Nov 23 14:08:43 vpx3-1 Satellite[1021]: 14:08:43.140 [MainWorkerPool-2] INFO
LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' updated for node
'vpx3-1'.
Nov 23 14:08:43 vpx3-1 Satellite[1021]: 14:08:43.140 [MainWorkerPool-2] INFO
LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' updated for node
'vpx3-2'.
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Handshake to peer 0 successful: Agreed network protocol version 114
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME
WRITE_ZEROES.
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: Peer
authenticated using 20 bytes HMAC
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Starting ack_recv thread (from drbd_r_vm-100-s [17349])
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Preparing remote state change 2055326401
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Committing remote state change 2055326401 (primary_nodes=0)
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn(
Connecting -> Connected ) peer( Unknown -> Secondary )
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004
vpx3-2: drbd_sync_handshake:
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004
vpx3-2: self
3B5634C1B39D3086:0000000000000000:0000000000000000:0000000000000000 bits:0
flags:20
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004
vpx3-2: peer
3B5634C1B39D3086:0000000000000000:0000000000000000:0000000000000000 bits:0
flags:20
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004
vpx3-2: uuid_compare()=0 by rule 40
Nov 23 14:08:43 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004
vpx3-2: pdsk( DUnknown -> UpToDate ) repl( Off -> Established )
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Preparing
cluster-wide state change 1246189751 (1->-1 3/1)
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: State change
1246189751: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Committing
cluster-wide state change 1246189751 (4ms)
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: role(
Secondary -> Primary )
Nov 23 14:08:44 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
new current UUID: B89F98A90ABEF3CF weak: FFFFFFFFFFFFFFFC
Nov 23 14:09:01 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:09:03 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:09:05 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: role( Primary
-> Secondary )
Nov 23 14:09:06 vpx3-1 pvestatd[1477]: status update time (14.130 seconds)
Nov 23 14:09:08 vpx3-1 Satellite[1021]: 14:09:08.181 [MainWorkerPool-1] INFO
LINSTOR/Satellite - Snapshot 'snap_vm-100-disk-2_test_snap_14_08' of resource
'vm-100-disk-2' registered.
Nov 23 14:09:08 vpx3-1 kernel: drbd vm-100-disk-2: susp-io( no -> user)
Nov 23 14:09:08 vpx3-1 Satellite[1021]: 14:09:08.623 [MainWorkerPool-1] INFO
LINSTOR/Satellite - Snapshot 'snap_vm-100-disk-2_test_snap_14_08' of resource
'vm-100-disk-2' registered.
Nov 23 14:10:00 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:10:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:10:14 vpx3-1 Satellite[1021]: 14:10:14.152 [StltWorkerPool_0002]
ERROR LINSTOR/Satellite - Storage configuration exception [Report number
5BF6D8AF-6B93F-000000]
Nov 23 14:10:14 vpx3-1 Satellite[1021]: 14:10:14.371 [MainWorkerPool-1] INFO
LINSTOR/Satellite - Snapshot 'snap_vm-100-disk-2_test_snap_14_08' ended.
Nov 23 14:11:00 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:11:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:11:19 vpx3-1 Satellite[1021]: 14:11:19.416 [StltWorkerPool_0001]
ERROR LINSTOR/Satellite - Storage configuration exception [Report number
5BF6D8AF-6B93F-000001]
Nov 23 14:11:20 vpx3-1 pvedaemon[17198]: snapshot create failed: starting
cleanup
Nov 23 14:11:21 vpx3-1 Satellite[1021]: 14:11:21.156 [MainWorkerPool-1] INFO
LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' updated for node
'vpx3-1'.
Nov 23 14:11:21 vpx3-1 Satellite[1021]: 14:11:21.157 [MainWorkerPool-1] INFO
LINSTOR/Satellite - Resource 'vm-100-state-test_snap_14_08' updated for node
'vpx3-2'.
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Preparing
cluster-wide state change 1830748394 (1->0 496/16)
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: State change
1830748394: primary_nodes=0, weak_nodes=0
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Cluster is now split
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Committing
cluster-wide state change 1830748394 (0ms)
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn(
Connected -> Disconnecting ) peer( Secondary -> Unknown )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004
vpx3-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
ack_receiver terminated
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Terminating ack_recv thread
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Connection closed
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2: conn(
Disconnecting -> StandAlone )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Terminating receiver thread
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08 vpx3-2:
Terminating sender thread
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
disk( UpToDate -> Detaching )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
disk( Detaching -> Diskless )
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08/0 drbd1004:
drbd_bm_resize called with capacity == 0
Nov 23 14:11:21 vpx3-1 kernel: drbd vm-100-state-test_snap_14_08: Terminating
worker thread
Nov 23 14:12:00 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:12:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:12:20 vpx3-1 pvedaemon[17198]: error with cfs lock
'storage-drbdstorage': Could not remove vm-100-state-test_snap_14_08: got lock
timeout - aborting command
Nov 23 14:12:20 vpx3-1 pvedaemon[17198]: Could not create cluster wide snapshot
for: vm-100-disk-2: exit code 10
Nov 23 14:12:20 vpx3-1 pvedaemon[13971]: <root@pam> end task
UPID:vpx3-1:0000432E:0071C1A2:5BF7FBD3:qmsnapshot:100:root@pam: Could not
create cluster wide snapshot for: vm-100-disk-2: exit code 10
Nov 23 14:12:26 vpx3-1 Satellite[1021]: 14:12:26.295 [StltWorkerPool_0000]
ERROR LINSTOR/Satellite - Failed to verify volume group name [Report number
5BF6D8AF-6B93F-000002]
Nov 23 14:12:40 vpx3-1 kernel: INFO: task kvm:17785 blocked for more than 120
seconds.
Nov 23 14:12:40 vpx3-1 kernel: Tainted: P O 4.15.18-9-pve #1
Nov 23 14:12:40 vpx3-1 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 14:12:40 vpx3-1 kernel: kvm D 0 17785 1 0x00000000
Nov 23 14:12:40 vpx3-1 kernel: Call Trace:
Nov 23 14:12:40 vpx3-1 kernel: __schedule+0x3e0/0x870
Nov 23 14:12:40 vpx3-1 kernel: schedule+0x36/0x80
Nov 23 14:12:40 vpx3-1 kernel: io_schedule+0x16/0x40
Nov 23 14:12:40 vpx3-1 kernel: __lock_page+0xff/0x140
Nov 23 14:12:40 vpx3-1 kernel: ? page_cache_tree_insert+0xe0/0xe0
Nov 23 14:12:40 vpx3-1 kernel: pagecache_get_page+0x195/0x2b0
Nov 23 14:12:40 vpx3-1 kernel: ? I_BDEV+0x20/0x20
Nov 23 14:12:40 vpx3-1 kernel: grab_cache_page_write_begin+0x23/0x40
Nov 23 14:12:40 vpx3-1 kernel: block_write_begin+0x2d/0xe0
Nov 23 14:12:40 vpx3-1 kernel: blkdev_write_begin+0x23/0x30
Nov 23 14:12:40 vpx3-1 kernel: generic_perform_write+0xb9/0x1b0
Nov 23 14:12:40 vpx3-1 kernel: __generic_file_write_iter+0x185/0x1c0
Nov 23 14:12:40 vpx3-1 kernel: blkdev_write_iter+0xa8/0x130
Nov 23 14:12:40 vpx3-1 kernel: new_sync_write+0xe7/0x140
Nov 23 14:12:40 vpx3-1 kernel: ? __blkdev_get+0x4d0/0x4d0
Nov 23 14:12:40 vpx3-1 kernel: ? new_sync_write+0xe7/0x140
Nov 23 14:12:40 vpx3-1 kernel: __vfs_write+0x29/0x40
Nov 23 14:12:40 vpx3-1 kernel: vfs_write+0xb5/0x1a0
Nov 23 14:12:40 vpx3-1 kernel: SyS_pwrite64+0x95/0xb0
Nov 23 14:12:40 vpx3-1 kernel: do_syscall_64+0x73/0x130
Nov 23 14:12:40 vpx3-1 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Nov 23 14:12:40 vpx3-1 kernel: RIP: 0033:0x7f309f856963
Nov 23 14:12:40 vpx3-1 kernel: RSP: 002b:00007f30653c55d0 EFLAGS: 00000293
ORIG_RAX: 0000000000000012
Nov 23 14:12:40 vpx3-1 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
00007f309f856963
Nov 23 14:12:40 vpx3-1 kernel: RDX: 0000000000001000 RSI: 00007f3087be2000 RDI:
0000000000000011
Nov 23 14:12:40 vpx3-1 kernel: RBP: 00007f306c694080 R08: 0000000000000000 R09:
00007a395bc07916
Nov 23 14:12:40 vpx3-1 kernel: R10: 0000000009f0d000 R11: 0000000000000293 R12:
00007f3087be2000
Nov 23 14:12:40 vpx3-1 kernel: R13: 00007f30918bcfd8 R14: 00007f306dfc11d0 R15:
00007f3061c0e400
Nov 23 14:13:00 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:13:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:14:01 vpx3-1 systemd[1]: Starting Proxmox VE replication runner...
Nov 23 14:14:02 vpx3-1 systemd[1]: Started Proxmox VE replication runner.
Nov 23 14:14:34 vpx3-1 pveproxy[5479]: worker exit
Nov 23 14:14:34 vpx3-1 pveproxy[1559]: worker 5479 finished
Nov 23 14:14:34 vpx3-1 pveproxy[1559]: starting 1 worker(s)
Nov 23 14:14:34 vpx3-1 pveproxy[1559]: worker 18255 started
Nov 23 14:14:40 vpx3-1 kernel: INFO: task kvm:17593 blocked for more than 120
seconds.
Nov 23 14:14:40 vpx3-1 kernel: Tainted: P O 4.15.18-9-pve #1
Nov 23 14:14:40 vpx3-1 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 14:14:40 vpx3-1 kernel: kvm D 0 17593 1 0x00000000
Nov 23 14:14:40 vpx3-1 kernel: Call Trace:
Nov 23 14:14:40 vpx3-1 kernel: __schedule+0x3e0/0x870
Nov 23 14:14:40 vpx3-1 kernel: schedule+0x36/0x80
Nov 23 14:14:40 vpx3-1 kernel: __drbd_make_request+0x27c/0x5b0 [drbd]
Nov 23 14:14:40 vpx3-1 kernel: ? wait_woken+0x80/0x80
Nov 23 14:14:40 vpx3-1 kernel: drbd_make_request+0x4f/0x80 [drbd]
Nov 23 14:14:40 vpx3-1 kernel: generic_make_request+0x123/0x2f0
Nov 23 14:14:40 vpx3-1 kernel: submit_bio+0x73/0x140
Nov 23 14:14:40 vpx3-1 kernel: ? submit_bio+0x73/0x140
Nov 23 14:14:40 vpx3-1 kernel: ? guard_bio_eod+0x2c/0xf0
Nov 23 14:14:40 vpx3-1 kernel: submit_bh_wbc+0x180/0x1b0
Nov 23 14:14:40 vpx3-1 kernel: __block_write_full_page+0x197/0x3f0
Nov 23 14:14:40 vpx3-1 kernel: block_write_full_page+0x104/0x120
Nov 23 14:14:40 vpx3-1 kernel: blkdev_writepage+0x18/0x20
Nov 23 14:14:40 vpx3-1 kernel: __writepage+0x1d/0x50
Nov 23 14:14:40 vpx3-1 kernel: write_cache_pages+0x22e/0x4b0
Nov 23 14:14:40 vpx3-1 kernel: ? __wb_calc_thresh+0x140/0x140
Nov 23 14:14:40 vpx3-1 kernel: ? futex_wait+0x1f5/0x260
Nov 23 14:14:40 vpx3-1 kernel: generic_writepages+0x61/0xa0
Nov 23 14:14:40 vpx3-1 kernel: blkdev_writepages+0x2f/0x40
Nov 23 14:14:40 vpx3-1 kernel: ? blkdev_writepages+0x2f/0x40
Nov 23 14:14:40 vpx3-1 kernel: do_writepages+0x1f/0x70
Nov 23 14:14:40 vpx3-1 kernel: __filemap_fdatawrite_range+0xc6/0x100
Nov 23 14:14:40 vpx3-1 kernel: file_write_and_wait_range+0x5a/0xb0
Nov 23 14:14:40 vpx3-1 kernel: blkdev_fsync+0x1b/0x50
Nov 23 14:14:40 vpx3-1 kernel: vfs_fsync_range+0x51/0xb0
Nov 23 14:14:40 vpx3-1 kernel: do_fsync+0x3d/0x70
Nov 23 14:14:40 vpx3-1 kernel: ? exit_to_usermode_loop+0xa5/0xd0
Nov 23 14:14:40 vpx3-1 kernel: SyS_fdatasync+0x13/0x20
Nov 23 14:14:40 vpx3-1 kernel: do_syscall_64+0x73/0x130
Nov 23 14:14:40 vpx3-1 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Nov 23 14:14:40 vpx3-1 kernel: RIP: 0033:0x7f309f58860d
Nov 23 14:14:40 vpx3-1 kernel: RSP: 002b:00007f305d3f55f0 EFLAGS: 00000293
ORIG_RAX: 000000000000004b
Nov 23 14:14:40 vpx3-1 kernel: RAX: ffffffffffffffda RBX: 00000000fffffffb RCX:
00007f309f58860d
Nov 23 14:14:40 vpx3-1 kernel: RDX: 00007f30918bcf70 RSI: 000055cb42a026e0 RDI:
0000000000000011
Nov 23 14:14:40 vpx3-1 kernel: RBP: 00007f306c6941c0 R08: 0000000000000000 R09:
00000000ffffffff
Nov 23 14:14:40 vpx3-1 kernel: R10: 00007f305d3f5620 R11: 0000000000000293 R12:
00007f30919c4bc0
Nov 23 14:14:40 vpx3-1 kernel: R13: 00007f30918bcfd8 R14: 00007f306dfc10f0 R15:
00007f308f45e400
Nov 23 14:14:40 vpx3-1 kernel: INFO: task kvm:17785 blocked for more than 120
seconds.
╭───────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ first ┊ vpx3-1 ┊ 7002 ┊ Unused ┊ UpToDate ┊
┊ first ┊ vpx3-2 ┊ 7002 ┊ Unused ┊ UpToDate ┊
┊ vm-100-disk-1 ┊ vpx3-1 ┊ 7001 ┊ Unused ┊ UpToDate ┊
┊ vm-100-disk-1 ┊ vpx3-2 ┊ 7001 ┊ Unused ┊ UpToDate ┊
┊ vm-100-disk-2 ┊ vpx3-1 ┊ 7003 ┊ Unused ┊ UpToDate ┊
┊ vm-100-disk-2 ┊ vpx3-2 ┊ 7003 ┊ Unused ┊ UpToDate ┊
╰───────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊
TotalCapacity ┊ SupportsSnapshots ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ drbdpool ┊ vpx3-1 ┊ LvmThinDriver ┊ vg_drbd/drbdthinpool ┊ 59.27 GiB ┊
63 GiB ┊ true ┊
┊ drbdpool ┊ vpx3-2 ┊ LvmThinDriver ┊ vg_drbd/drbdthinpool ┊ 59.27 GiB ┊
63 GiB ┊ true ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────────────────────────────╮
┊ ResourceName ┊ VolumeNr ┊ VolumeMinor ┊ Size ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ first ┊ 0 ┊ 1002 ┊ 10 MiB ┊ ok ┊
┊ vm-100-disk-1 ┊ 0 ┊ 1001 ┊ 2.01 GiB ┊ ok ┊
┊ vm-100-disk-2 ┊ 0 ┊ 1003 ┊ 2.01 GiB ┊ ok ┊
╰───────────────────────────────────────────────────────────╯
_______________________________________________
drbd-user mailing list
[email protected]
http://lists.linbit.com/mailman/listinfo/drbd-user
