Hi,
we are experiencing issues while we try to migrate an existing DRBD cluster
from drbdmanage to linstor (Debian stable with proxmox for VMs) following the
instructions at
https://docs.linbit.com/docs/users-guide-9.0/#s-linstor-migrate-from-dm
The issue is that the resource files created when I execute the migration
script (dmmmigrate.sh in my case) created by
monetbase2:~# drbdmanage export-ctrlvol > ctrlvol.json
monetbase2:~# linstor dm-migrate ctrlvol.json dmmmigrate.sh
contain wrong port and wrong VolumeMinor numbers.
However, the numbers are correctly assigned in the lines for the resource and
volume creation inside dmmigrate.sh. This is somewhat
puzzeling.
In order to debug it, I decided to manually execute the contents of
dmmmigrate.sh line by line. (After reverting to drbdmanage and a clean
linstor installation.) I took the volume vm-1802-disk-2
which according to [A] (see below) uses port 7014 and VolumeMinor 125.
First I made sure that 'drbdmanage shutdown -qc' worked and DBUS wouldn't
restart it (on both DRBD nodes). Then I disabled /etc/drbd.d/drbdctrl.res
and /etc/drbd.d/drbdmanage-resources.res on both nodes, started
linstor-satellite on them and the linstor-controller on node monetbase2.
Then, following [B] from dmmmigrate.sh below, on the controller node,
monetbase2, I executed:
monetbase2:~# LS_CONTROLLERS="localhost"
monetbase2:~# export LS_CONTROLLERS
monetbase2:~# linstor node create --node-type Combined munchbase2 10.0.7.13
monetbase2:~# linstor node create --node-type Combined monetbase2 10.0.7.12
monetbase2:~# linstor n l
monetbase2:~# linstor storage-pool-definition create drbdpool
monetbase2:~# linstor storage-pool create lvmthin munchbase2 drbdpool
drbdpool/drbdthinpool
monetbase2:~# linstor storage-pool create lvmthin monetbase2 drbdpool
drbdpool/drbdthinpool
monetbase2:~# linstor sp l
The created storage pool looked fine.
Going on:
monetbase2:~# linstor resource-definition create --port 7014 vm-1802-disk-2
monetbase2:~# linstor resource-definition drbd-options --allow-two-primaries
yes vm-1802-disk-2
monetbase2:~# linstor volume-definition create --vlmnr 0 --minor 125
vm-1802-disk-2 44040192K
monetbase2:~# linstor volume-definition set-property vm-1802-disk-2 0
OverrideVlmId vm-1802-disk-2_00
monetbase2:~# linstor volume-definition set-property vm-1802-disk-2 0
AllowLargerVolumeSize true
monetbase2:~# linstor resource create --node-id 1 --storage-pool drbdpool
munchbase2 vm-1802-disk-2
monetbase2:~# linstor resource create --node-id 0 --storage-pool drbdpool
monetbase2 vm-1802-disk-2
the last command produced an error message:
monetbase2:~# linstor resource create --node-id 0 --storage-pool drbdpool
monetbase2 vm-1802-disk-2
SUCCESS:
Description:
New resource 'vm-1802-disk-2' on node 'monetbase2' registered.
Details:
Resource 'vm-1802-disk-2' on node 'monetbase2' UUID is:
30b98a7f-cb31-4635-89cd-6860b13fa5cc
SUCCESS:
Description:
Volume with number '0' on resource 'vm-1802-disk-2' on node 'monetbase2'
successfully registered
Details:
Volume UUID is: 86fc67d7-18f1-41b1-9718-4b15ddf894f9
ERROR:
(Node: 'monetbase2') Failed to adjust DRBD resource vm-1802-disk-2
Show reports:
linstor error-reports show 5CA090AF-97F26-000000
ERROR:
(Node: 'munchbase2') Failed to adjust DRBD resource vm-1802-disk-2
Show reports:
linstor error-reports show 5CA09099-17F52-000000
Now, looking at e.g. the first error report (see [C] and [D] below for the
full error reports), I see:
The full command line executed was:
drbdadm -vvv adjust vm-1802-disk-2
The external command sent the following output data:
drbdsetup detach --diskless 125
drbdsetup del-minor 125
drbdsetup new-minor vm-1802-disk-2 1000 0
drbdsetup disconnect vm-1802-disk-2 1
drbdsetup del-path vm-1802-disk-2 1 ipv4:10.0.7.12:7014
ipv4:10.0.7.13:7014
drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000
ipv4:10.0.7.13:7000
that a wrong port and a wrong VolumeMinor number are forced.
Looking into /var/lib/linstor.d/vm-1802-disk-2.res (see [E] below),
I see that the wrong asignments are contained in this resource file.
I also see the wrong assignments using linstor now:
monetbase2:/var/lib/linstor.d# linstor vd l -p
+----------------------------------------------------------+
| ResourceName | VolumeNr | VolumeMinor | Size | State |
|----------------------------------------------------------|
| vm-1802-disk-2 | 0 | 1000 | 42 GiB | ok |
+----------------------------------------------------------+
monetbase2:/var/lib/linstor.d# linstor r l -p
+--------------------------------------------------------+
| ResourceName | Node | Port | Usage | State |
|--------------------------------------------------------|
| vm-1802-disk-2 | monetbase2 | 7000 | Unused | Diskless |
| vm-1802-disk-2 | munchbase2 | 7000 | Unused | Diskless |
+--------------------------------------------------------+
Unfortunately, this spoils my attempt to migrate from drbdmanage to
linstor on an active system since I could not figure out,
how to modify the port number and the VolumeMinor using the linstor client.
I'll revert to the drbdmanage setup for now.
However, I would be grateful if someone knowledgable could have a look
and see whether this is a bug or a mistake on my side. A workaround
(like how force these two numbers with a linstor client command) would be
appreciated.
Sorry for the long post. Hopefully I have provided enough details.
Thank you for having a look,
w.w.
[A] -------------------------
munchbase2:~# drbdmanage list-volumes -p -g Port
+----------------------------------------------------------------------------+
| Name | Vol ID | Size | Minor | Port | | State
|
|----------------------------------------------------------------------------|
| vm-1801-disk-1 | 0 | 32 GiB | 103 | 7000 | | ok |
| vm-103-disk-2 | 0 | 1 TiB | 104 | 7001 | | ok |
| vm-103-disk-3 | 0 | 2 TiB | 105 | 7002 | | ok |
| vm-103-disk-4 | 0 | 1 TiB | 106 | 7003 | | ok |
| vm-103-disk-5 | 0 | 500 GiB | 107 | 7004 | | ok |
| vm-102-disk-2 | 0 | 1 TiB | 108 | 7005 | | ok |
| vm-102-disk-3 | 0 | 2 TiB | 109 | 7006 | | ok |
| vm-102-disk-4 | 0 | 1 TiB | 111 | 7007 | | ok |
| vm-102-disk-5 | 0 | 500 GiB | 112 | 7008 | | ok |
| vm-102-disk-6 | 0 | 100 GiB | 113 | 7009 | | ok |
| vm-1802-disk-1 | 0 | 32 GiB | 114 | 7010 | | ok |
| vm-105-disk-2 | 0 | 200 GiB | 120 | 7011 | | ok |
| vm-104-disk-1 | 0 | 18 GiB | 119 | 7012 | | ok |
| vm-105-disk-1 | 0 | 32 GiB | 121 | 7013 | | ok |
| vm-1802-disk-2 | 0 | 42 GiB | 125 | 7014 | | ok |
| vm-1801-disk-12 | 0 | 32 GiB | 123 | 7015 | | ok |
| vm-103-disk-1 | 0 | 50 GiB | 126 | 7016 | | ok |
| vm-102-disk-1 | 0 | 50 GiB | 127 | 7017 | | ok |
| vm-106-disk-1 | 0 | 32 GiB | 128 | 7018 | | ok |
+----------------------------------------------------------------------------+
[B] -------------------------
dmmmigrate.sh:
...
# If the controller is not executed on the local host, set this variable:
LS_CONTROLLERS="localhost"
export LS_CONTROLLERS
...
### Nodes ###
linstor node create --node-type Combined munchbase2 10.0.7.13
linstor node create --node-type Combined monetbase2 10.0.7.12
### Storage ###
linstor storage-pool-definition create drbdpool
linstor storage-pool create lvmthin munchbase2 drbdpool drbdpool/drbdthinpool
linstor storage-pool create lvmthin monetbase2 drbdpool drbdpool/drbdthinpool
...
### Resource: vm-1802-disk-2 ###
linstor resource-definition create --port 7014 vm-1802-disk-2
linstor resource-definition drbd-options --allow-two-primaries yes
vm-1802-disk-2
linstor volume-definition create --vlmnr 0 --minor 125 vm-1802-disk-2
44040192K
linstor volume-definition set-property vm-1802-disk-2 0 OverrideVlmId
vm-1802-disk-2_00
linstor volume-definition set-property vm-1802-disk-2 0 AllowLargerVolumeSize
true
linstor resource create --node-id 1 --storage-pool drbdpool munchbase2
vm-1802-disk-2
linstor resource create --node-id 0 --storage-pool drbdpool monetbase2
vm-1802-disk-2
[C] ------------------
monetbase2:/etc/linstor# linstor error-reports show 5CA090AF-97F26-000000
ERROR REPORT 5CA090AF-97F26-000000
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.9.4
Build ID: ff7acd2330eb0cbbdcfae5c3a1ec7a8eba5df3c2
Build time: 2019-03-29T09:40:59+00:00
Error time: 2019-03-31 12:49:17
Node: monetbase2
============================================================
Reported error:
===============
Description:
Failed to adjust DRBD resource vm-1802-disk-2
Category: LinStorException
Class name: ResourceException
Class canonical name:
com.linbit.linstor.storage.layer.exceptions.ResourceException
Generated at: Method 'adjustDrbd', Source file
'DrbdLayer.java', Line #432
Error message: Failed to adjust DRBD resource
vm-1802-disk-2
Error context:
An error occurred while processing resource 'Node: 'monetbase2', Rsc:
'vm-1802-disk-2''
Call backtrace:
Method Native Class:Line number
adjustDrbd N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:432
process N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
process N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
processResourcesAndTheirSnapshots N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
phaseDispatchDeviceHandlers N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
devMgrLoop N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
run N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
run N java.lang.Thread:748
Caused by:
==========
Description:
Execution of the external command 'drbdadm' failed.
Cause:
The external command exited with error code 1.
Correction:
- Check whether the external program is operating properly.
- Check whether the command line is correct.
Contact a system administrator or a developer if the command line is no
longer valid
for the installed version of the external program.
Additional information:
The full command line executed was:
drbdadm -vvv adjust vm-1802-disk-2
The external command sent the following output data:
drbdsetup detach --diskless 125
drbdsetup del-minor 125
drbdsetup new-minor vm-1802-disk-2 1000 0
drbdsetup disconnect vm-1802-disk-2 1
drbdsetup del-path vm-1802-disk-2 1 ipv4:10.0.7.12:7014
ipv4:10.0.7.13:7014
drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000
ipv4:10.0.7.13:7000
The external command sent the follwing error information:
[ne] vol:0 minor differs: r=125 c=1000
Value of 'shared-secret' differs: r=bXj5Gn1qO2YXgQoX6wLE
c=aPDz2JoXsWxbscd2j70f
vm-1802-disk-2: Failure: (173) Combination of local address(port) and
remote address(port) already in use
Command 'drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000
ipv4:10.0.7.13:7000' terminated with exit code 10
Category: LinStorException
Class name: ExtCmdFailedException
Class canonical name: com.linbit.extproc.ExtCmdFailedException
Generated at: Method 'execute', Source file
'DrbdAdm.java', Line #455
Error message: The external command 'drbdadm' exited
with error code 1
Call backtrace:
Method Native Class:Line number
execute N
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:455
adjust N
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:86
adjustDrbd N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:414
process N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
process N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
processResourcesAndTheirSnapshots N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
phaseDispatchDeviceHandlers N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
devMgrLoop N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
run N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
run N java.lang.Thread:748
END OF ERROR REPORT.
[D] ------------------
monetbase2:/etc/linstor# linstor error-reports show 5CA09099-17F52-000000
ERROR REPORT 5CA09099-17F52-000000
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 0.9.4
Build ID: ff7acd2330eb0cbbdcfae5c3a1ec7a8eba5df3c2
Build time: 2019-03-29T09:40:59+00:00
Error time: 2019-03-31 12:49:17
Node: munchbase2
============================================================
Reported error:
===============
Description:
Failed to adjust DRBD resource vm-1802-disk-2
Category: LinStorException
Class name: ResourceException
Class canonical name:
com.linbit.linstor.storage.layer.exceptions.ResourceException
Generated at: Method 'adjustDrbd', Source file
'DrbdLayer.java', Line #432
Error message: Failed to adjust DRBD resource
vm-1802-disk-2
Error context:
An error occurred while processing resource 'Node: 'munchbase2', Rsc:
'vm-1802-disk-2''
Call backtrace:
Method Native Class:Line number
adjustDrbd N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:432
process N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
process N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
processResourcesAndTheirSnapshots N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
phaseDispatchDeviceHandlers N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
devMgrLoop N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
run N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
run N java.lang.Thread:748
Caused by:
==========
Description:
Execution of the external command 'drbdadm' failed.
Cause:
The external command exited with error code 1.
Correction:
- Check whether the external program is operating properly.
- Check whether the command line is correct.
Contact a system administrator or a developer if the command line is no
longer valid
for the installed version of the external program.
Additional information:
The full command line executed was:
drbdadm -vvv adjust vm-1802-disk-2
The external command sent the following output data:
drbdsetup new-peer vm-1802-disk-2 0 --_name=monetbase2
--allow-two-primaries=yes --shared-secret=aPDz2JoXsWxbscd2j70f
--cram-hmac-alg=sha1
drbdsetup new-path vm-1802-disk-2 0 ipv4:10.0.7.13:7000
ipv4:10.0.7.12:7000
The external command sent the follwing error information:
Value of 'rs-discard-granularity' differs: r=524288 c=65536
vm-1802-disk-2: Failure: (173) Combination of local address(port) and
remote address(port) already in use
Command 'drbdsetup new-path vm-1802-disk-2 0 ipv4:10.0.7.13:7000
ipv4:10.0.7.12:7000' terminated with exit code 10
Category: LinStorException
Class name: ExtCmdFailedException
Class canonical name: com.linbit.extproc.ExtCmdFailedException
Generated at: Method 'execute', Source file
'DrbdAdm.java', Line #455
Error message: The external command 'drbdadm' exited
with error code 1
Call backtrace:
Method Native Class:Line number
execute N
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:455
adjust N
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:86
adjustDrbd N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:414
process N
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
process N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
processResourcesAndTheirSnapshots N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
dispatchResources N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
phaseDispatchDeviceHandlers N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
devMgrLoop N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
run N
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
run N java.lang.Thread:748
END OF ERROR REPORT.
[E] --------------------------
/var/lib/linstor.d/vm-1802-disk-2.res:
# This file was generated by linstor(0.9.4), do not edit manually.
resource "vm-1802-disk-2"
{
template-file "linstor_common.conf";
net
{
cram-hmac-alg sha1;
shared-secret "TKoHhL4KThL165NGg/ja";
allow-two-primaries yes;
}
on monetbase2
{
volume 0
{
disk /dev/drbdpool/vm-1802-disk-2_00;
disk
{
discard-zeroes-if-aligned yes;
rs-discard-granularity 65536;
}
meta-disk internal;
device minor 1000;
}
node-id 0;
}
on munchbase2
{
volume 0
{
disk /dev/drbd/this/is/not/used;
disk
{
discard-zeroes-if-aligned yes;
rs-discard-granularity 65536;
}
meta-disk internal;
device minor 1000;
}
node-id 1;
}
connection
{
host monetbase2 address ipv4 10.0.7.12:7000;
host munchbase2 address ipv4 10.0.7.13:7000;
}
}
==========================================================2019-03-31==13:56==
Dr. Wolfgang Walkowiak Phone: +49-271-740-3889
Fakultaet IV / Physik Fax : +49-271-740-3886
Emmy Noether Campus
Universitaet Siegen --> [email protected]
Walter-Flex-Str. 3 [email protected]
57068 Siegen
Germany
=============================================================================
