GitHub user top-secrett added a comment to the discussion: Multiple secondary
storage in one zone; problem with volume snapshots
Example when first volume snapshot was copied to the second secstore and I try
to create the next volume snapshot
~~~
2024-09-26 16:58:32,135 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Executing
AsyncJobVO: {id:72454, userId: 298, accountId: 225, instanceType: Snapshot,
instanceId: 867, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo:
{"asyncBackup":"false","quiescevm":"false","response":"json","ctxUserId":"298","volumeId":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","httpmethod":"GET","ctxStartEventId":"1811467","id":"867","ctxDetails":"{\"interface
com.cloud.storage.Volume\":\"da420806-0ff3-43b8-9ee0-4aa2adfd9b99\",\"interface
com.cloud.storage.Snapshot\":\"c62b0aa0-5a1c-475c-b7f6-6c9fab17e325\"}","ctxAccountId":"225","uuid":"c62b0aa0-5a1c-475c-b7f6-6c9fab17e325","cmdEventType":"SNAPSHOT.CREATE"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 108597816265214, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null, removed: null}
2024-09-26 16:58:32,147 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96)
Access to Account
[{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}]
granted to Account
[{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}]
by DomainChecker
2024-09-26 16:58:32,167 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4830812:ctx-4bacce10) (logid:11a55db0) ===START=== 172.16.11.61
-- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:32,183 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96)
Access to org.apache.cloudstack.storage.volume.VolumeObject@f1e18b7 granted to
Account
[{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}]
by DomainChecker
2024-09-26 16:58:32,190 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4830812:ctx-4bacce10 ctx-aa1d9072) (logid:11a55db0) ===END===
172.16.11.61 -- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:32,192 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96)
Access to VM instance
{"id":5468,"instanceName":"i-225-5468-VM","type":"User","uuid":"d1e26871-d01f-44cc-ac61-f19f46ed741a"}
granted to Account
[{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}]
by DomainChecker
2024-09-26 16:58:32,206 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96) Sync
job-72455 execution on object VmWorkJobQueue.5468
2024-09-26 16:58:32,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96)
Executing AsyncJobVO: {id:72455, userId: 298, accountId: 225, instanceType:
null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo:
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAdaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAkwAB3pvbmVJZHN0ABBMamF2YS91dGlsL0xpc3Q7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAOEAAAAAAAABKgAAAAAAABVcdAAUVm9sdW1lQXBpU2VydmljZUltcGwAAHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAABzcQB-AAgAAAAAAAADY3NxAH4ACAAAAAAAABgMcA,
cmdVersion: 0, status: IN_PROGRESS, processStatus:
0, resultCode: 0, result: null, initMsid: 108597816265214, completeMsid: null,
lastUpdated: null, lastPolled: null, created: Thu Sep 26 16:58:32 MSK 2024,
removed: null}
2024-09-26 16:58:32,575 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Run
VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 5468, job origin:
72454
2024-09-26 16:58:32,576 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Execute VM work job:
com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":6156,"policyId":0,"snapshotId":867,"quiesceVm":false,"asyncBackup":false,"userId":298,"accountId":225,"vmId":5468,"handlerName":"VolumeApiServiceImpl"}
2024-09-26 16:58:32,588 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) StorpoolSnapshotStrategy.canHandle:
snapshot=backup-test-karasev-3_ROOT-5468_20240926135832,
uuid=c62b0aa0-5a1c-475c-b7f6-6c9fab17e325, op=TAKE
2024-09-26 16:58:32,622 DEBUG
[o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Taking snapshot of
org.apache.cloudstack.storage.snapshot.SnapshotObject@18f701c2
2024-09-26 16:58:32,653 DEBUG
[o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Taking snapshot of
org.apache.cloudstack.storage.snapshot.SnapshotObject@18f701c2 and encryption
required is false
2024-09-26 16:58:32,653 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CreateObjectCommand
2024-09-26 16:58:32,655 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) We are returning the default host to execute commands because
the command is not of Copy type.
2024-09-26 16:58:32,657 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Seq 77353-4517673376206130891: Sending { Cmd , MgmtId:
108597816265214, via: 77353(xcp-sr246-u1314.aisrp.local), Ver: v1, Flags:
100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00
GB)
21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6156
","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":"0","accountId":"225","followRedirects":"false"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2024-09-26 16:58:32,659 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Seq 77353-4517673376206130891: Executing: { Cmd , MgmtId:
108597816265214, via: 77353(xcp-sr246-u1314.aisrp.local), Ver: v1, Flags:
100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00
GB)
21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6
156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":"0","accountId":"225","followRedirects":"false"}},"wait":"0","bypassHostMaintenance":"false"}}]
}
2024-09-26 16:58:32,659 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase]
(DirectAgent-128:ctx-052ce805) (logid:f365dd96) Executing command
CreateObjectCommand:
[{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":false}},"name":"ROOT-5468","size":21474836480,"path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":6156,"vmName":"i-225-5468-VM","accountId":225,"format":"VHD","provisioningType":"THIN","poolId":53,"id":6156,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore
":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":false}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":867,"quiescevm":false,"parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":0,"accountId":225,"followRedirects":false}},"wait":0,"bypassHostMaintenance":false}].
2024-09-26 16:58:35,268 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4820302:ctx-f711cc5b) (logid:4e44467e) ===START=== 172.16.11.61
-- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:35,290 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4820302:ctx-f711cc5b ctx-e8b6659b) (logid:4e44467e) ===END===
172.16.11.61 -- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:38,370 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4830812:ctx-fd303fe2) (logid:ef8914d8) ===START=== 172.16.11.61
-- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:38,394 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4830812:ctx-fd303fe2 ctx-aec37e2b) (logid:ef8914d8) ===END===
172.16.11.61 -- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:39,977 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-128:ctx-052ce805) (logid:f365dd96) Seq 77353-4517673376206130891:
Response Received:
2024-09-26 16:58:39,977 DEBUG [c.c.a.t.Request] (DirectAgent-128:ctx-052ce805)
(logid:f365dd96) Seq 77353-4517673376206130891: Processing: { Ans: , MgmtId:
108597816265214, via: 77353(xcp-sr246-u1314.aisrp.local), Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e1971e58-5b6e-4f59-a256-154280cf4d16","id":"0","quiescevm":"false","physicalSize":"0","accountId":"0","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
}
2024-09-26 16:58:39,977 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Seq 77353-4517673376206130891: Received: { Ans: , MgmtId:
108597816265214, via: 77353(xcp-sr246-u1314.aisrp.local), Ver: v1, Flags: 10, {
CreateObjectAnswer } }
2024-09-26 16:58:40,045 DEBUG [o.a.c.s.h.HeuristicRuleHelper]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) No heuristic rules found for zone with ID [7] and heuristic
type [SNAPSHOT]. Returning null.
2024-09-26 16:58:40,049 DEBUG [c.c.s.StatsCollector]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Verifying image storage [7]. Capacity: total=[785 GB],
used=[243 GB], threshold=[89.99999761581421%].
2024-09-26 16:58:40,050 DEBUG [c.c.s.StatsCollector]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Verifying image storage [9]. Capacity: total=[785 GB],
used=[70 GB], threshold=[89.99999761581421%].
2024-09-26 16:58:40,051 DEBUG [c.c.s.StatsCollector]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Verifying image storage [7]. Capacity: total=[785 GB],
used=[243 GB], threshold=[89.99999761581421%].
2024-09-26 16:58:40,101 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) copyAsync inspecting src type SNAPSHOT copyAsync inspecting
dest type SNAPSHOT
2024-09-26 16:58:40,224 ERROR [o.a.c.s.e.DefaultEndPointSelector]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) IR24 select BACKUPSNAPSHOT from primary to secondary 867
dest=867
2024-09-26 16:58:40,236 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CopyCommand
2024-09-26 16:58:40,238 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) We are returning the default host to execute commands because
the source and destination objects are not NFS type.
2024-09-26 16:58:40,243 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Seq 77358-1940207014466653252: Sending { Cmd , MgmtId:
108597816265214, via: 77358(xcp-sr246-u1112.aisrp.local), Ver: v1, Flags:
100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e1971e58-5b6e-4f59-a256-154280cf4d16","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00
GB)
21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioni
ngType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":"0","accountId":"225","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6156","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack
.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00
GB)
21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.254/var/NFS","_role":"Image"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"executeInSequence":"true","options":{"snapshot.backup.to.second
ary":"true","fullSnapshot":"false"},"options2":{},"wait":"21600","bypassHostMaintenance":"false"}}]
}
2024-09-26 16:58:40,245 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Seq 77358-1940207014466653252: Executing: { Cmd , MgmtId:
108597816265214, via: 77358(xcp-sr246-u1112.aisrp.local), Ver: v1, Flags:
100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e1971e58-5b6e-4f59-a256-154280cf4d16","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00
GB)
21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisi
oningType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":"0","accountId":"225","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6156","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudst
ack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00
GB)
21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.254/var/NFS","_role":"Image"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"executeInSequence":"true","options":{"snapshot.backup.to.sec
ondary":"true","fullSnapshot":"false"},"options2":{},"wait":"21600","bypassHostMaintenance":"false"}}]
}
2024-09-26 16:58:40,246 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Executing command CopyCommand:
[{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e1971e58-5b6e-4f59-a256-154280cf4d16","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":false}},"name":"ROOT-5468","size":21474836480,"path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":6156,"vmName":"i-225-5468-VM","accountId":225,"format":"VHD","provisioningType":"THIN","poolId":53,"id":6156,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"parentSnapshotPath":"70424963-
5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":false}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":867,"quiescevm":false,"parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":0,"accountId":225,"followRedirects":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6156","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STORE
UUID=DM7100_LUN3","isManaged":false}},"name":"ROOT-5468","size":21474836480,"path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":6156,"vmName":"i-225-5468-VM","accountId":225,"format":"VHD","provisioningType":"THIN","poolId":53,"id":6156,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.254/var/NFS","_role":"Image"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":867,"quiescevm":false,"physicalSize":0,"accountId":225,"followRedirects":false}},"executeInSequence":true,"options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"false"},"options2":{},"wait":21600,"bypassHostMaintenance":false}].
2024-09-26 16:58:41,264 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) No file SR found for path:
/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156
2024-09-26 16:58:41,264 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Creating file SR for path
[/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] on
host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:41,495 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4824844:ctx-107e93ec) (logid:1bc5742a) ===START=== 172.16.11.61
-- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:41,518 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4824844:ctx-107e93ec ctx-f58fcdf2) (logid:1bc5742a) ===END===
172.16.11.61 -- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:41,520 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Unpluging PBD
[c9ae6161-7db8-f308-db93-4fa060c94f9c] of SR
[f6204e02-6efe-3005-95a3-095f19b458ac] as it is not working properly.
2024-09-26 16:58:41,536 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Forgetting SR
[f6204e02-6efe-3005-95a3-095f19b458ac] as it is not working properly.
2024-09-26 16:58:41,617 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Could not create file SR
[/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] on
host [4208d709-b553-47c0-aee2-7942d370b70c].
2024-09-26 16:58:41,617 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Exception in backupsnapshot
stage due to com.cloud.utils.exception.CloudRuntimeException: Could not
retrieve an already used file SR for path
[/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or
create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:41,617 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) BackupSnapshot Failed due to
Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or
create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:44,625 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4825573:ctx-e72bd43c) (logid:d5a19d4e) ===START=== 172.16.11.61
-- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:44,651 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4825573:ctx-e72bd43c ctx-707eaf3e) (logid:d5a19d4e) ===END===
172.16.11.61 -- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:47,762 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4824845:ctx-517f2c19) (logid:c6e8be8b) ===START=== 172.16.11.61
-- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:47,784 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4824845:ctx-517f2c19 ctx-9a6894c4) (logid:c6e8be8b) ===END===
172.16.11.61 -- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:47,863 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Seq 77358-1940207014466653252:
Response Received:
2024-09-26 16:58:47,864 DEBUG [c.c.a.t.Request] (DirectAgent-322:ctx-abb19c96)
(logid:f365dd96) Seq 77358-1940207014466653252: Processing: { Ans: , MgmtId:
108597816265214, via: 77358(xcp-sr246-u1112.aisrp.local), Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"BackupSnapshot
Failed due to Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or
create a new file SR on host
[4208d709-b553-47c0-aee2-7942d370b70c]","wait":"0","bypassHostMaintenance":"false"}}]
}
2024-09-26 16:58:47,864 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Seq 77358-1940207014466653252:
No more commands found
2024-09-26 16:58:47,864 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Seq 77358-1940207014466653252: Received: { Ans: , MgmtId:
108597816265214, via: 77358(xcp-sr246-u1112.aisrp.local), Ver: v1, Flags: 110,
{ CopyCmdAnswer } }
2024-09-26 16:58:47,885 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Failed to create snapshotBackupSnapshot Failed due to
Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or
create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:47,885 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Updating resource Type = snapshot count for Account = 225
Operation = decreasing Amount = 1
2024-09-26 16:58:47,894 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Updating resource Type = secondary_storage count for Account =
225 Operation = decreasing Amount = (20.00 GB) 21474836480
2024-09-26 16:58:47,901 ERROR [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Take snapshot: 6156 failed
2024-09-26 16:58:47,902 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to
Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or
create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:47,902 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501)
(logid:f365dd96) Rethrow exception
com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to
Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or
create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:47,902 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Done
with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 5468, job
origin: 72454
2024-09-26 16:58:47,902 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96)
Unable to complete AsyncJobVO: {id:72455, userId: 298, accountId: 225,
instanceType: null, instanceId: null, cmd:
com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo:
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAdaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAkwAB3pvbmVJZHN0ABBMamF2YS91dGlsL0xpc3Q7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAOEAAAAAAAABKgAAAAAAABVcdAAUVm9sdW1lQXBpU2VydmljZUltcGwAAHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAABzcQB-AAgAAAAAAAADY3NxAH4ACAAAAAAAABgMcA,
cmdVersion: 0, status: IN_PROGRESS, processStatu
s: 0, resultCode: 0, result: null, initMsid: 108597816265214, completeMsid:
null, lastUpdated: null, lastPolled: null, created: Thu Sep 26 16:58:32 MSK
2024, removed: null}, job origin:72454
2024-09-26 16:58:47,904 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96)
Complete async job-72455, jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvblZNT3AAAAACAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ATxCYWNrdXBTbmFwc2hvdCBGYWlsZWQgZHVlIHRvIEV4Y2VwdGlvbiBpbiBiYWNrdXBzbmFwc2hvdCBzdGFnZSBkdWUgdG8gY29tLmNsb3VkLnV0aWxzLmV4Y2VwdGlvbi5DbG91ZFJ1bnRpbWVFeGNlcHRpb246IENvdWxkIG5vdCByZXRyaWV2ZSBhbiBhbHJlYWR5IHVzZWQgZmlsZSBTUiBmb3IgcGF0aCBbL3Zhci9jbG91ZF9tb3VudC80NDBhY2NhOC02MjJkLTNkZmItODI1ZC1iM2U2MmQ3YmJhODcvc25hcHNob3RzLzIy
NS82MTU2XSBvciBjcmVhdGUgYSBuZXcgZmlsZSBTUiBvbiBob3N0IFs0MjA4ZDcwOS1iNTUzLTQ3YzAtYWVlMi03OTQyZDM3MGI3MGNddXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAAtc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAIQgAGZm9ybWF0SQAKbGluZU51bWJlckwAD2NsYXNzTG9hZGVyTmFtZXEAfgAFTAAOZGVjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAVMAAptb2R1bGVOYW1lcQB-AAVMAA1tb2R1bGVWZXJzaW9ucQB-AAV4cAEAAAF-dAADYXBwdAA6b3JnLmFwYWNoZS5jbG91ZHN0YWNrLnN0b3JhZ2Uuc25hcHNob3QuU25hcHNob3RTZXJ2aWNlSW1wbHQAGFNuYXBzaG90U2VydmljZUltcGwuamF2YXQADmJhY2t1cFNuYXBzaG90cHBzcQB-AAwBAAAAwXEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLnN0b3JhZ2Uuc25hcHNob3QuRGVmYXVsdFNuYXBzaG90U3RyYXRlZ3l0ABxEZWZhdWx0U25hcHNob3RTdHJhdGVneS5qYXZhcQB-ABFwcHNxAH4ADAEAAAWfcQB-AA50AC5jb20uY2xvdWQuc3RvcmFnZS5zbmFwc2hvdC5TbmFwc2hvdE1hbmFnZXJJbXBsdAAYU25hcHNob3RNYW5hZ2VySW1wbC5qYXZhdAAZYmFja3VwU25hcHNob3RUb1NlY29uZGFyeXBwc3EAfgAMAQAABWxxAH4ADnEAfgAWcQB-ABd0AAx0YWtlU25hcHNob3RwcHNxAH4ADAD_____cHQAMGpkay5pbnRlcm5hbC5yZWZsZ
WN0LkdlbmVyYXRlZE1ldGhvZEFjY2Vzc29yMTk1N3B0AAZpbnZva2VwcHNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AHXQACWphdmEuYmFzZXQABzExLjAuMjRzcQB-AAwCAAACNnB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAdcQB-ACFxAH4AInNxAH4ADAEAAAFYcQB-AA50AChvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5zdXBwb3J0LkFvcFV0aWxzdAANQW9wVXRpbHMuamF2YXQAHmludm9rZUpvaW5wb2ludFVzaW5nUmVmbGVjdGlvbnBwc3EAfgAMAQAAAMZxAH4ADnQAPG9yZy5zcHJpbmdmcmFtZXdvcmsuYW9wLmZyYW1ld29yay5SZWZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbnQAH1JlZmxlY3RpdmVNZXRob2RJbnZvY2F0aW9uLmphdmF0AA9pbnZva2VKb2lucG9pbnRwcHNxAH4ADAEAAACjcQB-AA5xAH4AK3EAfgAsdAAHcHJvY2VlZHBwc3EAfgAMAQAAAGFxAH4ADnQAP29yZy5zcHJpbmdmcmFtZXdvcmsuYW9wLmludGVyY2VwdG9yLkV4cG9zZUludm9jYXRpb25JbnRlcmNlcHRvcnQAIEV4cG9zZUludm9jYXRpb25JbnRlcmNlcHRvci5qYXZhcQB-AB1wcHNxAH4ADAEAAAC6cQB-AA5xAH4AK3EAfgAscQB-AC9wcHNxAH4ADAEAAADXcQB-AA50ADRvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuSmRrRHluYW1pY0FvcFByb3h5dA
AXSmRrRHluYW1pY0FvcFByb3h5LmphdmFxAH4AHXBwc3EAfgAMAP____9wdAAXY29tLnN1bi5wcm94eS4kUHJveHkyMzhwcQB-ABpwcHNxAH4ADAEAAArYcQB-AA50ADZvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS52b2x1bWUuVm9sdW1lU2VydmljZUltcGx0ABZWb2x1bWVTZXJ2aWNlSW1wbC5qYXZhcQB-ABpwcHNxAH4ADAEAAA68cQB-AA50ACZjb20uY2xvdWQuc3RvcmFnZS5Wb2x1bWVBcGlTZXJ2aWNlSW1wbHQAGVZvbHVtZUFwaVNlcnZpY2VJbXBsLmphdmF0AB1vcmNoZXN0cmF0ZVRha2VWb2x1bWVTbmFwc2hvdHBwc3EAfgAMAQAAE7lxAH4ADnEAfgA9cQB-AD5xAH4AP3Bwc3EAfgAMAP____9wdAAwamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3IxOTU2cHEAfgAdcHBzcQB-AAwCAAAAK3BxAH4AH3EAfgAgcQB-AB1xAH4AIXEAfgAic3EAfgAMAgAAAjZwcQB-ACRxAH4AJXEAfgAdcQB-ACFxAH4AInNxAH4ADAEAAABpcQB-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAABPHcQB-AA5xAH4APXEAfgA-cQB-AEhwcHNxAH4ADAD_____cHQAMGpkay5pbnRlcm5hbC5yZWZsZWN0LkdlbmVyYXRlZE1ldGhvZEFjY2Vzc29yMTAzNnBxAH4AHXBwc3EAfgAMAgAAACtwcQB-AB9xAH4AIHEAfgAdcQB-ACFxAH4AInNxAH4ADAIAAAI2cHEAfgAkcQB-ACVxAH4
AHXEAfgAhcQB-ACJzcQB-AAwBAAABWHEAfgAOcQB-ACdxAH4AKHEAfgApcHBzcQB-AAwBAAAAxnEAfgAOcQB-ACtxAH4ALHEAfgAtcHBzcQB-AAwBAAAAo3EAfgAOcQB-ACtxAH4ALHEAfgAvcHBzcQB-AAwBAAAAYXEAfgAOcQB-ADFxAH4AMnEAfgAdcHBzcQB-AAwBAAAAunEAfgAOcQB-ACtxAH4ALHEAfgAvcHBzcQB-AAwBAAAA13EAfgAOcQB-ADVxAH4ANnEAfgAdcHBzcQB-AAwA_____3B0ABdjb20uc3VuLnByb3h5LiRQcm94eTI0N3BxAH4ASHBwc3EAfgAMAQAAAGZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACjnEAfgAOdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AAwBAAAAMHEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4ADAEAAAA3cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABmcQB-AA50AEBvcmcuYXBhY2hlLmNsb3Vk
c3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-AGR0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA0cQB-AA5xAH4AZ3EAfgBkdAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAtcQB-AA50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AYHEAfgBhcHBzcQB-AAwBAAACWnEAfgAOcQB-AFtxAH4AXHEAfgBhcHBzcQB-AAwCAAACA3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AZXEAfgAhcQB-ACJzcQB-AAwCAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AGFxAH4AIXEAfgAic3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AIXEAfgAic3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgB2cQB-AGFxAH4AIXEAfgAic3EAfgAMAgAAAz1wdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AGFxAH4AIXEAfgAic3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
2024-09-26 16:58:47,905 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96)
Publish async job-72455 complete on message bus
2024-09-26 16:58:47,905 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Wake
up jobs related to job-72455
2024-09-26 16:58:47,905 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96)
Update db status for job-72455
2024-09-26 16:58:47,906 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Wake
up jobs joined with job-72455 and disjoin all subjobs created from job- 72455
2024-09-26 16:58:47,914 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Done
executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-72455
2024-09-26 16:58:47,916 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96)
Remove job-72455 from job monitoring
2024-09-26 16:58:47,930 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd]
(API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96)
Failed to create snapshot due to an internal error creating snapshot for volume
da420806-0ff3-43b8-9ee0-4aa2adfd9b99
2024-09-26 16:58:47,935 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Complete async
job-72454, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to create snapshot due to an internal error creating snapshot for volume
da420806-0ff3-43b8-9ee0-4aa2adfd9b99"}
2024-09-26 16:58:47,936 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Publish async
job-72454 complete on message bus
2024-09-26 16:58:47,936 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Wake up jobs
related to job-72454
2024-09-26 16:58:47,936 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Update db status
for job-72454
2024-09-26 16:58:47,937 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Wake up jobs
joined with job-72454 and disjoin all subjobs created from job- 72454
2024-09-26 16:58:47,941 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Done executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-72454
2024-09-26 16:58:47,941 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Remove job-72454
from job monitoring
2024-09-26 16:58:50,891 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4824742:ctx-04767efc) (logid:00d3b60a) ===START=== 172.16.11.61
-- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:50,913 DEBUG [c.c.a.ApiServlet]
(qtp501107890-4824742:ctx-04767efc ctx-ff2eca28) (logid:00d3b60a) ===END===
172.16.11.61 -- GET
jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
~~~
GitHub link:
https://github.com/apache/cloudstack/discussions/9754#discussioncomment-10810169
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
