Hello CloudStack developers / users,
We hit a reproducible VMware attach failure on Apache CloudStack 4.23.0 with
vCenter/vSphere 8.
Summary
=======
attachVolume fails with:
Failed to attach volume ... to VM ...
Agent log shows:
java.io.IOException: Server returned HTTP response code: 500
for URL: https://<vcenter>/folder/<vm>/<volume>.vmdk?dcPath=...&dsName=...
Root cause (chicken-and-egg)
============================
In VirtualMachineMO.attachDisk(...), ACS calls updateVmdkAdapter(vmdk,
controller)
*before* reconfigVMTask() adds the disk to the VM.
updateVmdkAdapter() -> getVmdkFileInfo() -> VmwareContext.getResourceContent()
downloads the VMDK descriptor via the vCenter datastore *folder* HTTP API.
On our vCenter 8, that /folder/ download returns HTTP 500 for VMDKs that are
*not yet attached* to any VM. The same VMDK downloads fine *after* it is
attached
(verified with govc datastore.download).
So attach never reaches reconfigVMTask().
Related secondary issue
=======================
VmdkAdapterType.getType("lsilogicsas") returns "none", so
updateAdapterTypeIfRequired()
on resize can rewrite ddb.adapterType from "lsilogicsas" to "lsilogic", which
then
interacts badly with LSI Logic SAS controllers / capacity rewrite paths
(spec.deviceChange.device / capacityInKB vs capacityInBytes mismatches).
Environment
===========
- ACS: 4.23.0.0
- Hypervisor: VMware / vCenter 8
- Guest: Windows Server 2022 (UEFI Secure), root/dataDiskController=lsilogicsas
- Storage: VMFS (SAS / NVM)
Suggested upstream fix (preferred)
==================================
1) In VirtualMachineMO.attachDisk: move updateVmdkAdapter() to *after* a
successful
reconfigVMTask() (disk is attached => folder download works), OR
2) Make adapter rewrite best-effort: if getVmdkFileInfo() fails, log a warning
and
continue with attach (SCSI controller on the VM is what matters).
Also consider treating "lsilogicsas" as already-valid in
updateAdapterTypeIfRequired()
(do not rewrite to lsilogic), since getType() currently maps it to none.
Proposed code change (best-effort skip)
=======================================
File: plugins/hypervisors/vmware/.../VirtualMachineMO.java
Methods: updateVmdkAdapter, updateAdapterTypeIfRequired
```java
public void updateVmdkAdapter(String vmdkFileName, String diskController)
throws Exception {
String currentAdapterType;
DiskControllerType diskControllerType =
DiskControllerType.getType(diskController);
VmdkAdapterType vmdkAdapterType =
VmdkAdapterType.getAdapterType(diskControllerType);
if (vmdkAdapterType == VmdkAdapterType.none) {
String message = "Failed to attach disk due to invalid vmdk adapter
type for vmdk file ["
+ vmdkFileName + "] with controller : " + diskControllerType;
logger.debug(message);
throw new Exception(message);
}
String newAdapterType = vmdkAdapterType.toString();
Pair<VmdkFileDescriptor, byte[]> vmdkInfo;
try {
vmdkInfo = this.getVmdkFileInfo(vmdkFileName);
} catch (Exception e) {
// vCenter /folder API can return HTTP 500 for VMDKs not yet attached
to a VM.
// Skip rewrite; controller type on the VM is what matters for attach.
logger.warn("Skipping VMDK adapter update for " + vmdkFileName
+ " (descriptor unreadable, attaching anyway): " + e.getMessage());
return;
}
VmdkFileDescriptor vmdkFileDescriptor = (VmdkFileDescriptor)
vmdkInfo.first();
boolean isVmfsSparseFile = vmdkFileDescriptor.isVmfsSparseFile();
if (!isVmfsSparseFile
&& !(currentAdapterType = vmdkFileDescriptor.getAdapterType())
.equalsIgnoreCase(newAdapterType)) {
logger.info("Updating adapter type to " + newAdapterType
+ " for VMDK file " + vmdkFileName);
Pair<DatacenterMO, String> dcInfo = this.getOwnerDatacenter();
byte[] newVmdkContent = VmdkFileDescriptor.changeVmdkAdapterType(
(byte[]) vmdkInfo.second(), newAdapterType);
String vmdkUploadUrl = this.getContext().composeDatastoreBrowseUrl(
((DatacenterMO) dcInfo.first()).getName(), vmdkFileName);
this.getContext().uploadResourceContent(vmdkUploadUrl, newVmdkContent);
logger.info("Updated VMDK file " + vmdkFileName);
}
}
public void updateAdapterTypeIfRequired(String vmdkFileName) throws Exception {
Pair<VmdkFileDescriptor, byte[]> vmdkInfo;
try {
vmdkInfo = this.getVmdkFileInfo(vmdkFileName);
} catch (Exception e) {
logger.warn("Skipping updateAdapterTypeIfRequired for "
+ vmdkFileName + ": " + e.getMessage());
return;
}
VmdkFileDescriptor vmdkFileDescriptor = (VmdkFileDescriptor)
vmdkInfo.first();
boolean isVmfsSparseFile = vmdkFileDescriptor.isVmfsSparseFile();
if (!isVmfsSparseFile) {
String currentAdapterTypeStr = vmdkFileDescriptor.getAdapterType();
// Do not treat lsilogicsas as "none" and rewrite to lsilogic.
if (currentAdapterTypeStr != null
&& currentAdapterTypeStr.equalsIgnoreCase("lsilogicsas")) {
return;
}
if (logger.isTraceEnabled()) {
logger.trace("Detected adapter type " + currentAdapterTypeStr
+ " for VMDK file " + vmdkFileName);
}
VmdkAdapterType currentAdapterType =
VmdkAdapterType.getType(currentAdapterTypeStr);
if (currentAdapterType == VmdkAdapterType.none) {
VmdkAdapterType newAdapterType = VmdkAdapterType.lsilogic;
logger.debug("Updating adapter type to " + newAdapterType
+ " from " + currentAdapterTypeStr
+ " for VMDK file " + vmdkFileName);
Pair<DatacenterMO, String> dcInfo = this.getOwnerDatacenter();
byte[] newVmdkContent = VmdkFileDescriptor.changeVmdkAdapterType(
(byte[]) vmdkInfo.second(), newAdapterType.toString());
String vmdkUploadUrl = this.getContext().composeDatastoreBrowseUrl(
((DatacenterMO) dcInfo.first()).getName(), vmdkFileName);
this.getContext().uploadResourceContent(vmdkUploadUrl,
newVmdkContent);
logger.debug("Updated VMDK file " + vmdkFileName);
}
}
}
```
Workaround we applied in production
===================================
Temporary bytecode NOP of updateVmdkAdapter / updateAdapterTypeIfRequired in
cloudstack-4.23.0.0.jar on the management server. After that, panel attachVolume
succeeds. Happy to open a GitHub PR / JIRA if useful.
Happy to provide more logs or test a patch build.
Thanks,
Umit Eyigun
TRTEK
[email protected]
