GitHub user TadiosAbebe created a discussion: setting custom timezone impacts
VM HA
I wasn’t entirely sure whether this should be classified as a bug or simply an
edge case, but I’m documenting it here in case it helps someone in the future.
Earlier today, while testing VM HA on a lab cluster, I noticed that a VM with
HA enabled took more than 3 hours to automatically restart after being shut
down from inside the OS. When I investigated the delay, the management server
logs repeatedly showed the message:
```
vm id: 258 - time since last state update (-10620285 ms) has not passed
graceful period yet
```
The value -10620285 ms caught my attention, it corresponds to roughly 3 hours,
which matched the delay I was observing.
looking into the code,
```
long milliSecondsSinceLastStateUpdate = currentTime.getTime() -
vmStateUpdateTime.getTime();
```
Here, currentTime is obtained from:
```
Date currentTime = DateUtil.currentGMTTime();
```
Meanwhile, vmStateUpdateTime comes from the database
(vm_instance.power_state_update_time), which is updated through:
```
StringBuilder sql = new StringBuilder("UPDATE `cloud`.`vm_instance` SET
" +
"`power_host` = ?, `power_state_update_time` = now(),
`power_state` = CASE ");
```
At that point, the root cause became clear: Our servers were configured with
the local timezone (GMT+3) for logging convenience. This meant:
currentGMTTime() returned UTC, NOW() in MySQL returned local time (UTC+3).
As a result, CloudStack ended up comparing timestamps from two different
timezones, producing the large negative time difference and causing the HA
mechanism to wait the full (incorrect) interval, approximately 3 hours.
To resolve this, I switched the MySQL server’s timezone to UTC (UTC+0) so that
both values align. Before realizing this, I briefly changed the timezone on the
management server, compute server, and database server to UTC as well. After
adjusting the database timezone, I had to restart the MySQL service for the
change to take effect.
I hope this helps someone encountering a similar issue. I would also appreciate
any suggestions from the community on how to maintain local timezones on
servers without impacting services that rely on strict time consistency like
this.
GitHub link: https://github.com/apache/cloudstack/discussions/12183
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
