Hi, One thing I screwed - I forgot to remove changes log from an internal review, so please ignore it. This is officially the first version.
Thanks, Mirela On Mon, Nov 12, 2018 at 12:31 PM Mirela Simonovic < mirela.simono...@aggios.com> wrote: > This series contains support for suspend to RAM (in the following text just > 'suspend') for Xen on arm64. The implementation is aligned with the design > specification that has been proposed on xen-devel list: > https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html > > At a high-level the series contains: > 1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call > 2) Support for resuming a guest on an interrupt targeted to that guest > 3) Support for suspending Xen after dom0 finalizes the suspend > 4) Support for resuming Xen on an interrupt that is targeted to a guest > > > -------------------------------------------------------------------------------- > In more details: > > *** About suspending/resuming guests > > The patches included in this series allow PSCI compliant guests that have > support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to > suspend and resume on top of Xen without any EL1 code changes. > > During their suspend procedure guests will hot-unplug their secondary CPUs, > triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the > suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI. > Guests will save/restore their own EL1 context on suspend/resume. > > A guest is expected to leave enabled interrupts that are considered to be > its > wake-up sources. Those interrupts will be able to wake up the guest. This > holds > regardless of the state of the underlying software layers, i.e. whether > Xen gets > suspended or not doesn't affect the ability of the guest to wake up. > > First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from > which > the guest assumes to start on resume. On resume, guests assume to be > running in > an environment whose state matches the CPU state after reset, e.g. with > reset > register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the > VCPU context and save the resume entry point into program counter before > the > guest's VCPU gets scheduled in on resume. This is done when the guest > finalizes > its suspend by calling PSCI SYSTEM_SUSPEND. In addition, we need to ensure > that > the resume-ready VCPU's context does not get overwritten later upon the > context > switch when the VCPU is scheduled out. > Xen also needs to take care that the guest's view of GIC and timer gets > saved. > Also, while a guest is suspended its watchdogs are paused, in order to > avoid > watchdog triggered shutdown of a guest that has been asleep for a period > of time > that is longer than the watchdog period. > > After this point, from Xen's point of view a suspended guest has one VCPU > blocked, waiting for an interrupt. When such an interrupt comes, Xen will > unblock the VCPU of the suspended domain, which results in the guest > resuming. > > *** About suspending/resuming Xen > > Xen starts its own suspend procedure once dom0 is suspended. Dom0 is > considered to be the decision maker for EL1 and EL2. > On suspend, Xen will first freeze all domains. Then, Xen disables physical > secondary CPUs, which leads to physical CPU_OFF to be called by each > secondary > CPU. After that Xen finalizes the suspend from the boot CPU. > > This consists of suspending the timer, i.e. suppressing its interrupts (we > don't > want to be woken up by a timer, there is no VCPU ready to be scheduled). > Then > the state of GIC is saved, console is suspended, and CPU context is saved. > The > saved context tells where Xen needs to continue execution on resume. > Since Xen will resume with MMU disabled, the first thing to do in resume > is to > resume memory management in order to be able to access the context that > needs to > be restored (we know virtual address of the context data). Finally Xen > calls > SYSTEM_SUSPEND PSCI to the EL3. > > When resuming, all the steps done in suspend need to be reverted. This is > completed by unblocking dom0's VCPU, because we always want the dom0 to > resume, > regardless of the target domain whose interrupt woke up Xen. > > *** Handling of unprivileged guests during Xen suspend/resume > > Any domU that is not suspended when dom0 suspends will be frozen, domUs > that are > already suspended remain suspended. On resume the suspended domUs still > remain > suspended (unless their wake interrupt caused Xen to wake) while the > others will > be thawed. > > For more details please refer to patches or the design specification: > https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html > > > -------------------------------------------------------------------------------- > The series does not include: > a) UART driver-specific suspend/resume that gets called when console > suspends > b) SMMU suspend/resume > c) Suspend coordination support that would allow dom0 to request domUs to > suspend > These will be submitted in the following series. > > Mirela Simonovic (16): > xen/arm: Implement PSCI system suspend call (virtual interface) > xen/arm: Save GIC and virtual timer context when the domain suspends > xen/arm: While a domain is suspended put its watchdogs on pause > xen/arm: Trigger Xen suspend when Dom0 completes suspend > xen/x86: Move freeze/thaw_domains into common files > xen/arm: Freeze domains on suspend and thaw them on resume > xen/arm: Disable/enable non-boot physical CPUs on suspend/resume > xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume > xen/arm: Implement GIC suspend/resume functions (gicv2 only) > xen/arm: Suspend/resume GIC on system suspend/resume > xen/arm: Suspend/resume timer interrupt generation > xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface) > xen/arm: Resume memory management on Xen resume > xen/arm: Save/restore context on suspend/resume > xen/arm: Resume Dom0 after Xen resumes > xen/arm: Suspend/resume console on Xen suspend/resume > > Saeed Nowshadi (2): > xen/arm: Move code that initializes VCPU context into a separate > function > xen/arm: Convert setting MMU page tables code into a routine > > xen/arch/arm/Makefile | 1 + > xen/arch/arm/arm64/entry.S | 178 ++++++++++++++++++++++++ > xen/arch/arm/arm64/head.S | 265 ++++++++++++++++++----------------- > xen/arch/arm/domain.c | 62 ++++++--- > xen/arch/arm/gic-v2.c | 147 ++++++++++++++++++++ > xen/arch/arm/gic.c | 27 ++++ > xen/arch/arm/mm.c | 1 + > xen/arch/arm/psci.c | 16 +++ > xen/arch/arm/suspend.c | 292 > +++++++++++++++++++++++++++++++++++++++ > xen/arch/arm/time.c | 22 +++ > xen/arch/arm/vpsci.c | 19 +++ > xen/arch/x86/acpi/power.c | 28 ---- > xen/common/domain.c | 29 ++++ > xen/common/schedule.c | 38 +++++ > xen/include/asm-arm/gic.h | 8 ++ > xen/include/asm-arm/perfc_defn.h | 1 + > xen/include/asm-arm/psci.h | 3 + > xen/include/asm-arm/suspend.h | 39 ++++++ > xen/include/asm-arm/time.h | 3 + > xen/include/xen/domain.h | 1 + > xen/include/xen/sched.h | 11 ++ > xen/include/xen/timer.h | 3 + > 22 files changed, 1019 insertions(+), 175 deletions(-) > create mode 100644 xen/arch/arm/suspend.c > create mode 100644 xen/include/asm-arm/suspend.h > > -- > 2.13.0 > >
_______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
