On 5/30/2016 4:40 AM, Stefano Stabellini wrote:
On Fri, 27 May 2016, Chenxiao Zhao wrote:
Hi,
My board is Hikey on which have octa-core of arm cortex-a53. I have applied
patches [1] to try vm save/restore on arm.
These patches originally do not working on arm64. I have made some changes
based on patch set [2].
Hello Chenxiao,
thanks for your interest in Xen on ARM save/restore.
Hi Stefano,
Thanks for your advice.
I found a possible reason that cause the restore failure is that xen
always failed on p2m_lookup for guest domain.
I called dump_p2m_lookup in p2m_look() and get the output like below:
(XEN) dom1 IPA 0x0000000039001000
(XEN) P2M @ 0000000801e7ce80 mfn:0x79f3a
(XEN) Using concatenated root table 0
(XEN) 1ST[0x0] = 0x0040000079f3c77f
(XEN) 2ND[0x1c8] = 0x0000000000000000
My question is:
1. who is responsible for restoring p2m table, Xen or guest kernel?
2. After restore, the vm always get zero memory space, but there is no
error reported on the restore progress. Does the memory requested by the
guest kernel or should be allocated early by hypervisor?
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 8 r----- 15.0
guest 1 0 1 --p--- 0.0
What I have got so far is
1. if I run 'xl save -p guest memState' to leave guest in suspend state, then
run 'xl unpause guest'.
the guest can resume successfully. so I suppose the guest works find on
suspend/resume.
2. if I run 'xl restore -p memState' to restore guest and use xenctx to dump
all vcpu's registers.
all the registers are identical to the state on save. After I run 'xl
unpause guest', I got no error but can not connect to console.
After restore the guest's PC is at a function called
user_disable_single_step(), which is called by single_step_handler().
My question is
1. How could I debug guest on restore progress? are there any tools available?
Nothing special. You can use ctrl-AAA on the console to switch to the
hypervisor console and see the state of the guest. You can also add some
debug printks; if the console doesn't work you can use
dom0_write_console in Linux to get messages out of your guest (you need
to compile Xen with debug=y for that to work).
2. From my understanding, the restore not working is because some status is
missing when saving.
e.g. on cpu_save, it know the domain is 64bit, but on cpu_load, it always
think it is a 32bit domain. so I have hard coded the domain type to
DOMAIN_64BIT.
Am I correct?
If Xen thinks the domain is 32-bit at restore, it must be a bug.
3. How could I dump all VM's status? I only found xenctx can dump vcpu's
registers.
You can use the hypervisor console via the ctrl-aaa menu.
I have attached my patch and log below.
Looking forward for your feedback.
Thanks
xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 8 r----- 11.7
root@linaro-alip:~# xl create guest.cfg
Parsing config from guest.cfg
[ 39.238723] xen-blkback: ring-ref 8, event-channel 3, protocol 1 (arm-abi)
persistent grants
root@linaro-alip:~# xl save -p guest memState
Saving to memState new xl format (info 0x3/0x0/931)
xc: info: Saving domain 1, type ARM
(XEN) HVM1 save: VCPU
(XEN) HVM1 save: A15_TIMER
(XEN) HVM1 save: GICV2_GICD
(XEN) HVM1 save: GICV2_GICC
(XEN) HVM1 save: GICV3
root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 1
PC: ffffffc0000ab028
LR: ffffffc00050458c
ELR_EL1: ffffffc000086b34
CPSR: 200001c5
SPSR_EL1: 60000145
SP_EL0: 0000007ff6f2a850
SP_EL1: ffffffc0140a7ca0
x0: 0000000000000001 x1: 00000000deadbeef x2: 0000000000000002
x3: 0000000000000002 x4: 0000000000000004 x5: 0000000000000000
x6: 000000000000001b x7: 0000000000000001 x8: 000000618e589e00
x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000000000
x12: 00000000000001a3 x13: 000000001911a7d9 x14: 0000000000002ee0
x15: 0000000000000005 x16: 00000000deadbeef x17: 0000000000000001
x18: 0000000000000007 x19: 0000000000000000 x20: ffffffc014163d58
x21: ffffffc014163cd8 x22: 0000000000000001 x23: 0000000000000140
x24: ffffffc000d5bb18 x25: ffffffc014163cd8 x26: 0000000000000000
x27: 0000000000000000 x28: 0000000000000000 x29: ffffffc0140a7ca0
SCTLR: 34d5d91d
TTBCR: 00000032b5193519
TTBR0: 002d000054876000
TTBR1: 0000000040dcf000
root@linaro-alip:~# xl destroy guest
(XEN) mm.c:1265:d0v1 gnttab_mark_dirty not implemented yet
root@linaro-alip:~# xl restore -p memState
Loading new save file memState (new xl fmt info 0x3/0x0/931)
Savefile contains xl domain config in JSON format
Parsing config from <saved>
xc: info: (XEN) HVM2 restore: VCPU 0
Found ARM domain from Xen 4.7
xc: info: Restoring domain
(XEN) HVM2 restore: A15_TIMER 0
(XEN) HVM2 restore: A15_TIMER 0
(XEN) HVM2 restore: GICV2_GICD 0
(XEN) HVM2 restore: GICV2_GICC 0
(XEN) GICH_LRs (vcpu 0) mask=0
(XEN) VCPU_LR[0]=0
(XEN) VCPU_LR[1]=0
(XEN) VCPU_LR[2]=0
(XEN) VCPU_LR[3]=0
xc: info: Restore successful
xc: info: XenStore: mfn 0x39001, dom 0, evt 1
xc: info: Console: mfn 0x39000, dom 0, evt 2
root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 2
PC: ffffffc0000ab028
LR: ffffffc00050458c
ELR_EL1: ffffffc000086b34
CPSR: 200001c5
SPSR_EL1: 60000145
SP_EL0: 0000007ff6f2a850
SP_EL1: ffffffc0140a7ca0
x0: 0000000000000000 x1: 00000000deadbeef x2: 0000000000000002
x3: 0000000000000002 x4: 0000000000000004 x5: 0000000000000000
x6: 000000000000001b x7: 0000000000000001 x8: 000000618e589e00
x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000000000
x12: 00000000000001a3 x13: 000000001911a7d9 x14: 0000000000002ee0
x15: 0000000000000005 x16: 00000000deadbeef x17: 0000000000000001
x18: 0000000000000007 x19: 0000000000000000 x20: ffffffc014163d58
x21: ffffffc014163cd8 x22: 0000000000000001 x23: 0000000000000140
x24: ffffffc000d5bb18 x25: ffffffc014163cd8 x26: 0000000000000000
x27: 0000000000000000 x28: 0000000000000000 x29: ffffffc0140a7ca0
SCTLR: 34d5d91d
TTBCR: 00000000b5193519
TTBR0: 002d000054876000
TTBR1: 0000000040dcf000
root@linaro-alip:~# xl unpause guest
root@linaro-alip:~# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 8 r----- 22.2
guest 2 0 1 r----- 4.8
root@linaro-alip:~# /usr/lib/xen/bin/xenctx -a 2
PC: ffffffc000084a00
LR: ffffffc00050458c
ELR_EL1: ffffffc000084a00
CPSR: 000003c5
SPSR_EL1: 000003c5
SP_EL0: 0000007ff6f2a850
SP_EL1: ffffffc0140a7ca0
x0: 0000000000000000 x1: 00000000deadbeef x2: 0000000000000002
x3: 0000000000000002 x4: 0000000000000004 x5: 0000000000000000
x6: 000000000000001b x7: 0000000000000001 x8: 000000618e589e00
x9: 0000000000000000 x10: 0000000000000000 x11: 0000000000000000
x12: 00000000000001a3 x13: 000000001911a7d9 x14: 0000000000002ee0
x15: 0000000000000005 x16: 00000000deadbeef x17: 0000000000000001
x18: 0000000000000007 x19: 0000000000000000 x20: ffffffc014163d58
x21: ffffffc014163cd8 x22: 0000000000000001 x23: 0000000000000140
x24: ffffffc000d5bb18 x25: ffffffc014163cd8 x26: 0000000000000000
x27: 0000000000000000 x28: 0000000000000000 x29: ffffffc0140a7ca0
SCTLR: 34d5d91d
TTBCR: 00000000b5193519
TTBR0: 002d000054876000
TTBR1: 0000000040dcf000
root@linaro-alip:~# xl console guest
xenconsole: Could not read tty from store: Success
root@linaro-alip:~#
diff --git a/xen/arch/arm/hvm.c b/xen/arch/arm/hvm.c
index aee3353..411bab4 100644
--- a/xen/arch/arm/hvm.c
+++ b/xen/arch/arm/hvm.c
@@ -120,7 +120,8 @@ static int cpu_save(struct domain *d, hvm_domain_context_t
*h)
ctxt.dfar = v->arch.dfar;
ctxt.dfsr = v->arch.dfsr;
#else
- /* XXX 64-bit */
+ ctxt.far = v->arch.far;
+ ctxt.esr = v->arch.esr;
#endif
#ifdef CONFIG_ARM_32
@@ -187,6 +188,9 @@ static int cpu_load(struct domain *d, hvm_domain_context_t
*h)
if ( hvm_load_entry(VCPU, h, &ctxt) != 0 )
return -EINVAL;
+#ifdef CONFIG_ARM64
+ v->arch.type = DOMAIN_64BIT;
+#endif
v->arch.sctlr = ctxt.sctlr;
v->arch.ttbr0 = ctxt.ttbr0;
v->arch.ttbr1 = ctxt.ttbr1;
@@ -199,7 +203,8 @@ static int cpu_load(struct domain *d, hvm_domain_context_t
*h)
v->arch.dfar = ctxt.dfar;
v->arch.dfsr = ctxt.dfsr;
#else
- /* XXX 64-bit */
+ v->arch.far = ctxt.far;
+ v->arch.esr = ctxt.esr;
#endif
#ifdef CONFIG_ARM_32
diff --git a/xen/include/public/arch-arm/hvm/save.h
b/xen/include/public/arch-arm/hvm/save.h
index db916b1..89e6e89 100644
--- a/xen/include/public/arch-arm/hvm/save.h
+++ b/xen/include/public/arch-arm/hvm/save.h
@@ -46,8 +46,12 @@ DECLARE_HVM_SAVE_TYPE(HEADER, 1, struct hvm_save_header);
struct hvm_hw_cpu
{
+#ifdef CONFIG_ARM_32
uint64_t vfp[34]; /* Vector floating pointer */
/* VFP v3 state is 34x64 bit, VFP v4 is not yet supported */
+#else
+ uint64_t vfp[66];
+#endif
/* Guest core registers */
struct vcpu_guest_core_regs core_regs;
@@ -60,6 +64,9 @@ struct hvm_hw_cpu
uint32_t dacr;
uint64_t par;
+ uint64_t far;
+ uint64_t esr;
+
uint64_t mair0, mair1;
uint64_t tpidr_el0;
uint64_t tpidr_el1;
[1] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01053.html
[2] http://lists.xen.org/archives/html/xen-devel/2014-04/msg01544.html
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
