Introduce support for allocating and initializing the root page table required for RISC-V stage-2 address translation.
To implement root page table allocation the following is introduced: - p2m_get_clean_page() and p2m_alloc_root_table(), p2m_allocate_root() helpers to allocate and zero a 16 KiB root page table, as mandated by the RISC-V privileged specification for Sv32x4/Sv39x4/Sv48x4/Sv57x4 modes. - Update p2m_init() to inititialize p2m_root_order. - Add maddr_to_page() and page_to_maddr() macros for easier address manipulation. - Introduce paging_ret_pages_to_domheap() to return some pages before allocate 16 KiB pages for root page table. - Allocate root p2m table after p2m pool is initialized. - Add construct_hgatp() to construct the hgatp register value based on p2m->root, p2m->hgatp_mode and VMID. Signed-off-by: Oleksii Kurochko <oleksii.kuroc...@gmail.com> --- Changes in v3: - Drop insterting of p2m->vmid in hgatp_from_page() as now vmid is allocated per-CPU, not per-domain, so it will be inserted later somewhere in context_switch or before returning control to a guest. - use BIT() to init nr_pages in p2m_allocate_root() instead of open-code BIT() macros. - Fix order in clear_and_clean_page(). - s/panic("Specify more xen,domain-p2m-mem-mb\n")/return NULL. - Use lock around a procedure of returning back pages necessary for p2m root table. - Update the comment about allocation of page for root page table. - Update an argument of hgatp_from_page() to "struct page_info *p2m_root_page" to be consistent with the function name. - Use p2m_get_hostp2m(d) instead of open-coding it. - Update the comment above the call of p2m_alloc_root_table(). - Update the comments in p2m_allocate_root(). - Move part which returns some page to domheap before root page table allocation to paging.c. - Pass p2m_domain * instead of struct domain * for p2m_alloc_root_table(). - Introduce construct_hgatp() instead of hgatp_from_page(). - Add vmid and hgatp_mode member of struct p2m_domain. - Add explanatory comment above clean_dcache_va_range() in clear_and_clean_page(). - Introduce P2M_ROOT_ORDER and P2M_ROOT_PAGES. - Drop vmid member from p2m_domain as now we are using per-pCPU VMID allocation. - Update a declaration of construct_hgatp() to recieve VMID as it isn't per-VM anymore. - Drop hgatp member of p2m_domain struct as with the new VMID scheme allocation construction of hgatp will be needed more often. - Drop is_hardware_domain() case in p2m_allocate_root(), just always allocate root using p2m pool pages. - Refactor p2m_alloc_root_table() and p2m_alloc_table(). --- Changes in v2: - This patch was created from "xen/riscv: introduce things necessary for p2m initialization" with the following changes: - [clear_and_clean_page()] Add missed call of clean_dcache_va_range(). - Drop p2m_get_clean_page() as it is going to be used only once to allocate root page table. Open-code it explicittly in p2m_allocate_root(). Also, it will help avoid duplication of the code connected to order and nr_pages of p2m root page table. - Instead of using order 2 for alloc_domheap_pages(), use get_order_from_bytes(KB(16)). - Clear and clean a proper amount of allocated pages in p2m_allocate_root(). - Drop _info from the function name hgatp_from_page_info() and its argument page_info. - Introduce HGATP_MODE_MASK and use MASK_INSR() instead of shift to calculate value of hgatp. - Drop unnecessary parentheses in definition of page_to_maddr(). - Add support of VMID. - Drop TLB flushing in p2m_alloc_root_table() and do that once when VMID is re-used. [Look at p2m_alloc_vmid()] - Allocate p2m root table after p2m pool is fully initialized: first return pages to p2m pool them allocate p2m root table. --- xen/arch/riscv/include/asm/mm.h | 4 + xen/arch/riscv/include/asm/p2m.h | 12 +++ xen/arch/riscv/include/asm/paging.h | 2 + xen/arch/riscv/include/asm/riscv_encoding.h | 6 ++ xen/arch/riscv/p2m.c | 90 +++++++++++++++++++++ xen/arch/riscv/paging.c | 30 +++++++ 6 files changed, 144 insertions(+) diff --git a/xen/arch/riscv/include/asm/mm.h b/xen/arch/riscv/include/asm/mm.h index 9283616c02..dd8cdc9782 100644 --- a/xen/arch/riscv/include/asm/mm.h +++ b/xen/arch/riscv/include/asm/mm.h @@ -167,6 +167,10 @@ extern struct page_info *frametable_virt_start; #define mfn_to_page(mfn) (frametable_virt_start + mfn_x(mfn)) #define page_to_mfn(pg) _mfn((pg) - frametable_virt_start) +/* Convert between machine addresses and page-info structures. */ +#define maddr_to_page(ma) mfn_to_page(maddr_to_mfn(ma)) +#define page_to_maddr(pg) mfn_to_maddr(page_to_mfn(pg)) + static inline void *page_to_virt(const struct page_info *pg) { return mfn_to_virt(mfn_x(page_to_mfn(pg))); diff --git a/xen/arch/riscv/include/asm/p2m.h b/xen/arch/riscv/include/asm/p2m.h index f8051ed893..3c37a708db 100644 --- a/xen/arch/riscv/include/asm/p2m.h +++ b/xen/arch/riscv/include/asm/p2m.h @@ -9,6 +9,10 @@ #include <asm/page-bits.h> +extern unsigned int p2m_root_order; +#define P2M_ROOT_ORDER p2m_root_order +#define P2M_ROOT_PAGES BIT(P2M_ROOT_ORDER, U) + #define paddr_bits PADDR_BITS /* Get host p2m table */ @@ -24,6 +28,12 @@ struct p2m_domain { /* Pages used to construct the p2m */ struct page_list_head pages; + /* The root of the p2m tree. May be concatenated */ + struct page_info *root; + + /* G-stage (stage-2) address translation mode */ + unsigned long hgatp_mode; + /* Indicate if it is required to clean the cache when writing an entry */ bool clean_pte; @@ -127,6 +137,8 @@ static inline void p2m_altp2m_check(struct vcpu *v, uint16_t idx) /* Not supported on RISCV. */ } +unsigned long construct_hgatp(struct p2m_domain *p2m, uint16_t vmid); + #endif /* ASM__RISCV__P2M_H */ /* diff --git a/xen/arch/riscv/include/asm/paging.h b/xen/arch/riscv/include/asm/paging.h index 8fdaeeb2e4..557fbd1abc 100644 --- a/xen/arch/riscv/include/asm/paging.h +++ b/xen/arch/riscv/include/asm/paging.h @@ -10,4 +10,6 @@ int paging_domain_init(struct domain *d); int paging_freelist_init(struct domain *d, unsigned long pages, bool *preempted); +bool paging_ret_pages_to_domheap(struct domain *d, unsigned int nr_pages); + #endif /* ASM_RISCV_PAGING_H */ diff --git a/xen/arch/riscv/include/asm/riscv_encoding.h b/xen/arch/riscv/include/asm/riscv_encoding.h index 6cc8f4eb45..8362df8784 100644 --- a/xen/arch/riscv/include/asm/riscv_encoding.h +++ b/xen/arch/riscv/include/asm/riscv_encoding.h @@ -133,11 +133,13 @@ #define HGATP_MODE_SV48X4 _UL(9) #define HGATP32_MODE_SHIFT 31 +#define HGATP32_MODE_MASK _UL(0x80000000) #define HGATP32_VMID_SHIFT 22 #define HGATP32_VMID_MASK _UL(0x1FC00000) #define HGATP32_PPN _UL(0x003FFFFF) #define HGATP64_MODE_SHIFT 60 +#define HGATP64_MODE_MASK _ULL(0xF000000000000000) #define HGATP64_VMID_SHIFT 44 #define HGATP64_VMID_MASK _ULL(0x03FFF00000000000) #define HGATP64_PPN _ULL(0x00000FFFFFFFFFFF) @@ -170,6 +172,7 @@ #define HGATP_VMID_SHIFT HGATP64_VMID_SHIFT #define HGATP_VMID_MASK HGATP64_VMID_MASK #define HGATP_MODE_SHIFT HGATP64_MODE_SHIFT +#define HGATP_MODE_MASK HGATP64_MODE_MASK #else #define MSTATUS_SD MSTATUS32_SD #define SSTATUS_SD SSTATUS32_SD @@ -181,8 +184,11 @@ #define HGATP_VMID_SHIFT HGATP32_VMID_SHIFT #define HGATP_VMID_MASK HGATP32_VMID_MASK #define HGATP_MODE_SHIFT HGATP32_MODE_SHIFT +#define HGATP_MODE_MASK HGATP32_MODE_MASK #endif +#define GUEST_ROOT_PAGE_TABLE_SIZE KB(16) + #define TOPI_IID_SHIFT 16 #define TOPI_IID_MASK 0xfff #define TOPI_IPRIO_MASK 0xff diff --git a/xen/arch/riscv/p2m.c b/xen/arch/riscv/p2m.c index 214b4861d2..cac07c51c9 100644 --- a/xen/arch/riscv/p2m.c +++ b/xen/arch/riscv/p2m.c @@ -1,8 +1,86 @@ +#include <xen/domain_page.h> #include <xen/mm.h> #include <xen/rwlock.h> #include <xen/sched.h> #include <asm/paging.h> +#include <asm/p2m.h> +#include <asm/riscv_encoding.h> + +unsigned int __read_mostly p2m_root_order; + +static void clear_and_clean_page(struct page_info *page) +{ + clear_domain_page(page_to_mfn(page)); + + /* + * If the IOMMU doesn't support coherent walks and the p2m tables are + * shared between the CPU and IOMMU, it is necessary to clean the + * d-cache. + */ + clean_dcache_va_range(page, PAGE_SIZE); +} + +static struct page_info *p2m_allocate_root(struct domain *d) +{ + struct page_info *page; + + /* + * As mentioned in the Priviliged Architecture Spec (version 20240411) + * in Section 18.5.1, for the paged virtual-memory schemes (Sv32x4, + * Sv39x4, Sv48x4, and Sv57x4), the root page table is 16 KiB and must + * be aligned to a 16-KiB boundary. + */ + page = alloc_domheap_pages(d, P2M_ROOT_ORDER, MEMF_no_owner); + if ( !page ) + return NULL; + + for ( unsigned int i = 0; i < P2M_ROOT_PAGES; i++ ) + clear_and_clean_page(page + i); + + return page; +} + +unsigned long construct_hgatp(struct p2m_domain *p2m, uint16_t vmid) +{ + unsigned long ppn; + + ppn = PFN_DOWN(page_to_maddr(p2m->root)) & HGATP_PPN; + + /* TODO: add detection of hgatp_mode instead of hard-coding it. */ +#if RV_STAGE1_MODE == SATP_MODE_SV39 + p2m->hgatp_mode = HGATP_MODE_SV39X4; +#elif RV_STAGE1_MODE == SATP_MODE_SV48 + p2m->hgatp_mode = HGATP_MODE_SV48X4; +#else +# error "add HGATP_MODE" +#endif + + return ppn | MASK_INSR(p2m->hgatp_mode, HGATP_MODE_MASK) | + MASK_INSR(vmid, HGATP_VMID_MASK); +} + +static int p2m_alloc_root_table(struct p2m_domain *p2m) +{ + struct domain *d = p2m->domain; + struct page_info *page; + const unsigned int nr_root_pages = P2M_ROOT_PAGES; + + /* + * Return back nr_root_pages to assure the root table memory is also + * accounted against the P2M pool of the domain. + */ + if ( !paging_ret_pages_to_domheap(d, nr_root_pages) ) + return -ENOMEM; + + page = p2m_allocate_root(d); + if ( !page ) + return -ENOMEM; + + p2m->root = page; + + return 0; +} int p2m_init(struct domain *d) { @@ -32,6 +110,8 @@ int p2m_init(struct domain *d) # error "Add init of p2m->clean_pte" #endif + p2m_root_order = get_order_from_bytes(GUEST_ROOT_PAGE_TABLE_SIZE); + return 0; } @@ -42,10 +122,20 @@ int p2m_init(struct domain *d) */ int p2m_set_allocation(struct domain *d, unsigned long pages, bool *preempted) { + struct p2m_domain *p2m = p2m_get_hostp2m(d); int rc; if ( (rc = paging_freelist_init(d, pages, preempted)) ) return rc; + /* + * First, initialize p2m pool. Then allocate the root + * table so that the necessary pages can be returned from the p2m pool, + * since the root table must be allocated using alloc_domheap_pages(...) + * to meet its specific requirements. + */ + if ( !p2m->root ) + p2m_alloc_root_table(p2m); + return 0; } diff --git a/xen/arch/riscv/paging.c b/xen/arch/riscv/paging.c index 8882be5ac9..bbe1186900 100644 --- a/xen/arch/riscv/paging.c +++ b/xen/arch/riscv/paging.c @@ -54,6 +54,36 @@ int paging_freelist_init(struct domain *d, unsigned long pages, return 0; } + +bool paging_ret_pages_to_domheap(struct domain *d, unsigned int nr_pages) +{ + struct page_info *page; + + ASSERT(spin_is_locked(&d->arch.paging.lock)); + + if ( ACCESS_ONCE(d->arch.paging.total_pages) < nr_pages ) + return false; + + for ( unsigned int i = 0; i < nr_pages; i++ ) + { + /* Return memory to domheap. */ + page = page_list_remove_head(&d->arch.paging.freelist); + if( page ) + { + ACCESS_ONCE(d->arch.paging.total_pages)--; + free_domheap_page(page); + } + else + { + printk(XENLOG_ERR + "Failed to free P2M pages, P2M freelist is empty.\n"); + return false; + } + } + + return true; +} + /* Domain paging struct initialization. */ int paging_domain_init(struct domain *d) { -- 2.50.1