RE: [PATCH] xen/arm: Add Cortex-A73 erratum 858921 workaround
> -Original Message- > From: Julien Grall > Sent: Monday, November 9, 2020 8:04 PM > To: Penny Zheng ; xen-devel@lists.xenproject.org; > sstabell...@kernel.org > Cc: Andre Przywara ; Bertrand Marquis > ; Wei Chen ; Kaly Xin > ; nd > Subject: Re: [PATCH] xen/arm: Add Cortex-A73 erratum 858921 workaround > > Hi, > > On 09/11/2020 08:21, Penny Zheng wrote: > > CNTVCT_EL0 or CNTPCT_EL0 counter read in Cortex-A73 (all versions) > > might return a wrong value when the counter crosses a 32bit boundary. > > > > Until now, there is no case for Xen itself to access CNTVCT_EL0, and > > it also should be the Guest OS's responsibility to deal with this > > part. > > > > But for CNTPCT, there exists several cases in Xen involving reading > > CNTPCT, so a possible workaround is that performing the read twice, > > and to return one or the other depending on whether a transition has > > taken place. > > > > Signed-off-by: Penny Zheng > > Acked-by: Julien Grall > Thank you. 😉 > On a related topic, do we need a fix similar to Linux commit 75a19a0202db > "arm64: arch_timer: Ensure counter register reads occur with seqlock held"? > Sure, I'll check this commit and talk with my teams for further work. Cheers -- Penny Zheng > Cheers, > > -- > Julien Grall
RE: [PATCH] design: design doc for shared memory on a dom0less system
Hi Julien Sorry for the late response, Since I'm considering sending the code together for better understanding. > -Original Message- > From: Julien Grall > Sent: Wednesday, January 26, 2022 6:58 PM > To: Penny Zheng ; xen-devel@lists.xenproject.org; > sstabell...@kernel.org > Cc: Bertrand Marquis ; Wei Chen > > Subject: Re: [PATCH] design: design doc for shared memory on a dom0less > system > > Hi, > > On 26/01/2022 10:09, Penny Zheng wrote: > > This commit provides a design doc for static shared memory on a > > dom0less system. > > > > Signed-off-by: Penny Zheng > > --- > > design/shm-dom0less.md | 182 > + > > 1 file changed, 182 insertions(+) > > create mode 100644 design/shm-dom0less.md > > > > diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md new file > > mode 100644 index 000..b46199d > > --- /dev/null > > +++ b/design/shm-dom0less.md > > @@ -0,0 +1,182 @@ > > +# Static Shared Memory between domains on a dom0less system > > + > > +This design aims to provide an overview of the new feature: setting > > +up static shared memory between domains on a dom0less system, through > > +device tree configuration. > > + > > +The new feature is driven by the need of finding a way to build up > > +communication channels on dom0less system, since the legacy ways > > +including grant table, etc are all absent there. > > Stefano has a series to add support for grant-table [2]. So I think you want > to > justify it differently. > The dom0less system I am referring here is that either dom0 is totally missing in the system, or even when dom0 is there, all the other domains are statically configured and CONFIG_XEN is not enabled, so in above scenario, all the legacy ways to communicate between domains are absent. These scenarios are specially applied to ARMv8R, where everything is static. I think Stefano is trying to enable PV drivers on dom0less guest with CONFIG_XEN still on. TBH, I could not find proper nouns for them, sometimes calling these domains dom0less guests but with dom0 still there makes me confused too. ;/ > > + > > +It was inspired by the patch serie of "xl/libxl-based shared memory", > > +see [1] for more details. > > + > > +# Static Shared Memory Device Tree Configuration > > + > > +The static shared memory device tree nodes allow users to statically > > +set up shared memory among a group of dom0less DomUs and Dom0, > > +enabling domains to do shm-based communication. > > + > > +- compatible > > + > > +"xen,domain-shared-memory-v1" > > + > > +- xen,shm-id > > From the document, it is not clear to me what is the purpose of the > identifier. > Could you clarify it? > It is more related to the implementation. I've already pushed an RFC patch for reviewing in community for better understanding. For example, in commit "xen/arm: introduce static shared memory", when parsing shared memory node to reserve it in advance, we notice that a shared memory region could be shared among multiple domains, so in order to prevent iterating over all reserved memory regions each time, we use bitmap there and shm-id is used as the index of the bitmap. Also, in Linux Doc about xen,shared-memory binding[3], it requires a xen,id property that identifies the shared memory region as specified in the VM config file. > > + > > +An u32 value represents the unique identifier of the shared memory > region. > > +User valuing per shared memory region shall follow the ascending order, > > +starting from xen,shm-id = <0x0>, to the maximum identifier > > +xen,shm-id = <0x126>. > > Why is it limit to 0x126? And also, why do they have to be allocated in > ascending order? > In current code, I make it the same number with NR_MEM_BANKS > > The special xen,shm-id = <0x127> is reserved for > > +INVALID_SHMID. > > Why do we need to reserve invalid? > It is removed in current codes. > > + > > +- xen,shared-mem > > + > > +An array takes a physical address, which is the base address of the > > +shared memory region in host physical address space, a size, and a > > guest > > +physical address, as the target address of the mapping. > > I think shared memory is useful without static allocation. So I think we want > to > make the host physical address optional. > Hmm,, so later,, you want to make one domain as owner, and parts of its guest RAM is shared with other borrower domains? So only GFN is enough here, right? > > +
[PATCH v1 00/13] Static shared memory on dom0less system
The static shared memory device tree nodes allow users to statically set up shared memory on dom0less system, enabling domains to do shm-based communication. The new feature is driven by the need of finding a way to build up communication channels on dom0less system, since the legacy ways including grant table, etc are all absent there. It was inspired by the patch serie of ["xl/libxl-based shared mem]( https://marc.info/?l=xen-devel&m=154404821731186ory";). Looking into related [design link]( https://lore.kernel.org/all/a50d9fde-1d06-7cda-2779-9eea9e1c0...@xen.org/T/) for more details. Penny Zheng (13): xen/arm: introduce static shared memory xen/arm: introduce a special domain DOMID_SHARED xen/arm: allocate static shared memory to dom_shared xen/arm: add P2M type parameter in guest_physmap_add_pages xen/arm: introduce get_pages_from_gfn xen/arm: set up shared memory foreign mapping for borrower domain xen/arm: create shared memory nodes in guest device tree xen/arm: destroy static shared memory when de-construct domain xen/arm: enable statically shared memory on Dom0 xen/arm: allocate static shared memory to a specific owner domain xen/arm: store shm-info for deferred foreign memory map xen/arm: defer foreign memory map in shm_init_late xen/arm: unmap foreign memory mapping when destroyed domain is owner domain docs/misc/arm/device-tree/booting.txt | 118 +++ xen/arch/arm/Kconfig | 7 + xen/arch/arm/bootfdt.c| 52 +++ xen/arch/arm/domain.c | 117 ++- xen/arch/arm/domain_build.c | 445 +- xen/arch/arm/include/asm/domain.h | 33 ++ xen/arch/arm/include/asm/p2m.h| 42 ++- xen/arch/arm/include/asm/setup.h | 3 + xen/arch/arm/setup.c | 28 ++ xen/common/domain.c | 11 +- xen/common/page_alloc.c | 5 + xen/common/vsprintf.c | 9 +- xen/include/public/xen.h | 6 + xen/include/xen/sched.h | 2 + 14 files changed, 864 insertions(+), 14 deletions(-) -- 2.25.1
[PATCH v1 02/13] xen/arm: introduce a special domain DOMID_SHARED
From: Penny Zheng
In case to own statically shared pages when owner domain is not
explicitly defined, this commits propose a special domain DOMID_SHARED,
and we assign it 0x7FF5, as one of the system domains.
Statically shared memory reuses the same way of initialization with static
memory, hence this commits proposes a new Kconfig CONFIG_STATIC_SHM to wrap
related codes, and this option depends on static memory(CONFIG_STATIC_MEMORY).
We intends to do shared domain creation after setup_virt_paging so shared
domain could successfully do p2m initialization.
Signed-off-by: Penny Zheng
---
xen/arch/arm/Kconfig | 7 +++
xen/arch/arm/domain.c | 12 ++--
xen/arch/arm/include/asm/domain.h | 6 ++
xen/arch/arm/setup.c | 22 ++
xen/common/domain.c | 11 +++
xen/common/page_alloc.c | 5 +
xen/common/vsprintf.c | 9 +
xen/include/public/xen.h | 6 ++
xen/include/xen/sched.h | 2 ++
9 files changed, 70 insertions(+), 10 deletions(-)
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index ecfa6822e4..c54accefb1 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -106,6 +106,13 @@ config TEE
source "arch/arm/tee/Kconfig"
+config STATIC_SHM
+ bool "Statically shared memory on a dom0less system" if UNSUPPORTED
+ depends on STATIC_MEMORY
+ default n
+ help
+ This option enables statically shared memory on a dom0less system.
+
endmenu
menu "ARM errata workaround via the alternative framework"
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 8110c1df86..1ff1df5d3f 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -44,6 +44,10 @@
DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
+#ifdef CONFIG_STATIC_SHM
+struct domain *__read_mostly dom_shared;
+#endif
+
static void do_idle(void)
{
unsigned int cpu = smp_processor_id();
@@ -703,7 +707,7 @@ int arch_domain_create(struct domain *d,
if ( is_idle_domain(d) )
return 0;
-ASSERT(config != NULL);
+ASSERT(is_shared_domain(d) ? config == NULL : config != NULL);
#ifdef CONFIG_IOREQ_SERVER
ioreq_domain_init(d);
@@ -712,12 +716,16 @@ int arch_domain_create(struct domain *d,
d->arch.directmap = flags & CDF_directmap;
/* p2m_init relies on some value initialized by the IOMMU subsystem */
-if ( (rc = iommu_domain_init(d, config->iommu_opts)) != 0 )
+if ( (rc = iommu_domain_init(d, is_shared_domain(d) ? 0 :
config->iommu_opts)) != 0 )
goto fail;
if ( (rc = p2m_init(d)) != 0 )
goto fail;
+/* DOMID_shared is sufficiently constructed after p2m initialization. */
+if ( is_shared_domain(d) )
+return 0;
+
rc = -ENOMEM;
if ( (d->shared_info = alloc_xenheap_pages(0, 0)) == NULL )
goto fail;
diff --git a/xen/arch/arm/include/asm/domain.h
b/xen/arch/arm/include/asm/domain.h
index c56f6e4398..ea7a7219a3 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -31,6 +31,12 @@ enum domain_type {
#define is_domain_direct_mapped(d) (d)->arch.directmap
+#ifdef CONFIG_STATIC_SHM
+extern struct domain *dom_shared;
+#else
+#define dom_shared NULL
+#endif
+
/*
* Is the domain using the host memory layout?
*
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index d5d0792ed4..f6a3b04958 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -855,6 +855,20 @@ static bool __init is_dom0less_mode(void)
return ( !dom0found && domUfound );
}
+#ifdef CONFIG_STATIC_SHM
+static void __init setup_shared_domain(void)
+{
+/*
+ * Initialise our DOMID_SHARED domain.
+ * This domain owns statically shared pages when owner domain is not
+ * explicitly defined.
+ */
+dom_shared = domain_create(DOMID_SHARED, NULL, CDF_directmap);
+if ( IS_ERR(dom_shared) )
+panic("Failed to create d[SHARED]: %ld\n", PTR_ERR(dom_shared));
+}
+#endif
+
size_t __read_mostly dcache_line_bytes;
/* C entry point for boot CPU */
@@ -1022,6 +1036,14 @@ void __init start_xen(unsigned long boot_phys_offset,
apply_alternatives_all();
enable_errata_workarounds();
+#ifdef CONFIG_STATIC_SHM
+/*
+ * This needs to be called **after** setup_virt_paging so shared
+ * domains could successfully do p2m initialization.
+ */
+setup_shared_domain();
+#endif
+
/* Create initial domain 0. */
if ( !is_dom0less_mode() )
create_dom0();
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 3742322d22..5cdd0b9f5b 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -643,11 +643,14 @@ struct domain *domain_create(domid_t domid,
rangeset_domain_initialise(d);
-/* DOMID_{XEN,IO,etc} (other than IDLE) are sufficiently constructed. */
-if ( is_system_
[PATCH v1 04/13] xen/arm: add P2M type parameter in guest_physmap_add_pages
From: Penny Zheng
In order to cover the scenario where users intend to set up guest
p2m foreign mapping with nr_pages, this commit adds a new P2M type
parameter in guest_physmap_add_pages.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c| 5 +++--
xen/arch/arm/include/asm/p2m.h | 5 +++--
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 6e6349caac..984e70e5fc 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -507,7 +507,7 @@ static bool __init append_static_memory_to_bank(struct
domain *d,
else
sgfn = gaddr_to_gfn(mfn_to_maddr(smfn));
-res = guest_physmap_add_pages(d, sgfn, smfn, nr_pages);
+res = guest_physmap_add_pages(d, sgfn, smfn, nr_pages, p2m_ram_rw);
if ( res )
{
dprintk(XENLOG_ERR, "Failed to map pages to DOMU: %d", res);
@@ -787,7 +787,8 @@ static int __init allocate_shared_memory(struct domain *d,
if ( mfn_eq(smfn, INVALID_MFN) )
return -EINVAL;
-ret = guest_physmap_add_pages(d, gaddr_to_gfn(gbase), smfn,
PFN_DOWN(psize));
+ret = guest_physmap_add_pages(d, gaddr_to_gfn(gbase), smfn,
PFN_DOWN(psize),
+ p2m_ram_rw);
if ( ret )
{
dprintk(XENLOG_ERR, "Failed to map shared memory to %pd.\n", d);
diff --git a/xen/arch/arm/include/asm/p2m.h b/xen/arch/arm/include/asm/p2m.h
index 8cce459b67..58590145b0 100644
--- a/xen/arch/arm/include/asm/p2m.h
+++ b/xen/arch/arm/include/asm/p2m.h
@@ -317,9 +317,10 @@ guest_physmap_add_page(struct domain *d, gfn_t gfn, mfn_t
mfn,
static inline int guest_physmap_add_pages(struct domain *d,
gfn_t gfn,
mfn_t mfn,
- unsigned int nr_pages)
+ unsigned int nr_pages,
+ p2m_type_t t)
{
-return p2m_insert_mapping(d, gfn, nr_pages, mfn, p2m_ram_rw);
+return p2m_insert_mapping(d, gfn, nr_pages, mfn, t);
}
mfn_t gfn_to_mfn(struct domain *d, gfn_t gfn);
--
2.25.1
[PATCH v1 05/13] xen/arm: introduce get_pages_from_gfn
From: Penny Zheng
In order to getting statically shared pages based on gfn with nr_pages, this
commit introduces a new helper get_pages_from_gfn to acquire a set of pages
based on [gfn, gfn + nr_gfns), with the same P2M type.
Signed-off-by: Penny Zheng
---
xen/arch/arm/include/asm/p2m.h | 37 ++
1 file changed, 37 insertions(+)
diff --git a/xen/arch/arm/include/asm/p2m.h b/xen/arch/arm/include/asm/p2m.h
index 58590145b0..b23024b9a1 100644
--- a/xen/arch/arm/include/asm/p2m.h
+++ b/xen/arch/arm/include/asm/p2m.h
@@ -370,6 +370,43 @@ static inline struct page_info *get_page_from_gfn(
return page;
}
+static inline int get_pages_from_gfn(struct domain *d, unsigned long gfn,
+ unsigned long nr_gfns,
+ struct page_info **pages, p2m_type_t *t,
+ p2m_query_t q)
+{
+p2m_type_t _t;
+unsigned long i = 0;
+int ret = 0;
+
+for ( ; i < nr_gfns; i++ )
+{
+pages[i] = get_page_from_gfn(d, gfn + i, t, q);
+
+if ( !pages[i] )
+{
+ret = -ENOENT;
+goto fail_get;
+}
+
+if ( i == 0 )
+_t = *t;
+/* Check if all pages share the same p2m type. */
+else if ( *t != _t )
+{
+ret = -EINVAL;
+goto fail_get;
+}
+}
+
+return ret;
+
+ fail_get:
+while( --i >= 0 )
+put_page(pages[i]);
+return ret;
+}
+
int get_page_type(struct page_info *page, unsigned long type);
bool is_iomem_page(mfn_t mfn);
static inline int get_page_and_type(struct page_info *page,
--
2.25.1
[PATCH v1 06/13] xen/arm: set up shared memory foreign mapping for borrower domain
From: Penny Zheng
This commits introduces a new helper guest_physmap_add_shm to set up shared
memory foreign mapping for borrower domain.
Firstly it should get and take reference of statically shared pages from
owner dom_shared. Then it will setup P2M foreign memory map of these statically
shared pages for borrower domain.
This commits only considers owner domain is the default dom_shared, the
other scenario will be covered in the following patches.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c | 52 +
1 file changed, 52 insertions(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 984e70e5fc..8cee5ffbd1 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -798,6 +798,48 @@ static int __init allocate_shared_memory(struct domain *d,
return ret;
}
+static int __init guest_physmap_add_shm(struct domain *od, struct domain *bd,
+unsigned long o_gfn,
+unsigned long b_gfn,
+unsigned long nr_gfns)
+{
+struct page_info **pages = NULL;
+p2m_type_t p2mt, t;
+int ret = 0;
+
+pages = xmalloc_array(struct page_info *, nr_gfns);
+if ( !pages )
+return -ENOMEM;
+
+/*
+ * Take reference of statically shared pages from owner domain.
+ * Reference will be released when destroying shared memory region.
+ */
+ret = get_pages_from_gfn(od, o_gfn, nr_gfns, pages, &p2mt, P2M_ALLOC);
+if ( ret )
+{
+ret = -EINVAL;
+goto fail_pages;
+}
+
+if ( p2m_is_ram(p2mt) )
+t = (p2mt == p2m_ram_rw) ? p2m_map_foreign_rw : p2m_map_foreign_ro;
+else
+{
+ret = -EINVAL;
+goto fail_pages;
+}
+
+/* Set up guest foreign map. */
+ret = guest_physmap_add_pages(bd, _gfn(b_gfn), page_to_mfn(pages[0]),
+ nr_gfns, t);
+
+ fail_pages:
+xfree(pages);
+
+return ret;
+}
+
static int __init process_shm(struct domain *d,
const struct dt_device_node *node)
{
@@ -855,6 +897,16 @@ static int __init process_shm(struct domain *d,
set_bit(shm_id, shm_mask);
}
+
+/*
+ * All domains are borrower domains when owner domain is the
+ * default dom_shared, so here we could just set up P2M foreign
+ * mapping for borrower domain immediately.
+ */
+ret = guest_physmap_add_shm(dom_shared, d, PFN_DOWN(pbase),
+PFN_DOWN(gbase), PFN_DOWN(psize));
+if ( ret )
+return ret;
}
return 0;
--
2.25.1
[PATCH v1 03/13] xen/arm: allocate static shared memory to dom_shared
From: Penny Zheng
This commit introduces process_shm to cope with static shared memory in
domain construction.
This commit only considers allocating static shared memory to dom_shared
when owner domain is not explicitly defined in device tree, the other
scenario will be covered in the following patches.
Static shared memory could reuse acquire_static_memory_bank() to acquire
and allocate static memory.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c | 116 +++-
1 file changed, 115 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 8be01678de..6e6349caac 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -527,7 +527,8 @@ static mfn_t __init acquire_static_memory_bank(struct
domain *d,
mfn_t smfn;
int res;
-device_tree_get_reg(cell, addr_cells, size_cells, pbase, psize);
+if ( cell )
+device_tree_get_reg(cell, addr_cells, size_cells, pbase, psize);
ASSERT(IS_ALIGNED(*pbase, PAGE_SIZE) && IS_ALIGNED(*psize, PAGE_SIZE));
if ( PFN_DOWN(*psize) > UINT_MAX )
{
@@ -751,6 +752,113 @@ static void __init assign_static_memory_11(struct domain
*d,
panic("Failed to assign requested static memory for direct-map domain
%pd.",
d);
}
+
+#ifdef CONFIG_STATIC_SHM
+static __initdata DECLARE_BITMAP(shm_mask, NR_MEM_BANKS);
+
+static mfn_t __init acquire_shared_memory_bank(struct domain *d,
+ u32 addr_cells, u32 size_cells,
+ paddr_t *pbase, paddr_t *psize)
+{
+/*
+ * Pages of statically shared memory shall be included
+ * in domain_tot_pages().
+ */
+d->max_pages += PFN_DOWN(*psize);
+
+return acquire_static_memory_bank(d, NULL, addr_cells, size_cells,
+ pbase, psize);
+
+}
+
+static int __init allocate_shared_memory(struct domain *d,
+ u32 addr_cells, u32 size_cells,
+ paddr_t pbase, paddr_t psize,
+ paddr_t gbase)
+{
+mfn_t smfn;
+int ret = 0;
+
+printk(XENLOG_INFO "Allocate static shared memory BANK
%#"PRIpaddr"-%#"PRIpaddr"\n",
+ pbase, pbase + psize);
+
+smfn = acquire_shared_memory_bank(d, addr_cells, size_cells, &pbase,
+ &psize);
+if ( mfn_eq(smfn, INVALID_MFN) )
+return -EINVAL;
+
+ret = guest_physmap_add_pages(d, gaddr_to_gfn(gbase), smfn,
PFN_DOWN(psize));
+if ( ret )
+{
+dprintk(XENLOG_ERR, "Failed to map shared memory to %pd.\n", d);
+return ret;
+}
+
+return ret;
+}
+
+static int __init process_shm(struct domain *d,
+ const struct dt_device_node *node)
+{
+struct dt_device_node *shm_node;
+int ret = 0;
+const struct dt_property *prop;
+const __be32 *cells;
+u32 shm_id;
+u32 addr_cells, size_cells;
+paddr_t gbase, pbase, psize;
+
+dt_for_each_child_node(node, shm_node)
+{
+if ( !dt_device_is_compatible(shm_node, "xen,domain-shared-memory-v1")
)
+continue;
+
+if ( !dt_property_read_u32(shm_node, "xen,shm-id", &shm_id) )
+{
+printk("Shared memory node does not provide \"xen,shm-id\"
property.\n");
+return -ENOENT;
+}
+
+addr_cells = dt_n_addr_cells(shm_node);
+size_cells = dt_n_size_cells(shm_node);
+prop = dt_find_property(shm_node, "xen,shared-mem", NULL);
+if ( !prop )
+{
+printk("Shared memory node does not provide \"xen,shared-mem\"
property.\n");
+return -ENOENT;
+}
+cells = (const __be32 *)prop->value;
+/* xen,shared-mem = ; */
+device_tree_get_reg(&cells, addr_cells, size_cells, &pbase, &psize);
+ASSERT(IS_ALIGNED(pbase, PAGE_SIZE) && IS_ALIGNED(psize, PAGE_SIZE));
+gbase = dt_read_number(cells, addr_cells);
+
+/* TODO: Consider owner domain is not the default dom_shared. */
+/*
+ * Per shared memory region could be shared between multiple domains.
+ * In case re-allocating the same shared memory region, we use bitmask
+ * shm_mask to record whether this shared memory region has ever been
+ * allocated already.
+ */
+if ( !test_bit(shm_id, shm_mask) )
+{
+/*
+ * Allocate statically shared pages to the default dom_shared.
+ * Set up P2M, and dom_shared is a direct-map domain,
+ * so GFN == PFN.
+ */
+ret = allocate_shared_memory(dom_shared, addr_cells, size_cells,
+
[PATCH v1 08/13] xen/arm: destroy static shared memory when de-construct domain
From: Penny Zheng
This commit introduces a new helper destroy_domain_shm to destroy static
shared memory at domain de-construction.
This patch only considers the scenario where the owner domain is the
default dom_shared, for user-defined owner domain, it will be covered in
the following patches.
Since all domains are borrower domains, we could simply remove guest P2M
foreign mapping of statically shared memory region and drop the reference
added at guest_physmap_add_shm.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain.c | 48 +++
1 file changed, 48 insertions(+)
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 1ff1df5d3f..f0bfd67fe5 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -34,6 +34,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -993,6 +994,48 @@ static int relinquish_memory(struct domain *d, struct
page_list_head *list)
return ret;
}
+#ifdef CONFIG_STATIC_SHM
+static int domain_destroy_shm(struct domain *d)
+{
+int ret = 0;
+unsigned long i = 0UL, j;
+
+if ( d->arch.shm_mem == NULL )
+return ret;
+else
+{
+for ( ; i < d->arch.shm_mem->nr_banks; i++ )
+{
+unsigned long nr_gfns = PFN_DOWN(d->arch.shm_mem->bank[i].size);
+gfn_t gfn = gaddr_to_gfn(d->arch.shm_mem->bank[i].start);
+
+for ( j = 0; j < nr_gfns; j++ )
+{
+mfn_t mfn;
+
+mfn = gfn_to_mfn(d, gfn_add(gfn, j));
+if ( !mfn_valid(mfn) )
+{
+dprintk(XENLOG_ERR,
+"Domain %pd page number %lx invalid.\n",
+d, gfn_x(gfn) + i);
+return -EINVAL;
+}
+
+ret = guest_physmap_remove_page(d, gfn_add(gfn, j), mfn, 0);
+if ( ret )
+return ret;
+
+/* Drop the reference. */
+put_page(mfn_to_page(mfn));
+}
+}
+}
+
+return ret;
+}
+#endif
+
/*
* Record the current progress. Subsequent hypercall continuations will
* logically restart work from this point.
@@ -1039,6 +1082,11 @@ int domain_relinquish_resources(struct domain *d)
*/
domain_vpl011_deinit(d);
+#ifdef CONFIG_STATIC_SHM
+ret = domain_destroy_shm(d);
+if ( ret )
+return ret;
+#endif
#ifdef CONFIG_IOREQ_SERVER
ioreq_server_destroy_all(d);
#endif
--
2.25.1
[PATCH v1 07/13] xen/arm: create shared memory nodes in guest device tree
From: Penny Zheng
We expose the shared memory to the domU using the "xen,shared-memory-v1"
reserved-memory binding. See
Documentation/devicetree/bindings/reserved-memory/xen,shared-memory.txt
in Linux for the corresponding device tree binding.
To save the cost of re-parsing shared memory device tree configuration when
creating shared memory nodes in guest device tree, this commit adds new field
"shm_mem" to store shm-info per domain.
For each shared memory region, a range is exposed under
the /reserved-memory node as a child node. Each range sub-node is
named xen-shmem@ and has the following properties:
- compatible:
compatible = "xen,shared-memory-v1"
- reg:
the base guest physical address and size of the shared memory region
- xen,id:
a string that identifies the shared memory region.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c | 144 ++
xen/arch/arm/include/asm/domain.h | 1 +
xen/arch/arm/include/asm/setup.h | 3 +
3 files changed, 148 insertions(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 8cee5ffbd1..997df46ddd 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -840,6 +840,28 @@ static int __init guest_physmap_add_shm(struct domain *od,
struct domain *bd,
return ret;
}
+static int __init append_shm_bank_to_domain(struct domain *d,
+paddr_t start, paddr_t size,
+u32 shm_id)
+{
+/* Allocate memory at first insertion. */
+if ( d->arch.shm_mem == NULL )
+{
+d->arch.shm_mem = xmalloc_bytes(sizeof(struct meminfo));
+if ( d->arch.shm_mem == NULL )
+return -ENOMEM;
+
+memset(d->arch.shm_mem, 0, sizeof(struct meminfo));
+}
+
+d->arch.shm_mem->bank[d->arch.shm_mem->nr_banks].start = start;
+d->arch.shm_mem->bank[d->arch.shm_mem->nr_banks].size = size;
+d->arch.shm_mem->bank[d->arch.shm_mem->nr_banks].shm_id = shm_id;
+d->arch.shm_mem->nr_banks++;
+
+return 0;
+}
+
static int __init process_shm(struct domain *d,
const struct dt_device_node *node)
{
@@ -907,6 +929,14 @@ static int __init process_shm(struct domain *d,
PFN_DOWN(gbase), PFN_DOWN(psize));
if ( ret )
return ret;
+
+/*
+ * Record static shared memory region info for later setting
+ * up shm-node in guest device tree.
+ */
+ret = append_shm_bank_to_domain(d, gbase, psize, shm_id);
+if ( ret )
+return ret;
}
return 0;
@@ -1237,6 +1267,115 @@ static int __init make_memory_node(const struct domain
*d,
return res;
}
+#ifdef CONFIG_STATIC_SHM
+static int __init make_shm_memory_node(const struct domain *d,
+ void *fdt,
+ int addrcells, int sizecells,
+ struct meminfo *mem)
+{
+unsigned long i = 0;
+int res = 0;
+int reg_size = addrcells + sizecells;
+
+if ( mem->nr_banks == 0 )
+return -ENOENT;
+
+/*
+ * For each shared memory region, a range is exposed under
+ * the /reserved-memory node as a child node. Each range sub-node is
+ * named xen-shmem@.
+ */
+dt_dprintk("Create xen-shmem node\n");
+
+for ( ; i < mem->nr_banks; i++ )
+{
+u64 start = mem->bank[i].start;
+u64 size = mem->bank[i].size;
+u32 shm_id = mem->bank[i].shm_id;
+/* Placeholder for xen-shmem@ + a 64-bit number + \0 */
+char buf[27];
+const char compat[] = "xen,shared-memory-v1";
+__be32 *reg, *cells;
+ unsigned int len;
+
+snprintf(buf, sizeof(buf), "xen-shmem@%"PRIx64, mem->bank[i].start);
+res = fdt_begin_node(fdt, buf);
+if ( res )
+return res;
+
+res = fdt_property(fdt, "compatible", compat, sizeof(compat));
+if ( res )
+return res;
+
+ len = reg_size * sizeof(__be32);
+reg = xmalloc_bytes(len);
+if ( reg == NULL )
+return -ENOMEM;
+cells = reg;
+
+dt_child_set_range(&cells, addrcells, sizecells, start, size);
+
+res = fdt_property(fdt, "reg", reg, len);
+xfree(reg);
+if (res)
+return res;
+
+dt_dprintk("Shared memory bank %lu: %#"PRIx64"->%#"PRIx64"\n",
+ i, start, start + size);
+
+res = fdt_property_cell(fdt, "xen,id", shm_id);
+if (res)
+return res;
+
+res = fdt_end_node(fdt);
+if (res)
+return res;
+}
+
+ret
[PATCH v1 01/13] xen/arm: introduce static shared memory
From: Penny Zheng
This patch serie introduces a new feature: setting up static
shared memory on a dom0less system, through device tree configuration.
This commit parses shared memory node at boot-time, and reserve it in
bootinfo.reserved_mem to avoid other use.
Signed-off-by: Penny Zheng
---
docs/misc/arm/device-tree/booting.txt | 118 ++
xen/arch/arm/bootfdt.c| 52
2 files changed, 170 insertions(+)
diff --git a/docs/misc/arm/device-tree/booting.txt
b/docs/misc/arm/device-tree/booting.txt
index a94125394e..f702ade817 100644
--- a/docs/misc/arm/device-tree/booting.txt
+++ b/docs/misc/arm/device-tree/booting.txt
@@ -355,3 +355,121 @@ device-tree:
This will reserve a 512MB region starting at the host physical address
0x3000 to be exclusively used by DomU1.
+
+Static Shared Memory
+=
+
+The static shared memory device tree nodes allow users to statically set up
+shared memory on dom0less system, enabling domains to do shm-based
+communication.
+
+- compatible
+
+"xen,domain-shared-memory-v1"
+
+- xen,shm-id
+
+An u8 value represents the unique identifier of the shared memory region.
+The maximum identifier shall be "xen,shm-id = <0xff>".
+
+- xen,shared-mem
+
+An array takes a physical address, which is the base address of the
+shared memory region in host physical address space, a size, and a guest
+physical address, as the target address of the mapping.
+
+- role (Optional)
+
+A string property specifying the ownership of a shared memory region,
+the value must be one of the following: "owner", or "borrower"
+A shared memory region could be explicitly backed by one domain, which is
+called "owner domain", and all the other domains who are also sharing
+this region are called "borrower domain".
+If not specified, the default value is "borrower" and owner is
+"dom_shared", a system domain.
+
+As an example:
+
+chosen {
+#address-cells = <0x1>;
+#size-cells = <0x1>;
+xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";
+
+..
+
+/* this is for Dom0 */
+dom0-shared-mem@1000 {
+compatible = "xen,domain-shared-memory-v1";
+role = "owner";
+xen,shm-id = <0x0>;
+xen,shared-mem = <0x1000 0x1000 0x1000>;
+}
+
+domU1 {
+compatible = "xen,domain";
+#address-cells = <0x1>;
+#size-cells = <0x1>;
+memory = <0 131072>;
+cpus = <2>;
+vpl011;
+
+/*
+ * shared memory region identified as 0x0(xen,shm-id = <0x0>)
+ * is shared between Dom0 and DomU1.
+ */
+domU1-shared-mem@1000 {
+compatible = "xen,domain-shared-memory-v1";
+role = "borrower";
+xen,shm-id = <0x0>;
+xen,shared-mem = <0x1000 0x1000 0x5000>;
+}
+
+/*
+ * shared memory region identified as 0x1(xen,shm-id = <0x1>)
+ * is shared between DomU1 and DomU2.
+ */
+domU1-shared-mem@5000 {
+compatible = "xen,domain-shared-memory-v1";
+xen,shm-id = <0x1>;
+xen,shared-mem = <0x5000 0x2000 0x6000>;
+}
+
+..
+
+};
+
+domU2 {
+compatible = "xen,domain";
+#address-cells = <0x1>;
+#size-cells = <0x1>;
+memory = <0 65536>;
+cpus = <1>;
+
+/*
+ * shared memory region identified as 0x1(xen,shm-id = <0x1>)
+ * is shared between domU1 and domU2.
+ */
+domU2-shared-mem@5000 {
+compatible = "xen,domain-shared-memory-v1";
+xen,shm-id = <0x1>;
+xen,shared-mem = <0x5000 0x2000 0x7000>;
+}
+
+..
+};
+};
+
+This is an example with two static shared memory regions.
+
+For the static shared memory region identified as 0x0, host physical
+address starting at 0x1000 of 256MB will be reserved to be shared between
+Dom0 and DomU1.It will get mapped at 0x1000 in Dom0 guest physical address
+space, and at 0x5000 in DomU1 guest physical address space. Dom0 is
+explicitly defined as the owner domain, and DomU1 is the borrower domain.
+
+For the static shared memory region identified as 0x1, host physical
+address starting at 0x5000 of 512MB will be reserved to be shared between
+DomU1 and DomU2. It will get mapped at 0x6000 in DomU1 guest physical
+address space, and at 0x7000 in DomU2 guest physical address space. DomU1
+and DomU2 are both the borrower domain, the owner domain is the default owner
+domain dom_shared.
diff --git
[PATCH v1 09/13] xen/arm: enable statically shared memory on Dom0
From: Penny Zheng
To add statically shared memory nodes in Dom0, user shall put according
static shared memory configuration under /chosen node.
This commit adds shm-processing function process_shm in construct_dom0
to enable statically shared memory on Dom0.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c | 14 ++
1 file changed, 14 insertions(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 997df46ddd..d35f98ff9c 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2561,6 +2561,11 @@ static int __init handle_node(struct domain *d, struct
kernel_info *kinfo,
if ( res )
return res;
}
+
+res = make_resv_memory_node(d, kinfo->fdt, addrcells, sizecells,
+d->arch.shm_mem);
+if ( res )
+return res;
}
res = fdt_end_node(kinfo->fdt);
@@ -3572,6 +3577,9 @@ static int __init construct_dom0(struct domain *d)
{
struct kernel_info kinfo = {};
int rc;
+#ifdef CONFIG_STATIC_SHM
+const struct dt_device_node *chosen = dt_find_node_by_path("/chosen");
+#endif
/* Sanity! */
BUG_ON(d->domain_id != 0);
@@ -3606,6 +3614,12 @@ static int __init construct_dom0(struct domain *d)
allocate_memory_11(d, &kinfo);
find_gnttab_region(d, &kinfo);
+#ifdef CONFIG_STATIC_SHM
+rc = process_shm(d, chosen);
+if ( rc < 0 )
+return rc;
+#endif
+
/* Map extra GIC MMIO, irqs and other hw stuffs to dom0. */
rc = gic_map_hwdom_extra_mappings(d);
if ( rc < 0 )
--
2.25.1
[PATCH v1 10/13] xen/arm: allocate static shared memory to a specific owner domain
From: Penny Zheng
If owner property is defined, then owner domain of a static shared memory
region is not the default dom_shared anymore, but a specific domain.
This commit implements allocating static shared memory to a specific domain
when owner property is defined.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c | 63 -
1 file changed, 48 insertions(+), 15 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index d35f98ff9c..7ee4d33e0b 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -872,6 +872,8 @@ static int __init process_shm(struct domain *d,
u32 shm_id;
u32 addr_cells, size_cells;
paddr_t gbase, pbase, psize;
+const char *role_str;
+bool owner_dom_shared = true;
dt_for_each_child_node(node, shm_node)
{
@@ -899,6 +901,13 @@ static int __init process_shm(struct domain *d,
gbase = dt_read_number(cells, addr_cells);
/* TODO: Consider owner domain is not the default dom_shared. */
+/*
+ * "role" property is optional and if it is defined explicitly,
+ * so the owner domain is not the default "dom_shared" domain.
+ */
+if ( dt_property_read_string(shm_node, "role", &role_str) == 0 )
+owner_dom_shared = false;
+
/*
* Per shared memory region could be shared between multiple domains.
* In case re-allocating the same shared memory region, we use bitmask
@@ -907,17 +916,38 @@ static int __init process_shm(struct domain *d,
*/
if ( !test_bit(shm_id, shm_mask) )
{
-/*
- * Allocate statically shared pages to the default dom_shared.
- * Set up P2M, and dom_shared is a direct-map domain,
- * so GFN == PFN.
- */
-ret = allocate_shared_memory(dom_shared, addr_cells, size_cells,
- pbase, psize, pbase);
-if ( ret )
-return ret;
-
-set_bit(shm_id, shm_mask);
+if ( !owner_dom_shared )
+{
+if ( strcmp(role_str, "owner") == 0 )
+{
+/*
+ * Allocate statically shared pages to a specific owner
+ * domain.
+ */
+ret = allocate_shared_memory(d, shm_id, addr_cells,
+ size_cells, pbase, psize,
+ gbase);
+if ( ret )
+return ret;
+
+set_bit(shm_id, shm_mask);
+}
+}
+else
+{
+/*
+ * Allocate statically shared pages to the default dom_shared.
+ * Set up P2M, and dom_shared is a direct-map domain,
+ * so GFN == PFN.
+ */
+ret = allocate_shared_memory(dom_shared, shm_id,
+ addr_cells, size_cells, pbase,
+ psize, pbase);
+if ( ret )
+return ret;
+
+set_bit(shm_id, shm_mask);
+}
}
/*
@@ -925,10 +955,13 @@ static int __init process_shm(struct domain *d,
* default dom_shared, so here we could just set up P2M foreign
* mapping for borrower domain immediately.
*/
-ret = guest_physmap_add_shm(dom_shared, d, PFN_DOWN(pbase),
-PFN_DOWN(gbase), PFN_DOWN(psize));
-if ( ret )
-return ret;
+if ( owner_dom_shared )
+{
+ret = guest_physmap_add_shm(dom_shared, d, PFN_DOWN(pbase),
+PFN_DOWN(gbase), PFN_DOWN(psize));
+if ( ret )
+return ret;
+}
/*
* Record static shared memory region info for later setting
--
2.25.1
[PATCH v1 13/13] xen/arm: unmap foreign memory mapping when destroyed domain is owner domain
From: Penny Zheng
When destroyed domain is an owner domain of a static shared memory
region, then we need to ensure that all according borrower domains
shall not have the access to this static shared memory region too.
This commit covers above scenario through unmapping all borrowers'
according foreign memory mapping when destroyed domain is a owner
domain of a static shared memory region.
NOTE: It will best for users to destroy all borrowers before the owner
domain in case encountering data abort when accidentally accessing
the static shared memory region.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain.c | 88 ++-
1 file changed, 71 insertions(+), 17 deletions(-)
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 73ffbfb918..8f4a8dcbfc 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -998,10 +998,39 @@ static int relinquish_memory(struct domain *d, struct
page_list_head *list)
}
#ifdef CONFIG_STATIC_SHM
+static int destroy_shm(struct domain *d, gfn_t gfn, unsigned long nr_gfns)
+{
+unsigned long i = 0;
+int ret = 0;
+
+for ( ; i < nr_gfns; i++ )
+{
+mfn_t mfn;
+
+mfn = gfn_to_mfn(d, gfn_add(gfn, i));
+if ( !mfn_valid(mfn) )
+{
+dprintk(XENLOG_ERR,
+"Domain %pd page number %lx invalid.\n",
+d, gfn_x(gfn) + i);
+return -EINVAL;
+}
+
+ret = guest_physmap_remove_page(d, gfn_add(gfn, i), mfn, 0);
+if ( ret )
+return ret;
+
+/* Drop the reference. */
+put_page(mfn_to_page(mfn));
+}
+
+return ret;
+}
+
static int domain_destroy_shm(struct domain *d)
{
int ret = 0;
-unsigned long i = 0UL, j;
+unsigned long i = 0UL;
if ( d->arch.shm_mem == NULL )
return ret;
@@ -1009,29 +1038,54 @@ static int domain_destroy_shm(struct domain *d)
{
for ( ; i < d->arch.shm_mem->nr_banks; i++ )
{
+u32 shm_id = d->arch.shm_mem->bank[i].shm_id;
unsigned long nr_gfns = PFN_DOWN(d->arch.shm_mem->bank[i].size);
gfn_t gfn = gaddr_to_gfn(d->arch.shm_mem->bank[i].start);
-for ( j = 0; j < nr_gfns; j++ )
+if ( test_bit(shm_id, shm_list_mask) )
{
-mfn_t mfn;
-
-mfn = gfn_to_mfn(d, gfn_add(gfn, j));
-if ( !mfn_valid(mfn) )
+domid_t od = shm_list[shm_id].owner_dom;
+unsigned long j;
+/*
+ * If it is a owner domain, then after it gets destroyed,
+ * static shared memory region shall be unaccessible to all
+ * borrower domains too.
+ */
+if ( d->domain_id == od )
{
-dprintk(XENLOG_ERR,
-"Domain %pd page number %lx invalid.\n",
-d, gfn_x(gfn) + i);
-return -EINVAL;
+struct domain *bd;
+
+for ( j = 0; j < shm_list[shm_id].nr_borrower; j++ )
+{
+bd =
get_domain_by_id(shm_list[shm_id].borrower_dom[j]);
+/*
+ * borrower domain could be dead already, in such case
+ * no need to do the unmapping.
+ */
+if ( bd != NULL )
+{
+gfn_t b_gfn = gaddr_to_gfn(
+ shm_list[shm_id].borrower_gbase[j]);
+ret = destroy_shm(bd, b_gfn, nr_gfns);
+if ( ret )
+dprintk(XENLOG_ERR,
+"Domain %pd: failed to destroy static
shared memory.\n",
+bd);
+}
+}
+
+continue;
}
-
-ret = guest_physmap_remove_page(d, gfn_add(gfn, j), mfn, 0);
-if ( ret )
-return ret;
-
-/* Drop the reference. */
-put_page(mfn_to_page(mfn));
}
+/*
+ * As borrower domain, remove foreign memory mapping and drop the
+ * reference count.
+ */
+ret = destroy_shm(d, gfn, nr_gfns);
+if ( ret )
+dprintk(XENLOG_ERR,
+"Domain %pd: failed to destroy static shared
memory.\n",
+d);
}
}
--
2.25.1
[PATCH v1 11/13] xen/arm: store shm-info for deferred foreign memory map
From: Penny Zheng
In a few scenarios where owner domain, is defined after borrower domain in
device tree configuration, then statically shared pages haven't been properly
allocated if borrower domain tries to do foreign memory map during
domain construction.
In order to cover such scenario, we defer all borrower domains' foreign
memory map after all domain construction finished, then only need to store
shm-info during domain construction.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain.c | 3 +++
xen/arch/arm/domain_build.c | 34 ++-
xen/arch/arm/include/asm/domain.h | 25 +++
3 files changed, 61 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index f0bfd67fe5..73ffbfb918 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -47,6 +47,9 @@ DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
#ifdef CONFIG_STATIC_SHM
struct domain *__read_mostly dom_shared;
+
+shm_info_t shm_list[NR_MEM_BANKS];
+DECLARE_BITMAP(shm_list_mask, NR_MEM_BANKS);
#endif
static void do_idle(void)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 7ee4d33e0b..4b19160674 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -771,7 +771,7 @@ static mfn_t __init acquire_shared_memory_bank(struct
domain *d,
}
-static int __init allocate_shared_memory(struct domain *d,
+static int __init allocate_shared_memory(struct domain *d, u32 shm_id,
u32 addr_cells, u32 size_cells,
paddr_t pbase, paddr_t psize,
paddr_t gbase)
@@ -795,6 +795,18 @@ static int __init allocate_shared_memory(struct domain *d,
return ret;
}
+/*
+ * If owner domain is not default dom_shared, shm-info of owner domain
+ * shall also be recorded for later deferred foreign memory map.
+ */
+if ( d != dom_shared )
+{
+shm_list[shm_id].owner_dom = d->domain_id;
+shm_list[shm_id].owner_gbase = gbase;
+shm_list[shm_id].size = psize;
+set_bit(shm_id, shm_list_mask);
+}
+
return ret;
}
@@ -962,6 +974,26 @@ static int __init process_shm(struct domain *d,
if ( ret )
return ret;
}
+else
+{
+if ( strcmp(role_str, "borrower") == 0 )
+{
+/*
+ * In a few scenarios where owner domain, is defined after
+ * borrower domain in device tree configuration, statically
+ * shared pages haven't been properly allocated if borrower
+ * domain here tries to do foreign memory map.
+ * In order to cover such scenario, we defer all borrower
+ * domains'foreign memory map after all domain construction
+ * finished, and only store shm-info here for later use.
+ */
+shm_list[shm_id].borrower_dom[shm_list[shm_id].nr_borrower] =
+d->domain_id;
+shm_list[shm_id].borrower_gbase[shm_list[shm_id].nr_borrower] =
+gbase;
+shm_list[shm_id].nr_borrower++;
+}
+}
/*
* Record static shared memory region info for later setting
diff --git a/xen/arch/arm/include/asm/domain.h
b/xen/arch/arm/include/asm/domain.h
index 6df37d2c46..1c0f2e22ca 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -10,6 +10,7 @@
#include
#include
#include
+#include
#include
struct hvm_domain
@@ -33,6 +34,30 @@ enum domain_type {
#ifdef CONFIG_STATIC_SHM
extern struct domain *dom_shared;
+
+/* Maximum number of borrower domains. */
+#define NR_SHM_DOMAIN 32
+/*
+ * shm_list is indexed by unique identifier "xen,shm-id", but it only stores
+ * a subset of static shared memory regions, of which owner domain is not the
+ * default dom_shared.
+ * shm_list_mask bitmask is to record the position of these static shared
+ * memory regions.
+ * Per bit represents a entry in shm_list, and setting it 1 means the
+ * static shared memory region here is owned by a specific domain, then bit 0
+ * means the static shared memory region here is either owned by the default
+ * dom_shared or no static shared memory region here at all.
+ */
+typedef struct {
+domid_t owner_dom;
+paddr_t owner_gbase;
+paddr_t size;
+domid_t borrower_dom[NR_SHM_DOMAIN];
+paddr_t borrower_gbase[NR_SHM_DOMAIN];
+unsigned long nr_borrower;
+} shm_info_t;
+extern shm_info_t shm_list[NR_MEM_BANKS];
+extern unsigned long shm_list_mask[BITS_TO_LONGS(NR_MEM_BANKS)];
#else
#define dom_shared NULL
#endif
--
2.25.1
[PATCH v1 12/13] xen/arm: defer foreign memory map in shm_init_late
From: Penny Zheng
This commit introduces a new helper shm_init_late to implement
deferred foreign memory mapping of static shared memory.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c | 51 +++
xen/arch/arm/include/asm/domain.h | 1 +
xen/arch/arm/setup.c | 6
3 files changed, 58 insertions(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 4b19160674..f6ef5a702f 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -1006,6 +1006,57 @@ static int __init process_shm(struct domain *d,
return 0;
}
+
+int __init shm_init_late(void)
+{
+unsigned long i = 0UL, shm_id;
+int ret = 0;
+struct domain *od, **bd = NULL;
+unsigned long o_gfn, b_gfn, nr_gfns;
+
+for ( shm_id = find_first_bit(shm_list_mask, NR_MEM_BANKS);
+ shm_id < NR_MEM_BANKS;
+ shm_id = find_next_bit(shm_list_mask, NR_MEM_BANKS, shm_id + 1) )
+
+{
+/* Acquire the only owner domain. */
+od = get_domain_by_id(shm_list[shm_id].owner_dom);
+if ( od == NULL )
+return -ESRCH;
+o_gfn = PFN_DOWN(shm_list[shm_id].owner_gbase);
+nr_gfns = PFN_DOWN(shm_list[shm_id].size);
+
+bd = xmalloc_array(struct domain *, shm_list[shm_id].nr_borrower);
+if ( !bd )
+return -ENOMEM;
+/* Set up foreign memory map for all borrower domains. */
+for ( i = 0; i < shm_list[shm_id].nr_borrower; i++ )
+{
+bd[i] = get_domain_by_id(shm_list[shm_id].borrower_dom[i]);
+if ( bd[i] == NULL )
+{
+return -ESRCH;
+goto fail;
+}
+
+b_gfn = PFN_DOWN(shm_list[shm_id].borrower_gbase[i]);
+ret = guest_physmap_add_shm(od, bd[i], o_gfn, b_gfn, nr_gfns);
+if ( ret )
+{
+ret = -EINVAL;
+goto fail;
+}
+}
+
+xfree(bd);
+}
+return ret;
+
+ fail:
+xfree(bd);
+
+return ret;
+}
#endif /* CONFIG_STATIC_SHM */
#else
static void __init allocate_static_memory(struct domain *d,
diff --git a/xen/arch/arm/include/asm/domain.h
b/xen/arch/arm/include/asm/domain.h
index 1c0f2e22ca..c3f2155f5c 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -58,6 +58,7 @@ typedef struct {
} shm_info_t;
extern shm_info_t shm_list[NR_MEM_BANKS];
extern unsigned long shm_list_mask[BITS_TO_LONGS(NR_MEM_BANKS)];
+extern int shm_init_late(void);
#else
#define dom_shared NULL
#endif
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index f6a3b04958..4987b7 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -1051,7 +1051,13 @@ void __init start_xen(unsigned long boot_phys_offset,
printk(XENLOG_INFO "Xen dom0less mode detected\n");
if ( acpi_disabled )
+{
create_domUs();
+#ifdef CONFIG_STATIC_SHM
+if ( shm_init_late() )
+panic("Failed to set up deferred foreign memory mapping of static
shared memory.\n");
+#endif
+}
/*
* This needs to be called **before** heap_init_late() so modules
--
2.25.1
RE: [PATCH v1 02/13] xen/arm: introduce a special domain DOMID_SHARED
Hi Stefano
> -Original Message-
> From: Stefano Stabellini
> Sent: Friday, March 18, 2022 9:59 AM
> To: Penny Zheng
> Cc: xen-devel@lists.xenproject.org; nd ; Penny Zheng
> ; Stefano Stabellini
> ; Julien Grall ; Bertrand Marquis
> ; Volodymyr Babchuk
> ; Andrew Cooper
> ; George Dunlap ;
> Jan Beulich ; Wei Liu
> Subject: Re: [PATCH v1 02/13] xen/arm: introduce a special domain
> DOMID_SHARED
>
> On Fri, 11 Mar 2022, Penny Zheng wrote:
> > From: Penny Zheng
> >
> > In case to own statically shared pages when owner domain is not
> > explicitly defined, this commits propose a special domain
> > DOMID_SHARED, and we assign it 0x7FF5, as one of the system domains.
> >
> > Statically shared memory reuses the same way of initialization with
> > static memory, hence this commits proposes a new Kconfig
> > CONFIG_STATIC_SHM to wrap related codes, and this option depends on
> static memory(CONFIG_STATIC_MEMORY).
>
> Why does it depend on CONFIG_STATIC_MEMORY? This is a genuine question,
> I am not trying to scope-creep the series. Is there an actual technical
> dependency on CONFIG_STATIC_MEMORY? If not, it would be super useful to
> be able to share memory statically even between normal dom0less guests (of
> course it would be responsibility of the user to provide the right addresses
> and
> avoid mapping clashes.) I know that some of our users have requested this
> feature in the past.
>
I may find a proper way to rephrase here. My poor English writing skill...
When I implemented domain on static allocation, statically configured guest RAM
is
treated as static memory in Xen and I introduced a few helpers to
initialize/allocate/free
static memory, like acquire_staticmem_pages, etc, and all these helpers are
guarded with
CONFIG_STATIC_MEMORY.
I want to reuse these helpers on static shared memory, so CONFIG_STATIC_SHM
depends
on CONFIG_STATIC_MEMORY.
So I'm not restricting sharing static memory between domain on static
allocation, current
Implementation is also useful to normal dom0less guests.
>
> > We intends to do shared domain creation after setup_virt_paging so
> > shared domain could successfully do p2m initialization.
> >
> > Signed-off-by: Penny Zheng
> > ---
> > xen/arch/arm/Kconfig | 7 +++
> > xen/arch/arm/domain.c | 12 ++--
> > xen/arch/arm/include/asm/domain.h | 6 ++
> > xen/arch/arm/setup.c | 22 ++
> > xen/common/domain.c | 11 +++
> > xen/common/page_alloc.c | 5 +
> > xen/common/vsprintf.c | 9 +
> > xen/include/public/xen.h | 6 ++
> > xen/include/xen/sched.h | 2 ++
> > 9 files changed, 70 insertions(+), 10 deletions(-)
> >
> > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig index
> > ecfa6822e4..c54accefb1 100644
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -106,6 +106,13 @@ config TEE
> >
> > source "arch/arm/tee/Kconfig"
> >
> > +config STATIC_SHM
> > + bool "Statically shared memory on a dom0less system" if UNSUPPORTED
> > + depends on STATIC_MEMORY
> > + default n
> > + help
> > + This option enables statically shared memory on a dom0less system.
> > +
> > endmenu
> >
> > menu "ARM errata workaround via the alternative framework"
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index
> > 8110c1df86..1ff1df5d3f 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -44,6 +44,10 @@
> >
> > DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
> >
> > +#ifdef CONFIG_STATIC_SHM
> > +struct domain *__read_mostly dom_shared; #endif
>
> This one should probably go to xen/common/domain.c to stay close to the
> other special domains.
>
Ack. Thx
>
> > static void do_idle(void)
> > {
> > unsigned int cpu = smp_processor_id(); @@ -703,7 +707,7 @@ int
> > arch_domain_create(struct domain *d,
> > if ( is_idle_domain(d) )
> > return 0;
> >
> > -ASSERT(config != NULL);
> > +ASSERT(is_shared_domain(d) ? config == NULL : config != NULL);
> >
> > #ifdef CONFIG_IOREQ_SERVER
> > ioreq_domain_init(d);
> > @@ -712,12 +716,16 @@ int arch_domain_create(struct domain *d,
> > d->arch.directmap = flags & CDF_directmap;
> >
> > /* p2m_init relies on some value initialized by the IOMMU subsystem */
> > -if ( (rc = iommu_domain_init(d, conf
RE: [PATCH v1 03/13] xen/arm: allocate static shared memory to dom_shared
Hi Stefano
> -Original Message-
> From: Stefano Stabellini
> Sent: Friday, March 18, 2022 10:00 AM
> To: Penny Zheng
> Cc: xen-devel@lists.xenproject.org; nd ; Stefano Stabellini
> ; Julien Grall ; Bertrand Marquis
> ; Volodymyr Babchuk
>
> Subject: Re: [PATCH v1 03/13] xen/arm: allocate static shared memory to
> dom_shared
>
> On Fri, 11 Mar 2022, Penny Zheng wrote:
> > From: Penny Zheng
> >
> > This commit introduces process_shm to cope with static shared memory
> > in domain construction.
> >
> > This commit only considers allocating static shared memory to
> > dom_shared when owner domain is not explicitly defined in device tree,
> > the other scenario will be covered in the following patches.
> >
> > Static shared memory could reuse acquire_static_memory_bank() to
> > acquire and allocate static memory.
> >
> > Signed-off-by: Penny Zheng
> > ---
> > xen/arch/arm/domain_build.c | 116
> > +++-
> > 1 file changed, 115 insertions(+), 1 deletion(-)
> >
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index 8be01678de..6e6349caac 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -527,7 +527,8 @@ static mfn_t __init
> acquire_static_memory_bank(struct domain *d,
> > mfn_t smfn;
> > int res;
> >
> > -device_tree_get_reg(cell, addr_cells, size_cells, pbase, psize);
> > +if ( cell )
> > +device_tree_get_reg(cell, addr_cells, size_cells, pbase,
> > + psize);
>
> Why this change?
>
This helper is also used for acquiring static memory as guest RAM for
statically configured
domain.
And since we are reusing it for static shared memory, but try to avoid parsing
the property
here, the "xen,static-shm" property getting parsed in different ways in
process_shm.
So this change is needed here.
And I think I need to add in-code comment to explain. ;)
>
> > ASSERT(IS_ALIGNED(*pbase, PAGE_SIZE) && IS_ALIGNED(*psize,
> PAGE_SIZE));
> > if ( PFN_DOWN(*psize) > UINT_MAX )
> > {
> > @@ -751,6 +752,113 @@ static void __init assign_static_memory_11(struct
> domain *d,
> > panic("Failed to assign requested static memory for direct-map
> domain %pd.",
> >d);
> > }
> > +
> > +#ifdef CONFIG_STATIC_SHM
> > +static __initdata DECLARE_BITMAP(shm_mask, NR_MEM_BANKS);
> > +
> > +static mfn_t __init acquire_shared_memory_bank(struct domain *d,
> > + u32 addr_cells, u32
> > size_cells,
> > + paddr_t *pbase,
> > +paddr_t *psize) {
> > +/*
> > + * Pages of statically shared memory shall be included
> > + * in domain_tot_pages().
> > + */
> > +d->max_pages += PFN_DOWN(*psize);
> > +
> > +return acquire_static_memory_bank(d, NULL, addr_cells, size_cells,
> > + pbase, psize);
> > +
> > +}
> > +
> > +static int __init allocate_shared_memory(struct domain *d,
> > + u32 addr_cells, u32 size_cells,
> > + paddr_t pbase, paddr_t psize,
> > + paddr_t gbase) {
> > +mfn_t smfn;
> > +int ret = 0;
> > +
> > +printk(XENLOG_INFO "Allocate static shared memory
> BANK %#"PRIpaddr"-%#"PRIpaddr"\n",
> > + pbase, pbase + psize);
> > +
> > +smfn = acquire_shared_memory_bank(d, addr_cells, size_cells, &pbase,
> > + &psize);
> > +if ( mfn_eq(smfn, INVALID_MFN) )
> > +return -EINVAL;
> > +
> > +ret = guest_physmap_add_pages(d, gaddr_to_gfn(gbase), smfn,
> PFN_DOWN(psize));
> > +if ( ret )
> > +{
> > +dprintk(XENLOG_ERR, "Failed to map shared memory to %pd.\n", d);
> > +return ret;
> > +}
> > +
> > +return ret;
> > +}
> > +
> > +static int __init process_shm(struct domain *d,
> > + const struct dt_device_node *node) {
> > +struct dt_device_node *shm_node;
> > +int ret = 0;
> > +const struct dt_property *prop;
> > +const __be32 *cells;
> > +u32 shm_id;
> > +u32 addr_cells, size_cells;
> > +paddr_t gbase, pbase, psize;
> > +
> > +dt_for_each_chi
RE: [PATCH v1 06/13] xen/arm: set up shared memory foreign mapping for borrower domain
Hi Stefano
Sorry for the late response, got sidetracked an emergency issue. ;/
> -Original Message-
> From: Stefano Stabellini
> Sent: Friday, March 18, 2022 10:00 AM
> To: Penny Zheng
> Cc: xen-devel@lists.xenproject.org; nd ; Stefano Stabellini
> ; Julien Grall ; Bertrand Marquis
> ; Volodymyr Babchuk
>
> Subject: Re: [PATCH v1 06/13] xen/arm: set up shared memory foreign
> mapping for borrower domain
>
> On Fri, 11 Mar 2022, Penny Zheng wrote:
> > From: Penny Zheng
> >
> > This commits introduces a new helper guest_physmap_add_shm to set up
> > shared memory foreign mapping for borrower domain.
> >
> > Firstly it should get and take reference of statically shared pages
> > from owner dom_shared. Then it will setup P2M foreign memory map of
> > these statically shared pages for borrower domain.
> >
> > This commits only considers owner domain is the default dom_shared,
> > the other scenario will be covered in the following patches.
> >
> > Signed-off-by: Penny Zheng
> > ---
> > xen/arch/arm/domain_build.c | 52
> > +
> > 1 file changed, 52 insertions(+)
> >
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index 984e70e5fc..8cee5ffbd1 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -798,6 +798,48 @@ static int __init allocate_shared_memory(struct
> domain *d,
> > return ret;
> > }
> >
> > +static int __init guest_physmap_add_shm(struct domain *od, struct domain
> *bd,
> > +unsigned long o_gfn,
> > +unsigned long b_gfn,
> > +unsigned long nr_gfns)
>
> They should probably be gfn_t type
>
>
Sure, will do.
> > +{
> > +struct page_info **pages = NULL;
> > +p2m_type_t p2mt, t;
> > +int ret = 0;
> > +
> > +pages = xmalloc_array(struct page_info *, nr_gfns);
> > +if ( !pages )
> > +return -ENOMEM;
> > +
> > +/*
> > + * Take reference of statically shared pages from owner domain.
> > + * Reference will be released when destroying shared memory region.
> > + */
> > +ret = get_pages_from_gfn(od, o_gfn, nr_gfns, pages, &p2mt, P2M_ALLOC);
> > +if ( ret )
> > +{
> > +ret = -EINVAL;
> > +goto fail_pages;
> > +}
> > +
> > +if ( p2m_is_ram(p2mt) )
> > +t = (p2mt == p2m_ram_rw) ? p2m_map_foreign_rw :
> p2m_map_foreign_ro;
> > +else
> > +{
> > +ret = -EINVAL;
> > +goto fail_pages;
> > +}
>
> One idea is to initialize p2mt = p2m_ram_rw and pass it to
> get_pages_from_gfn. Then get_pages_from_gfn can return error immediately
> if any of the pages are of different type.
>
> This way there is no need for checking again here.
>
Right now, the memory attribute of static shared memory is RW as default,
What if we add memory attribute setting in device tree configuration, sometimes,
Users want to specify that borrower domain only has RO right, hmm, then the
Initialization for p2mt could be either p2m_ram_rw or p2m_ram_ro?
In such case, we could add another parameter in guest_physmap_add_shm to
show the p2m type...
Hope I understand what you suggested here.
>
> > +/* Set up guest foreign map. */
> > +ret = guest_physmap_add_pages(bd, _gfn(b_gfn), page_to_mfn(pages[0]),
> > + nr_gfns, t);
> > +
> > + fail_pages:
> > +xfree(pages);
> > +
> > +return ret;
> > +}
> > +
> > static int __init process_shm(struct domain *d,
> >const struct dt_device_node *node) {
> > @@ -855,6 +897,16 @@ static int __init process_shm(struct domain *d,
> >
> > set_bit(shm_id, shm_mask);
> > }
> > +
> > +/*
> > + * All domains are borrower domains when owner domain is the
> > + * default dom_shared, so here we could just set up P2M foreign
> > + * mapping for borrower domain immediately.
> > + */
> > +ret = guest_physmap_add_shm(dom_shared, d, PFN_DOWN(pbase),
> > +PFN_DOWN(gbase), PFN_DOWN(psize));
> > +if ( ret )
> > +return ret;
> > }
> >
> > return 0;
> > --
> > 2.25.1
> >
---
Cheers,
Penny Zheng
RE: [PATCH v1 11/13] xen/arm: store shm-info for deferred foreign memory map
Hi Stefano
> -Original Message-
> From: Stefano Stabellini
> Sent: Friday, March 18, 2022 10:01 AM
> To: Penny Zheng
> Cc: xen-devel@lists.xenproject.org; nd ; Stefano Stabellini
> ; Julien Grall ; Bertrand Marquis
> ; Volodymyr Babchuk
>
> Subject: Re: [PATCH v1 11/13] xen/arm: store shm-info for deferred foreign
> memory map
>
> On Fri, 11 Mar 2022, Penny Zheng wrote:
> > From: Penny Zheng
> >
> > In a few scenarios where owner domain, is defined after borrower
> > domain in device tree configuration, then statically shared pages
> > haven't been properly allocated if borrower domain tries to do foreign
> > memory map during domain construction.
> >
> > In order to cover such scenario, we defer all borrower domains'
> > foreign memory map after all domain construction finished, then only
> > need to store shm-info during domain construction.
> >
> > Signed-off-by: Penny Zheng
> > ---
> > xen/arch/arm/domain.c | 3 +++
> > xen/arch/arm/domain_build.c | 34 ++-
> > xen/arch/arm/include/asm/domain.h | 25 +++
> > 3 files changed, 61 insertions(+), 1 deletion(-)
> >
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index
> > f0bfd67fe5..73ffbfb918 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -47,6 +47,9 @@ DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
> >
> > #ifdef CONFIG_STATIC_SHM
> > struct domain *__read_mostly dom_shared;
> > +
> > +shm_info_t shm_list[NR_MEM_BANKS];
>
> Instead of adding shm_list, maybe we can we re-use mem->bank
> (bootinfo.reserved_mem)?
>
> It is already storing the physical address and size (added in patch #1 with
> process_shm_node). We should be able to find the other info from the mfn:
> mfn_to_page, page_get_owner, mfn_to_gfn. At most, we need to mark the
> memory bank as shared and we could do that with another field in struct
> membank.
>
>
> > +DECLARE_BITMAP(shm_list_mask, NR_MEM_BANKS);
>
> This is the third bitmask we introduce :-)
>
> Can we narrow it down to a single bitmask? Maybe we don't need it at all if we
> switch to using bootinfo.mem.bank.
>
>
> > #endif
> >
> > static void do_idle(void)
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index 7ee4d33e0b..4b19160674 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -771,7 +771,7 @@ static mfn_t __init
> > acquire_shared_memory_bank(struct domain *d,
> >
> > }
> >
> > -static int __init allocate_shared_memory(struct domain *d,
> > +static int __init allocate_shared_memory(struct domain *d, u32
> > +shm_id,
>
> No need for it to be u32?
>
>
> > u32 addr_cells, u32 size_cells,
> > paddr_t pbase, paddr_t psize,
> > paddr_t gbase) @@ -795,6
> > +795,18 @@ static int __init allocate_shared_memory(struct domain *d,
> > return ret;
> > }
> >
> > +/*
> > + * If owner domain is not default dom_shared, shm-info of owner domain
> > + * shall also be recorded for later deferred foreign memory map.
> > + */
> > +if ( d != dom_shared )
> > +{
> > +shm_list[shm_id].owner_dom = d->domain_id;
> > +shm_list[shm_id].owner_gbase = gbase;
> > +shm_list[shm_id].size = psize;
> > +set_bit(shm_id, shm_list_mask);
> > +}
> > return ret;
> > }
> >
> > @@ -962,6 +974,26 @@ static int __init process_shm(struct domain *d,
> > if ( ret )
> > return ret;
> > }
> > +else
> > +{
> > +if ( strcmp(role_str, "borrower") == 0 )
> > +{
> > +/*
> > + * In a few scenarios where owner domain, is defined after
> > + * borrower domain in device tree configuration, statically
> > + * shared pages haven't been properly allocated if borrower
> > + * domain here tries to do foreign memory map.
> > + * In order to cover such scenario, we defer all borrower
> > + * domains'foreign memory map after all domain construction
> > + * finished, and only store shm-info here for later use.
> > + */
> > +
[PATCH 0/5] populate/unpopulate memory when domain on static
Today when a domain unpopulates the memory on runtime, they will always hand the memory over to the heap allocator. And it will be a problem if domain on static allocation.Since guest RAM for domain on static allocation is static memory, which is pre-reserved through domain congifuration, it shall never go back to heap. This patch serie intends to fix this issue, by keeping page allocated and storing it in page list when unpopulating memory, and retrieving page from page list when populating memory. Penny Zheng (5): xen/arm: field "flags" to cover all internal CDF_XXX xen/arm: introduce CDF_staticmem xen/arm: unpopulate memory when domain on static allocation xen/arm: retrieve reserved pages on populate_physmap xen/arm: no need to store pages in resv_page_list when domain is directly mapped xen/arch/arm/domain.c | 3 ++- xen/arch/arm/domain_build.c | 5 +++- xen/arch/arm/include/asm/domain.h | 7 +++-- xen/common/domain.c | 4 +++ xen/common/memory.c | 45 ++- xen/include/xen/domain.h | 2 ++ xen/include/xen/sched.h | 6 + 7 files changed, 67 insertions(+), 5 deletions(-) -- 2.25.1
[PATCH v1 2/5] xen/arm: introduce CDF_staticmem
In order to have an easy and quick way to find out whether this domain is
on static allocation, this commit introduces a new flag CDF_staticmem and a
new helper is_domain_on_static_allocation.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c | 5 -
xen/arch/arm/include/asm/domain.h | 2 ++
xen/include/xen/domain.h | 2 ++
3 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 8be01678de..4e62fd0bf1 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3191,9 +3191,12 @@ void __init create_domUs(void)
if ( !dt_device_is_compatible(node, "xen,domain") )
continue;
+if ( dt_find_property(node, "xen,static-mem", NULL) )
+flags |= CDF_staticmem;
+
if ( dt_property_read_bool(node, "direct-map") )
{
-if ( !IS_ENABLED(CONFIG_STATIC_MEMORY) || !dt_find_property(node,
"xen,static-mem", NULL) )
+if ( !IS_ENABLED(CONFIG_STATIC_MEMORY) || !(flags & CDF_staticmem)
)
panic("direct-map is not valid for domain %s without static
allocation.\n",
dt_node_name(node));
diff --git a/xen/arch/arm/include/asm/domain.h
b/xen/arch/arm/include/asm/domain.h
index 95fef29111..4379f20a12 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -31,6 +31,8 @@ enum domain_type {
#define is_domain_direct_mapped(d) (((d)->arch.flags) & CDF_directmap)
+#define is_domain_on_static_allocation(d) (((d)->arch.flags) & CDF_staticmem)
+
/*
* Is the domain using the host memory layout?
*
diff --git a/xen/include/xen/domain.h b/xen/include/xen/domain.h
index 1c3c88a14d..35dc7143a4 100644
--- a/xen/include/xen/domain.h
+++ b/xen/include/xen/domain.h
@@ -34,6 +34,8 @@ void arch_get_domain_info(const struct domain *d,
#ifdef CONFIG_ARM
/* Should domain memory be directly mapped? */
#define CDF_directmap(1U << 1)
+/* Is domain memory on static allocation? */
+#define CDF_staticmem(1U << 2)
#endif
/*
--
2.25.1
[PATCH v1 1/5] xen/arm: field "flags" to cover all internal CDF_XXX
With more and more CDF_xxx internal flags in and to save the space, this
commit introduces a new field "flags" to store CDF_* internal flags
directly.
Another new CDF_xxx will be introduced in the next patch.
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain.c | 3 ++-
xen/arch/arm/include/asm/domain.h | 5 +++--
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 8110c1df86..35c157d499 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -709,7 +709,8 @@ int arch_domain_create(struct domain *d,
ioreq_domain_init(d);
#endif
-d->arch.directmap = flags & CDF_directmap;
+/* Holding CDF_* internal flags. */
+d->arch.flags = flags;
/* p2m_init relies on some value initialized by the IOMMU subsystem */
if ( (rc = iommu_domain_init(d, config->iommu_opts)) != 0 )
diff --git a/xen/arch/arm/include/asm/domain.h
b/xen/arch/arm/include/asm/domain.h
index ed63c2b6f9..95fef29111 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -29,7 +29,7 @@ enum domain_type {
#define is_64bit_domain(d) (0)
#endif
-#define is_domain_direct_mapped(d) (d)->arch.directmap
+#define is_domain_direct_mapped(d) (((d)->arch.flags) & CDF_directmap)
/*
* Is the domain using the host memory layout?
@@ -103,7 +103,8 @@ struct arch_domain
void *tee;
#endif
-bool directmap;
+/* Holding CDF_* constant. Internal flags for domain creation. */
+uint32_t flags;
} __cacheline_aligned;
struct arch_vcpu
--
2.25.1
[PATCH v1 3/5] xen/arm: unpopulate memory when domain on static allocation
Today when a domain unpopulates the memory on runtime, they will always
hand the memory over to the heap allocator. And it will be a problem if domain
on static allocation.
Guest RAM for domain on static allocation is static memory, which is reserved
to only this domain, so it shall never go back to heap.
For above purpose, this commit tries to keep page allocated and store it
in page list d->resv_page_list on guest_remove_page, when domain on
static allocation.
Signed-off-by: Penny Zheng
---
xen/common/domain.c | 4
xen/common/memory.c | 22 +-
xen/include/xen/sched.h | 6 ++
3 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 351029f8b2..e572f27fce 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -602,6 +602,10 @@ struct domain *domain_create(domid_t domid,
INIT_PAGE_LIST_HEAD(&d->page_list);
INIT_PAGE_LIST_HEAD(&d->extra_page_list);
INIT_PAGE_LIST_HEAD(&d->xenpage_list);
+#ifdef CONFIG_STATIC_MEMORY
+INIT_PAGE_LIST_HEAD(&d->resv_page_list);
+#endif
+
spin_lock_init(&d->node_affinity_lock);
d->node_affinity = NODE_MASK_ALL;
diff --git a/xen/common/memory.c b/xen/common/memory.c
index 69b0cd1e50..2afc3c6f10 100644
--- a/xen/common/memory.c
+++ b/xen/common/memory.c
@@ -35,6 +35,10 @@
#include
#endif
+#ifndef is_domain_on_static_allocation
+#define is_domain_on_static_allocation(d) 0
+#endif
+
struct memop_args {
/* INPUT */
struct domain *domain; /* Domain to be affected. */
@@ -405,13 +409,29 @@ int guest_remove_page(struct domain *d, unsigned long
gmfn)
* device must retrieve the same pfn when the hypercall populate_physmap
* is called.
*
+ * When domain on static allocation, they should always get pages from the
+ * reserved static region when the hypercall populate_physmap is called.
+ *
* For this purpose (and to match populate_physmap() behavior), the page
* is kept allocated.
*/
-if ( !rc && !is_domain_direct_mapped(d) )
+if ( !rc && !(is_domain_direct_mapped(d) ||
+ is_domain_on_static_allocation(d)) )
put_page_alloc_ref(page);
put_page(page);
+#ifdef CONFIG_STATIC_MEMORY
+/*
+ * When domain on static allocation, we shall store pages to
resv_page_list,
+ * so the hypercall populate_physmap could retrieve pages from it,
+ * rather than allocating from heap.
+ */
+if ( is_domain_on_static_allocation(d) )
+{
+page_list_add_tail(page, &d->resv_page_list);
+d->resv_pages++;
+}
+#endif
#ifdef CONFIG_X86
out_put_gfn:
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 406d9bc610..d7e047bf36 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -376,6 +376,9 @@ struct domain
struct page_list_head page_list; /* linked list */
struct page_list_head extra_page_list; /* linked list (size extra_pages) */
struct page_list_head xenpage_list; /* linked list (size xenheap_pages) */
+#ifdef CONFIG_STATIC_MEMORY
+struct page_list_head resv_page_list; /* linked list (size resv_pages) */
+#endif
/*
* This field should only be directly accessed by domain_adjust_tot_pages()
@@ -389,6 +392,9 @@ struct domain
unsigned int extra_pages; /* pages not included in
domain_tot_pages() */
atomic_t shr_pages; /* shared pages */
atomic_t paged_pages; /* paged-out pages */
+#ifdef CONFIG_STATIC_MEMORY
+unsigned int resv_pages;/* reserved pages from static region.
*/
+#endif
/* Scheduling. */
void*sched_priv;/* scheduler-specific data */
--
2.25.1
[PATCH v1 5/5] xen/arm: no need to store pages in resv_page_list when domain is directly mapped
When domain on static allocation and is directly mapped, in terms of
GPA == HPA(guest physical address == host physical address), we could use
mfn_to_page() to easily find the page, so there is no need to store pages
in resv_page_list.
Signed-off-by: Penny Zheng
---
xen/common/memory.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/xen/common/memory.c b/xen/common/memory.c
index 2122ceeba7..2865e09a33 100644
--- a/xen/common/memory.c
+++ b/xen/common/memory.c
@@ -445,8 +445,11 @@ int guest_remove_page(struct domain *d, unsigned long gmfn)
* When domain on static allocation, we shall store pages to
resv_page_list,
* so the hypercall populate_physmap could retrieve pages from it,
* rather than allocating from heap.
+ * No need to store pages in resv_page_list when domain on static
+ * allocation and directly mapped, since we could use mfn_to_page() to
+ * easily find the page.
*/
-if ( is_domain_on_static_allocation(d) )
+if ( is_domain_on_static_allocation(d) && !is_domain_direct_mapped(d) )
{
page_list_add_tail(page, &d->resv_page_list);
d->resv_pages++;
--
2.25.1
[PATCH v1 4/5] xen/arm: retrieve reserved pages on populate_physmap
When domain on static allocation populates memory through populate_physmap,
other than allocating from heap, it shall allocate from resv_page_list to
make sure that all guest RAM are still restricted in statically configured
regions.
Signed-off-by: Penny Zheng
---
xen/common/memory.c | 20
1 file changed, 20 insertions(+)
diff --git a/xen/common/memory.c b/xen/common/memory.c
index 2afc3c6f10..2122ceeba7 100644
--- a/xen/common/memory.c
+++ b/xen/common/memory.c
@@ -249,6 +249,26 @@ static void populate_physmap(struct memop_args *a)
mfn = _mfn(gpfn);
}
+#ifdef CONFIG_STATIC_MEMORY
+else if ( is_domain_on_static_allocation(d) )
+{
+for ( j = 0; j < (1U << a->extent_order); j++ )
+{
+page = page_list_remove_head(&d->resv_page_list);
+if ( unlikely(!page) )
+{
+gdprintk(XENLOG_INFO,
+ "Could not allocate guest page number %lx\n",
+ gfn_x(_gfn(gpfn)));
+goto out;
+}
+d->resv_pages--;
+
+if ( j == 0 )
+mfn = page_to_mfn(page);
+}
+}
+#endif
else
{
page = alloc_domheap_pages(d, a->extent_order, a->memflags);
--
2.25.1
RE: [PATCH v1 3/5] xen/arm: unpopulate memory when domain on static allocation
Hi Jan
> -Original Message-
> From: Jan Beulich
> Sent: Wednesday, March 30, 2022 5:53 PM
> To: Penny Zheng
> Cc: Wei Chen ; Henry Wang ;
> Andrew Cooper ; George Dunlap
> ; Julien Grall ; Stefano Stabellini
> ; Wei Liu ; xen-
> de...@lists.xenproject.org
> Subject: Re: [PATCH v1 3/5] xen/arm: unpopulate memory when domain on
> static allocation
>
> On 30.03.2022 11:36, Penny Zheng wrote:
> > --- a/xen/common/memory.c
> > +++ b/xen/common/memory.c
> > @@ -35,6 +35,10 @@
> > #include
> > #endif
> >
> > +#ifndef is_domain_on_static_allocation #define
> > +is_domain_on_static_allocation(d) 0
>
> Nit: "false", not "0".
>
> > @@ -405,13 +409,29 @@ int guest_remove_page(struct domain *d,
> unsigned long gmfn)
> > * device must retrieve the same pfn when the hypercall
> populate_physmap
> > * is called.
> > *
> > + * When domain on static allocation, they should always get pages from
> the
> > + * reserved static region when the hypercall populate_physmap is
> > called.
> > + *
> > * For this purpose (and to match populate_physmap() behavior), the
> > page
> > * is kept allocated.
> > */
> > -if ( !rc && !is_domain_direct_mapped(d) )
> > +if ( !rc && !(is_domain_direct_mapped(d) ||
> > + is_domain_on_static_allocation(d)) )
> > put_page_alloc_ref(page);
> >
> > put_page(page);
> > +#ifdef CONFIG_STATIC_MEMORY
> > +/*
> > + * When domain on static allocation, we shall store pages to
> resv_page_list,
> > + * so the hypercall populate_physmap could retrieve pages from it,
> > + * rather than allocating from heap.
> > + */
> > +if ( is_domain_on_static_allocation(d) )
> > +{
> > +page_list_add_tail(page, &d->resv_page_list);
> > +d->resv_pages++;
> > +}
> > +#endif
>
> I think this is wrong, as a result of not integrating with put_page().
> The page should only go on that list once its last ref was dropped. I don't
> recall
> for sure, but iirc staticmem pages are put on the domain's page list just like
> other pages would be. But then you also corrupt the list when this isn't the
> last
> ref which is put.
Yes, staticmem pages are put on the domain's page list.
Here, I tried to only destroy the P2M mapping, and keep the page still allocated
to this domain.
resv_page_list is just providing an easy way to track down the unpopulated
memory.
'''
But then you also corrupt the list when this isn't the last
ref which is put.
'''
I'm sorry, would you like to be more specific on this comment?
I want these pages to linked in the domain's page list, then it could be
freed properly when domain get destroyed through relinquish_memory.
>
> As a result I also think that you shouldn't need to touch the earlier if().
>
> Jan
RE: [PATCH v1 4/5] xen/arm: retrieve reserved pages on populate_physmap
Hi Jan
> -Original Message-
> From: Jan Beulich
> Sent: Wednesday, March 30, 2022 5:59 PM
> To: Penny Zheng
> Cc: Wei Chen ; Henry Wang ;
> Andrew Cooper ; George Dunlap
> ; Julien Grall ; Stefano Stabellini
> ; Wei Liu ; xen-
> de...@lists.xenproject.org
> Subject: Re: [PATCH v1 4/5] xen/arm: retrieve reserved pages on
> populate_physmap
>
> On 30.03.2022 11:36, Penny Zheng wrote:
> > --- a/xen/common/memory.c
> > +++ b/xen/common/memory.c
> > @@ -249,6 +249,26 @@ static void populate_physmap(struct memop_args
> *a)
> >
> > mfn = _mfn(gpfn);
> > }
> > +#ifdef CONFIG_STATIC_MEMORY
> > +else if ( is_domain_on_static_allocation(d) )
> > +{
> > +for ( j = 0; j < (1U << a->extent_order); j++ )
> > +{
> > +page = page_list_remove_head(&d->resv_page_list);
>
> How do you guarantee the pages are contiguous, as required by a non-zero
> a->extent_order? Did you perhaps mean to forbid non-zero-order requests
> in this configuration?,
True, true, thanks for pointing that out.
I would not intend to add complex algorithm here to find contiguous pages of
requested
order, forbidding non-zero-order with error message shall be added here.
>
> > +if ( unlikely(!page) )
> > +{
> > +gdprintk(XENLOG_INFO,
> > + "Could not allocate guest page number
> > %lx\n",
> > + gfn_x(_gfn(gpfn)));
> > +goto out;
>
> And earlier allocated pages are simply lost / leaked?
>
That was unconsidered, thanks for pointing that out.
Since now we are forbidding non-zero-order requests, earlier allocated pages
shall
not need to be considered.
> Jan
RE: [PATCH v1 3/5] xen/arm: unpopulate memory when domain on static allocation
Hi Jan
> -Original Message-
> From: Jan Beulich
> Sent: Thursday, March 31, 2022 3:06 PM
> To: Penny Zheng
> Cc: Wei Chen ; Henry Wang ;
> Andrew Cooper ; George Dunlap
> ; Julien Grall ; Stefano Stabellini
> ; Wei Liu ; xen-
> de...@lists.xenproject.org
> Subject: Re: [PATCH v1 3/5] xen/arm: unpopulate memory when domain on
> static allocation
>
> On 31.03.2022 08:13, Penny Zheng wrote:
> > Hi Jan
> >
> >> -Original Message-
> >> From: Jan Beulich
> >> Sent: Wednesday, March 30, 2022 5:53 PM
> >> To: Penny Zheng
> >> Cc: Wei Chen ; Henry Wang
> ;
> >> Andrew Cooper ; George Dunlap
> >> ; Julien Grall ; Stefano
> >> Stabellini ; Wei Liu ; xen-
> >> de...@lists.xenproject.org
> >> Subject: Re: [PATCH v1 3/5] xen/arm: unpopulate memory when domain on
> >> static allocation
> >>
> >> On 30.03.2022 11:36, Penny Zheng wrote:
> >>> --- a/xen/common/memory.c
> >>> +++ b/xen/common/memory.c
> >>> @@ -35,6 +35,10 @@
> >>> #include
> >>> #endif
> >>>
> >>> +#ifndef is_domain_on_static_allocation #define
> >>> +is_domain_on_static_allocation(d) 0
> >>
> >> Nit: "false", not "0".
> >>
> >>> @@ -405,13 +409,29 @@ int guest_remove_page(struct domain *d,
> >> unsigned long gmfn)
> >>> * device must retrieve the same pfn when the hypercall
> >> populate_physmap
> >>> * is called.
> >>> *
> >>> + * When domain on static allocation, they should always get
> >>> + pages from
> >> the
> >>> + * reserved static region when the hypercall populate_physmap is
> called.
> >>> + *
> >>> * For this purpose (and to match populate_physmap() behavior), the
> page
> >>> * is kept allocated.
> >>> */
> >>> -if ( !rc && !is_domain_direct_mapped(d) )
> >>> +if ( !rc && !(is_domain_direct_mapped(d) ||
> >>> + is_domain_on_static_allocation(d)) )
> >>> put_page_alloc_ref(page);
> >>>
> >>> put_page(page);
> >>> +#ifdef CONFIG_STATIC_MEMORY
> >>> +/*
> >>> + * When domain on static allocation, we shall store pages to
> >> resv_page_list,
> >>> + * so the hypercall populate_physmap could retrieve pages from it,
> >>> + * rather than allocating from heap.
> >>> + */
> >>> +if ( is_domain_on_static_allocation(d) )
> >>> +{
> >>> +page_list_add_tail(page, &d->resv_page_list);
> >>> +d->resv_pages++;
> >>> +}
> >>> +#endif
> >>
> >> I think this is wrong, as a result of not integrating with put_page().
> >> The page should only go on that list once its last ref was dropped. I
> >> don't recall for sure, but iirc staticmem pages are put on the
> >> domain's page list just like other pages would be. But then you also
> >> corrupt the list when this isn't the last ref which is put.
> >
> > Yes, staticmem pages are put on the domain's page list.
> > Here, I tried to only destroy the P2M mapping, and keep the page still
> > allocated to this domain.
>
> Well, much depends on what you call "allocated". For populate_physmap you
> then take pages from resv_page_list. So I'd like to distinguish "allocated"
> from
> "reserved": The pages are allocated to the domain when they're on the normal
> page list; they're reserved when they're on the new list you introduce. And
> what you want to initiate here is an "allocated" -> "reserved" transition.
>
> > resv_page_list is just providing an easy way to track down the unpopulated
> memory.
> > '''
> > But then you also corrupt the list when this isn't the last ref which
> > is put.
> > '''
> > I'm sorry, would you like to be more specific on this comment?
> > I want these pages to linked in the domain's page list, then it could
> > be freed properly when domain get destroyed through relinquish_memory.
>
> Clearly they can't be on both lists. Hence you can put them on the new list
> only _after_ having taken them off the "normal" list. That "taking off the
> normal list" should happen when the last ref is droppe
RE: [PATCH v1 3/5] xen/arm: unpopulate memory when domain on static allocation
Hi Julien and Jan > -Original Message- > From: Julien Grall > Sent: Saturday, April 2, 2022 2:54 AM > To: Penny Zheng ; Jan Beulich > Cc: Wei Chen ; Henry Wang ; > Andrew Cooper ; George Dunlap > ; Stefano Stabellini ; Wei > Liu ; xen-devel@lists.xenproject.org > Subject: Re: [PATCH v1 3/5] xen/arm: unpopulate memory when domain on > static allocation > > Hi Penny, > > On 31/03/2022 11:30, Penny Zheng wrote: > > Another reason I want to keep page allocated is that if putting pages > > in resv_page_list upon dropping the last ref, we need to do a lot > > things on pages to totally let it free, like set its owner to NULL, > > changing page state from in_use to free, etc. > This is not only about optimization here. Bad things can happen if you let a > page in state free that is not meant to be used by the buddy allocator (e.g. > it > was reserved for static memory). > > I discovered it earlier this year when trying to optimize > init_heap_pages() for Live-Update. It was quite hard to debug because the > corruption very rarely happened. So let me explain it before you face the same > issue :). > > free_heap_pages() will try to merge the about-to-be-freed chunk with the > predecessor and/or successor. To know if the page can be merged, the > algorithm is looking at whether the chunk is suitably aligned (e.g. same > order) and if the page is in state free. > > AFAICT, the pages belonging to the buddy allocator could be right next to > region reserved memory. So there is a very slim chance that > free_heap_pages() may decide to merge a chunk from the static region with > the about-to-be-free chunk. Nothing very good will ensue. > Oh,,, that's a thousand true. If the free static region is the buddy of the about-to-be-free chunk, current code will merge the static region to the heap, which is unacceptable. I'll fix it in this patch serie too and I'm also preferring option 1~ And for unpopulating memory on runtime, I'll do what Jan suggests, adding a new logic of moving the page from d->page_list to d->resv_page_list in arch_free_heap_page() for reserved pages. > Technically, this is already a bug in the already merged implementation of the > static memory allocator. > > I can see two ways to fix it: >1) Update free_heap_pages() to check whether the page has PGC_reserved > set. >2) Use a different state for pages used by the static allocator. > > So far my preference is leaning towards 1. But I would like to hear other > opinions. > > Cheers, > > -- > Julien Grall
RE: [PATCH v1 02/13] xen/arm: introduce a special domain DOMID_SHARED
Hi Julien and Stefano > -Original Message- > From: Julien Grall > Sent: Saturday, April 9, 2022 5:12 PM > To: Stefano Stabellini ; Jan Beulich > > Cc: Penny Zheng ; nd ; Penny Zheng > ; Bertrand Marquis > ; Volodymyr Babchuk > ; Andrew Cooper > ; George Dunlap ; > Wei Liu ; xen-devel@lists.xenproject.org > Subject: Re: [PATCH v1 02/13] xen/arm: introduce a special domain > DOMID_SHARED > > Hi Stefano, > > On 21/03/2022 20:03, Stefano Stabellini wrote: > > On Mon, 21 Mar 2022, Jan Beulich wrote: > >> On 18.03.2022 22:50, Stefano Stabellini wrote: > >>> On Fri, 18 Mar 2022, Jan Beulich wrote: > >>>> On 11.03.2022 07:11, Penny Zheng wrote: > >>>>> In case to own statically shared pages when owner domain is not > >>>>> explicitly defined, this commits propose a special domain > >>>>> DOMID_SHARED, and we assign it 0x7FF5, as one of the system domains. > >>>>> > >>>>> Statically shared memory reuses the same way of initialization > >>>>> with static memory, hence this commits proposes a new Kconfig > >>>>> CONFIG_STATIC_SHM to wrap related codes, and this option depends > on static memory(CONFIG_STATIC_MEMORY). > >>>>> > >>>>> We intends to do shared domain creation after setup_virt_paging so > >>>>> shared domain could successfully do p2m initialization. > >>>> > >>>> There's nothing said here, in the earlier patch, or in the cover > >>>> letter about the security aspects of this. There is a reason we > >>>> haven't been allowing arbitrary, un-supervised sharing of memory > >>>> between domains. It wants clarifying why e.g. grants aren't an > >>>> option to achieve what you need, and how you mean to establish > >>>> which domains are / aren't permitted to access any individual page > owned by this domain. > >>> > >>> > >>> I'll let Penny write a full reply but I'll chime in to try to help > >>> with the explanation. > >>> > >>> This is not arbitrary un-supervised sharing of memory between > >>> domains, which indeed is concerning. > >>> > >>> This is statically-configured, supervised by the system > >>> configurator, sharing of memory between domains. > >>> > >>> And in fact safety (which is just a different aspect of security) is > >>> one of the primary goals for this work. > >>> > >>> In safety-critical environments, it is not considered safe to > >>> dynamically change important configurations at runtime. Everything > >>> should be statically defined and statically verified. > >>> > >>> In this case, if the system configuration knows a priori that there > >>> are only 2 VM and they need to communication over shared memory, it > >>> is safer to pre-configure the shared memory at build time rather > >>> than let the VMs attempt to share memory at runtime. It is faster too. > >>> > >>> The only way to trigger this static shared memory configuration > >>> should be via device tree, which is at the same level as the XSM > >>> rules themselves. > >>> > >>> Hopefully I made things clearer and not murkier :-) > >> > >> It adds some helpful background, yes, but at the same time it doesn't > >> address the security concern at all: How are access permissions > >> managed when the owning domain is a special one? I haven't spotted > >> any recording of the domains which are actually permitted to map / > >> access the pages in questions. (But of course I also only looked at > >> non-Arm-specific code. I'd expect such code not to live in arch- > >> specific files.) > > > > All this static memory sharing is statically done at __init time only. > > It should not be possible to trigger any further memory sharing at > > runtime (if there is, that would be a bug). > > Looking at the code, get_pg_owner() will be able to handle DOMID_SHARED. > So anyone that is permitted to access DOMID_SHARED will be able to map any > memory region at runtime. > > > There are no new interfaces for the guest to map this memory because > > it is already "pre-mapped". > > It can via XENMAPSPACE_gmfn_foreign (assuming proper permission). > Correct me if I'm wrong: The existing XENMAPSPACE_gmfn_foreign only allows privileged Dom0 to map memory pages from one foreign DomU to itself. So It can happen that Dom0 is using XENMAPSPACE_gmfn_foreign to (maliciously?) access shared memory owned by DOMID_SHARED, and for now only Dom0 could. So, maybe we should enhance the check of xsm_map_gmfn_foreign() to forbid the access to DOMID_SHARED, hmm, but static shared memory region could actually be owned by any arbitrary domain. So, how about we add restriction on the page itself? Pages of static shared memory region are static memory(with PGC_reserved flag on), so maybe we could restrict XENMAPSPACE_gmfn_foreign to access any PGC_reserved pages? > Cheers, > > -- Cheers, -- Penny Zheng > Julien Grall
RE: [PATCH v1 02/13] xen/arm: introduce a special domain DOMID_SHARED
Hi jan
> -Original Message-
> From: Jan Beulich
> Sent: Friday, March 18, 2022 4:53 PM
> To: Penny Zheng
> Cc: nd ; Penny Zheng
> ; Stefano Stabellini
> ; Julien Grall ; Bertrand Marquis
> ; Volodymyr Babchuk
> ; Andrew Cooper
> ; George Dunlap ;
> Wei Liu ; xen-devel@lists.xenproject.org
> Subject: Re: [PATCH v1 02/13] xen/arm: introduce a special domain
> DOMID_SHARED
>
> On 11.03.2022 07:11, Penny Zheng wrote:
> > In case to own statically shared pages when owner domain is not
> > explicitly defined, this commits propose a special domain
> > DOMID_SHARED, and we assign it 0x7FF5, as one of the system domains.
> >
> > Statically shared memory reuses the same way of initialization with
> > static memory, hence this commits proposes a new Kconfig
> > CONFIG_STATIC_SHM to wrap related codes, and this option depends on
> static memory(CONFIG_STATIC_MEMORY).
> >
> > We intends to do shared domain creation after setup_virt_paging so
> > shared domain could successfully do p2m initialization.
>
> There's nothing said here, in the earlier patch, or in the cover letter about
> the
> security aspects of this. There is a reason we haven't been allowing
> arbitrary,
> un-supervised sharing of memory between domains. It wants clarifying why e.g.
> grants aren't an option to achieve what you need, and how you mean to
> establish which domains are / aren't permitted to access any individual page
> owned by this domain.
>
I'll add the security aspects what Stefano explains in the cover letter next
serie
for better understanding.
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -106,6 +106,13 @@ config TEE
> >
> > source "arch/arm/tee/Kconfig"
> >
> > +config STATIC_SHM
> > + bool "Statically shared memory on a dom0less system" if UNSUPPORTED
> > + depends on STATIC_MEMORY
> > + default n
>
> Nit: "default n" is redundant and hence would imo better be omitted.
>
> > @@ -712,12 +716,16 @@ int arch_domain_create(struct domain *d,
> > d->arch.directmap = flags & CDF_directmap;
> >
> > /* p2m_init relies on some value initialized by the IOMMU subsystem */
> > -if ( (rc = iommu_domain_init(d, config->iommu_opts)) != 0 )
> > +if ( (rc = iommu_domain_init(d, is_shared_domain(d) ? 0 :
> > + config->iommu_opts)) != 0 )
>
> Nit: Overlong line.
>
> > --- a/xen/arch/arm/setup.c
> > +++ b/xen/arch/arm/setup.c
> > @@ -855,6 +855,20 @@ static bool __init is_dom0less_mode(void)
> > return ( !dom0found && domUfound ); }
> >
> > +#ifdef CONFIG_STATIC_SHM
> > +static void __init setup_shared_domain(void) {
> > +/*
> > + * Initialise our DOMID_SHARED domain.
> > + * This domain owns statically shared pages when owner domain is not
> > + * explicitly defined.
> > + */
> > +dom_shared = domain_create(DOMID_SHARED, NULL, CDF_directmap);
> > +if ( IS_ERR(dom_shared) )
> > +panic("Failed to create d[SHARED]: %ld\n",
> > +PTR_ERR(dom_shared));
>
> I don't think this should be a panic - the system ought to be able to come up
> fine, just without actually using this domain. After all this is an optional
> feature which may not actually be used.
>
> Also, along the lines of what Stefano has said, this setting up of the domain
> would also better live next to where the other special domains are set up. And
> even if it was to remain here, ...
>
The reason why I place the setting up here is that DOMID_SHARED needs to map
pre-configured static shared memory in its p2m table, so it must be set up
after system P2M initialization(setup_virt_paging()). setup_system_domains()
is called before system P2M initialization on xen/arch/arm/setup.c, which
can't meet the requirement.
> > @@ -1022,6 +1036,14 @@ void __init start_xen(unsigned long
> boot_phys_offset,
> > apply_alternatives_all();
> > enable_errata_workarounds();
> >
> > +#ifdef CONFIG_STATIC_SHM
> > +/*
> > + * This needs to be called **after** setup_virt_paging so shared
> > + * domains could successfully do p2m initialization.
> > + */
> > +setup_shared_domain();
> > +#endif
>
> ... the #ifdef-ary here should be avoided by moving the other #ifdef inside
> the
> function body.
>
> > --- a/xen/common/domain.c
> > +++ b/xen/common/domain.c
> > @@ -643,11 +643,14 @@ struct domain *domain_create(domid_t domid,
> >
> > rangeset_domain_initialise(d);
[PATCH v2 1/6] xen/arm: do not free reserved memory into heap
Pages as guest RAM for static domain, shall be reserved to this domain only.
So in case reserved pages being used for other purpose, users
shall not free them back to heap, even when last ref gets dropped.
free_staticmem_pages will be called by free_domheap_pages in runtime
for static domain freeing memory resource, so let's drop the __init
flag.
Signed-off-by: Penny Zheng
---
v2 changes:
- new commit
---
xen/common/page_alloc.c | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 319029140f..9a3e9c1328 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -2488,7 +2488,13 @@ void free_domheap_pages(struct page_info *pg, unsigned
int order)
scrub = 1;
}
-free_heap_pages(pg, order, scrub);
+#ifdef CONFIG_STATIC_MEMORY
+if ( pg->count_info & PGC_reserved )
+/* Reserved page shall not go back to the heap. */
+free_staticmem_pages(pg, 1 << order, scrub);
+else
+#endif
+free_heap_pages(pg, order, scrub);
}
if ( drop_dom_ref )
@@ -2636,7 +2642,7 @@ struct domain *get_pg_owner(domid_t domid)
#ifdef CONFIG_STATIC_MEMORY
/* Equivalent of free_heap_pages to free nr_mfns pages of static memory. */
-void __init free_staticmem_pages(struct page_info *pg, unsigned long nr_mfns,
+void free_staticmem_pages(struct page_info *pg, unsigned long nr_mfns,
bool need_scrub)
{
mfn_t mfn = page_to_mfn(pg);
--
2.25.1
[PATCH V2 0/6] populate/unpopulate memory when domain on static
Today when a domain unpopulates the memory on runtime, they will always hand the memory over to the heap allocator. And it will be a problem if it is a static domain. Pages as guest RAM for static domain shall always be reserved to only this domain and not be used for any other purposes, so they shall never go back to heap allocator. This patch serie intends to fix this issue, by adding pages on the new list resv_page_list after having taken them off the "normal" list, when unpopulating memory, and retrieving pages from resv page list(resv_page_list) when populating memory. --- v2 changes: - let "flags" live in the struct domain. So other arch can take advantage of it in the future - change name from "is_domain_on_static_allocation" to "is_domain_static()" - put reserved pages on resv_page_list after having taken them off the "normal" list - introduce acquire_reserved_page to retrieve reserved pages from resv_page_list - forbid non-zero-order requests in populate_physmap - let is_domain_static return ((void)(d), false) on x86 - fix coding style Penny Zheng (6): xen/arm: do not free reserved memory into heap xen/arm: do not merge reserved pages in free_heap_pages() xen: add field "flags" to cover all internal CDF_XXX xen/arm: introduce CDF_staticmem xen/arm: unpopulate memory when domain is static xen/arm: retrieve reserved pages on populate_physmap xen/arch/arm/domain.c | 2 -- xen/arch/arm/domain_build.c | 5 +++- xen/arch/arm/include/asm/domain.h | 5 ++-- xen/arch/arm/include/asm/mm.h | 17 + xen/common/domain.c | 7 ++ xen/common/memory.c | 29 ++ xen/common/page_alloc.c | 40 +-- xen/include/xen/domain.h | 2 ++ xen/include/xen/mm.h | 1 + xen/include/xen/sched.h | 9 +++ 10 files changed, 110 insertions(+), 7 deletions(-) -- 2.25.1
[PATCH v2 2/6] xen/arm: do not merge reserved pages in free_heap_pages()
There is a slim chance that free_heap_pages() may decide to merge a chunk from the static region(PGC_reserved) with the about-to-be-free chunk. So in order to avoid the above scenario, this commit updates free_heap_pages() to check whether the predecessor and/or successor has PGC_reserved set, when trying to merge the about-to-be-freed chunk with the predecessor and/or successor. Signed-off-by: Penny Zheng --- v2 changes: - new commit --- xen/common/page_alloc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 9a3e9c1328..8ba38bca9a 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -1479,6 +1479,7 @@ static void free_heap_pages( /* Merge with predecessor block? */ if ( !mfn_valid(page_to_mfn(predecessor)) || !page_state_is(predecessor, free) || + (predecessor->count_info & PGC_reserved) || (PFN_ORDER(predecessor) != order) || (phys_to_nid(page_to_maddr(predecessor)) != node) ) break; @@ -1502,6 +1503,7 @@ static void free_heap_pages( /* Merge with successor block? */ if ( !mfn_valid(page_to_mfn(successor)) || !page_state_is(successor, free) || + (successor->count_info & PGC_reserved) || (PFN_ORDER(successor) != order) || (phys_to_nid(page_to_maddr(successor)) != node) ) break; -- 2.25.1
[PATCH v2 5/6] xen/arm: unpopulate memory when domain is static
Today when a domain unpopulates the memory on runtime, they will always
hand the memory back to the heap allocator. And it will be a problem if domain
is static.
Pages as guest RAM for static domain shall be reserved to only this domain
and not be used for any other purposes, so they shall never go back to heap
allocator.
This commit puts reserved pages on the new list resv_page_list only after
having taken them off the "normal" list, when the last ref dropped.
Signed-off-by: Penny Zheng
---
v2 changes:
- put reserved pages on resv_page_list after having taken them off
the "normal" list
---
xen/arch/arm/include/asm/mm.h | 17 +
xen/common/domain.c | 4
xen/include/xen/sched.h | 6 ++
3 files changed, 27 insertions(+)
diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h
index 424aaf2823..a2dde01cfa 100644
--- a/xen/arch/arm/include/asm/mm.h
+++ b/xen/arch/arm/include/asm/mm.h
@@ -358,6 +358,23 @@ void clear_and_clean_page(struct page_info *page);
unsigned int arch_get_dma_bitsize(void);
+/*
+ * Put free pages on the resv page list after having taken them
+ * off the "normal" page list, when pages from static memory
+ */
+#ifdef CONFIG_STATIC_MEMORY
+#define arch_free_heap_page(d, pg) {\
+if ( (pg)->count_info & PGC_reserved ) \
+{ \
+page_list_del(pg, page_to_list(d, pg)); \
+page_list_add_tail(pg, &(d)->resv_page_list); \
+(d)->resv_pages++; \
+} \
+else\
+page_list_del(pg, page_to_list(d, pg)); \
+}
+#endif
+
#endif /* __ARCH_ARM_MM__ */
/*
* Local variables:
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 859cc13d3b..b499cf8d2c 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -605,6 +605,10 @@ struct domain *domain_create(domid_t domid,
INIT_PAGE_LIST_HEAD(&d->page_list);
INIT_PAGE_LIST_HEAD(&d->extra_page_list);
INIT_PAGE_LIST_HEAD(&d->xenpage_list);
+#ifdef CONFIG_STATIC_MEMORY
+INIT_PAGE_LIST_HEAD(&d->resv_page_list);
+#endif
+
spin_lock_init(&d->node_affinity_lock);
d->node_affinity = NODE_MASK_ALL;
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 68eb08058e..85ef378752 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -376,6 +376,9 @@ struct domain
struct page_list_head page_list; /* linked list */
struct page_list_head extra_page_list; /* linked list (size extra_pages) */
struct page_list_head xenpage_list; /* linked list (size xenheap_pages) */
+#ifdef CONFIG_STATIC_MEMORY
+struct page_list_head resv_page_list; /* linked list (size resv_pages) */
+#endif
/*
* This field should only be directly accessed by domain_adjust_tot_pages()
@@ -395,6 +398,9 @@ struct domain
#ifdef CONFIG_MEM_PAGING
atomic_t paged_pages; /* paged-out pages */
#endif
+#ifdef CONFIG_STATIC_MEMORY
+unsigned int resv_pages;/* reserved pages from static region.
*/
+#endif
/* Scheduling. */
void*sched_priv;/* scheduler-specific data */
--
2.25.1
[PATCH v2 3/6] xen: add field "flags" to cover all internal CDF_XXX
With more and more CDF_xxx internal flags in and to save the space, this
commit introduces a new field "flags" in struct domain to store CDF_*
internal flags directly.
Another new CDF_xxx will be introduced in the next patch.
Signed-off-by: Penny Zheng
---
v2 changes:
- let "flags" live in the struct domain. So other arch can take
advantage of it in the future
- fix coding style
---
xen/arch/arm/domain.c | 2 --
xen/arch/arm/include/asm/domain.h | 3 +--
xen/common/domain.c | 3 +++
xen/include/xen/sched.h | 3 +++
4 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 8110c1df86..74189d9878 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -709,8 +709,6 @@ int arch_domain_create(struct domain *d,
ioreq_domain_init(d);
#endif
-d->arch.directmap = flags & CDF_directmap;
-
/* p2m_init relies on some value initialized by the IOMMU subsystem */
if ( (rc = iommu_domain_init(d, config->iommu_opts)) != 0 )
goto fail;
diff --git a/xen/arch/arm/include/asm/domain.h
b/xen/arch/arm/include/asm/domain.h
index ed63c2b6f9..36ec00e62d 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -29,7 +29,7 @@ enum domain_type {
#define is_64bit_domain(d) (0)
#endif
-#define is_domain_direct_mapped(d) (d)->arch.directmap
+#define is_domain_direct_mapped(d) ((d)->flags & CDF_directmap)
/*
* Is the domain using the host memory layout?
@@ -103,7 +103,6 @@ struct arch_domain
void *tee;
#endif
-bool directmap;
} __cacheline_aligned;
struct arch_vcpu
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 351029f8b2..859cc13d3b 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -568,6 +568,9 @@ struct domain *domain_create(domid_t domid,
/* Sort out our idea of is_system_domain(). */
d->domain_id = domid;
+/* Holding CDF_* internal flags. */
+d->flags = flags;
+
/* Debug sanity. */
ASSERT(is_system_domain(d) ? config == NULL : config != NULL);
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index ed8539f6d2..68eb08058e 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -591,6 +591,9 @@ struct domain
struct ioreq_server *server[MAX_NR_IOREQ_SERVERS];
} ioreq_server;
#endif
+
+/* Holding CDF_* constant. Internal flags for domain creation. */
+uint32_t flags;
};
static inline struct page_list_head *page_to_list(
--
2.25.1
[PATCH v2 4/6] xen/arm: introduce CDF_staticmem
In order to have an easy and quick way to find out whether this domain memory
is statically configured, this commit introduces a new flag CDF_staticmem and a
new helper is_domain_static to tell.
Signed-off-by: Penny Zheng
---
v2 changes:
- change name from "is_domain_on_static_allocation" to "is_domain_static()"
---
xen/arch/arm/domain_build.c | 5 -
xen/arch/arm/include/asm/domain.h | 2 ++
xen/include/xen/domain.h | 2 ++
3 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 8be01678de..4e62fd0bf1 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3191,9 +3191,12 @@ void __init create_domUs(void)
if ( !dt_device_is_compatible(node, "xen,domain") )
continue;
+if ( dt_find_property(node, "xen,static-mem", NULL) )
+flags |= CDF_staticmem;
+
if ( dt_property_read_bool(node, "direct-map") )
{
-if ( !IS_ENABLED(CONFIG_STATIC_MEMORY) || !dt_find_property(node,
"xen,static-mem", NULL) )
+if ( !IS_ENABLED(CONFIG_STATIC_MEMORY) || !(flags & CDF_staticmem)
)
panic("direct-map is not valid for domain %s without static
allocation.\n",
dt_node_name(node));
diff --git a/xen/arch/arm/include/asm/domain.h
b/xen/arch/arm/include/asm/domain.h
index 36ec00e62d..b097433f9f 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -31,6 +31,8 @@ enum domain_type {
#define is_domain_direct_mapped(d) ((d)->flags & CDF_directmap)
+#define is_domain_static(d) ((d)->flags & CDF_staticmem)
+
/*
* Is the domain using the host memory layout?
*
diff --git a/xen/include/xen/domain.h b/xen/include/xen/domain.h
index 1c3c88a14d..35dc7143a4 100644
--- a/xen/include/xen/domain.h
+++ b/xen/include/xen/domain.h
@@ -34,6 +34,8 @@ void arch_get_domain_info(const struct domain *d,
#ifdef CONFIG_ARM
/* Should domain memory be directly mapped? */
#define CDF_directmap(1U << 1)
+/* Is domain memory on static allocation? */
+#define CDF_staticmem(1U << 2)
#endif
/*
--
2.25.1
[PATCH v2 6/6] xen/arm: retrieve reserved pages on populate_physmap
When static domain populates memory through populate_physmap on runtime,
other than allocating from heap, it shall retrieve reserved pages from
resv_page_list to make sure that guest RAM is still restricted in statically
configured memory regions. And this commit introduces a new helper
acquire_reserved_page to make it work.
Signed-off-by: Penny Zheng
---
v2 changes:
- introduce acquire_reserved_page to retrieve reserved pages from
resv_page_list
- forbid non-zero-order requests in populate_physmap
- let is_domain_static return ((void)(d), false) on x86
---
xen/common/memory.c | 29 +
xen/common/page_alloc.c | 28
xen/include/xen/mm.h| 1 +
3 files changed, 58 insertions(+)
diff --git a/xen/common/memory.c b/xen/common/memory.c
index 69b0cd1e50..f7729dfa5c 100644
--- a/xen/common/memory.c
+++ b/xen/common/memory.c
@@ -35,6 +35,10 @@
#include
#endif
+#ifndef is_domain_static
+#define is_domain_static(d) ((void)(d), false)
+#endif
+
struct memop_args {
/* INPUT */
struct domain *domain; /* Domain to be affected. */
@@ -245,6 +249,31 @@ static void populate_physmap(struct memop_args *a)
mfn = _mfn(gpfn);
}
+#ifdef CONFIG_STATIC_MEMORY
+else if ( is_domain_static(d) )
+{
+/*
+ * No easy way to guarantee the retreived pages are contiguous,
+ * so forbid non-zero-order requests here.
+ */
+if ( a->extent_order != 0 )
+{
+gdprintk(XENLOG_INFO,
+ "Could not allocate non-zero-order pages for
static %pd.\n.",
+ d);
+goto out;
+}
+
+mfn = acquire_reserved_page(d, a->memflags);
+if ( mfn_eq(mfn, INVALID_MFN) )
+{
+gdprintk(XENLOG_INFO,
+ "%pd: failed to retrieve a reserved page.\n.",
+ d);
+goto out;
+}
+}
+#endif
else
{
page = alloc_domheap_pages(d, a->extent_order, a->memflags);
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 8ba38bca9a..759d612eb8 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -2770,6 +2770,34 @@ int __init acquire_domstatic_pages(struct domain *d,
mfn_t smfn,
return 0;
}
+
+/*
+ * Acquire a page from reserved page list(resv_page_list), when populating
+ * memory for static domain on runtime.
+ */
+mfn_t acquire_reserved_page(struct domain *d, unsigned int memflags)
+{
+struct page_info *page;
+mfn_t smfn;
+
+/* Acquire a page from reserved page list(resv_page_list). */
+page = page_list_remove_head(&d->resv_page_list);
+if ( unlikely(!page) )
+{
+printk(XENLOG_ERR
+ "%pd: failed to acquire a reserved page %"PRI_mfn".\n",
+ d, mfn_x(page_to_mfn(page)));
+return INVALID_MFN;
+}
+d->resv_pages--;
+
+smfn = page_to_mfn(page);
+
+if ( acquire_domstatic_pages(d, smfn, 1, memflags) )
+return INVALID_MFN;
+
+return smfn;
+}
#endif
/*
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 3be754da92..6ef5a6adcd 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -91,6 +91,7 @@ void free_staticmem_pages(struct page_info *pg, unsigned long
nr_mfns,
bool need_scrub);
int acquire_domstatic_pages(struct domain *d, mfn_t smfn, unsigned int nr_mfns,
unsigned int memflags);
+mfn_t acquire_reserved_page(struct domain *d, unsigned int memflags);
#endif
/* Map machine page range in Xen virtual address space. */
--
2.25.1
RE: [PATCH V2 0/6] populate/unpopulate memory when domain on static
Hi jan > -Original Message- > From: Jan Beulich > Sent: Tuesday, April 19, 2022 4:47 PM > To: Penny Zheng > Cc: Wei Chen ; Andrew Cooper > ; George Dunlap ; > Julien Grall ; Stefano Stabellini ; > Wei > Liu ; Bertrand Marquis ; > Volodymyr Babchuk ; xen- > de...@lists.xenproject.org > Subject: Re: [PATCH V2 0/6] populate/unpopulate memory when domain on > static > > On 18.04.2022 14:22, Penny Zheng wrote: > > Today when a domain unpopulates the memory on runtime, they will > > always hand the memory over to the heap allocator. And it will be a > > problem if it is a static domain. Pages as guest RAM for static domain > > shall always be reserved to only this domain and not be used for any > > other purposes, so they shall never go back to heap allocator. > > > > This patch serie intends to fix this issue, by adding pages on the new > > list resv_page_list after having taken them off the "normal" list, > > when unpopulating memory, and retrieving pages from resv page > > list(resv_page_list) when populating memory. > > > > --- > > v2 changes: > > - let "flags" live in the struct domain. So other arch can take > > advantage of it in the future > > - change name from "is_domain_on_static_allocation" to > "is_domain_static()" > > I have reservations against this new name: This could mean far more aspects of > the domain are static than just its memory assignment. Was this intended (or > at least considered)? > Ok. Julien gave me two suggestions back the day, maybe the other "is_domain_using_staticmem()" is better and to the point. I'll change it in the next serie. > Jan
RE: [PATCH v2 1/6] xen/arm: do not free reserved memory into heap
Hi jan > -Original Message- > From: Jan Beulich > Sent: Tuesday, April 19, 2022 4:59 PM > To: Penny Zheng > Cc: Wei Chen ; Andrew Cooper > ; George Dunlap ; > Julien Grall ; Stefano Stabellini ; > Wei > Liu ; xen-devel@lists.xenproject.org > Subject: Re: [PATCH v2 1/6] xen/arm: do not free reserved memory into heap > > On 18.04.2022 14:22, Penny Zheng wrote: > > Pages as guest RAM for static domain, shall be reserved to this domain only. > > Is there "used" missing as the 2nd word of the sentence? > > > So in case reserved pages being used for other purpose, users shall > > not free them back to heap, even when last ref gets dropped. > > > > free_staticmem_pages will be called by free_domheap_pages in runtime > > for static domain freeing memory resource, so let's drop the __init > > flag. > > > > Signed-off-by: Penny Zheng > > --- > > v2 changes: > > - new commit > > --- > > xen/common/page_alloc.c | 10 -- > > 1 file changed, 8 insertions(+), 2 deletions(-) > > With this diffstat the patch subject prefix is somewhat misleading; I first > thought I could skip this patch. > Oh, sorry. Will change the 'xen/arm' to 'xen' > > --- a/xen/common/page_alloc.c > > +++ b/xen/common/page_alloc.c > > @@ -2488,7 +2488,13 @@ void free_domheap_pages(struct page_info *pg, > unsigned int order) > > scrub = 1; > > } > > > > -free_heap_pages(pg, order, scrub); > > +#ifdef CONFIG_STATIC_MEMORY > > +if ( pg->count_info & PGC_reserved ) > > +/* Reserved page shall not go back to the heap. */ > > +free_staticmem_pages(pg, 1 << order, scrub); > > 1UL with, in particular, the function parameter by "unsigned long". > > By calling free_staticmem_pages() at runtime, you make the previous race free > (because of init-time only) update of .count_info there racy. Making a clone > of > that function just for this difference would likely be excessive, so I'd > suggest to > change the code there to > > /* In case initializing page of static memory, mark it PGC_reserved. > */ > if ( !(pg[i].count_info & PGC_reserved) ) > pg[i].count_info |= PGC_reserved; > Learned! > > +else > > +#endif > > +free_heap_pages(pg, order, scrub); > > Of course it would be nice to avoid the #ifdef-ary here. May I ask that you > introduce a stub free_staticmem_pages() for the !CONFIG_STATIC_MEMORY > case, such that the construct can become > Sure, will do. > if ( !(pg->count_info & PGC_reserved) ) > free_heap_pages(pg, order, scrub); > else > /* Reserved page shall not go back to the heap. */ > free_staticmem_pages(pg, 1 << order, scrub); > > Another question is whether the distinction should be made here in the first > place. Would it perhaps better belong in free_heap_pages() itself, thus also > covering other potential call sites? Of course this depends on where, long > term, > reserved pages can / will be used. > For domains to be truly static, Xen's own allocations to manage the domain > may also want to come from the reserved set ... > Yes, you're right. I'll defer the distinction to free_heap_pages. And refine the in-code comment above free_staticmem_pages, in the first place, I was intending to make it equivalent of free_heap_pages to free static memory. However as you said, if letting free_heap_pages call free_staticmem_pages, it will cover other potential call site. We've already been trying to enable p2m pool on arm, and in the future, maybe the pages constituting the pool shall not come from heap, but from reserved set, if the domain is fully static > > @@ -2636,7 +2642,7 @@ struct domain *get_pg_owner(domid_t domid) > > > > #ifdef CONFIG_STATIC_MEMORY > > /* Equivalent of free_heap_pages to free nr_mfns pages of static > > memory. */ -void __init free_staticmem_pages(struct page_info *pg, > > unsigned long nr_mfns, > > +void free_staticmem_pages(struct page_info *pg, unsigned long > > +nr_mfns, > > bool need_scrub) > > This line now wants its indentation adjusted. > > Jan
RE: [PATCH v1 06/13] xen/arm: set up shared memory foreign mapping for borrower domain
Hi julien
> -Original Message-
> From: Julien Grall
> Sent: Saturday, April 9, 2022 6:59 AM
> To: Penny Zheng ; xen-devel@lists.xenproject.org
> Cc: nd ; Stefano Stabellini ; Bertrand
> Marquis ; Volodymyr Babchuk
>
> Subject: Re: [PATCH v1 06/13] xen/arm: set up shared memory foreign
> mapping for borrower domain
>
> Hi Penny,
>
> On 11/03/2022 06:11, Penny Zheng wrote:
> > From: Penny Zheng
> >
> > This commits introduces a new helper guest_physmap_add_shm to set up
> > shared memory foreign mapping for borrower domain.
> >
> > Firstly it should get and take reference of statically shared pages
> > from owner dom_shared. Then it will setup P2M foreign memory map of
> > these statically shared pages for borrower domain.
> >
> > This commits only considers owner domain is the default dom_shared,
> > the other scenario will be covered in the following patches.
> >
> > Signed-off-by: Penny Zheng
> > ---
> > xen/arch/arm/domain_build.c | 52
> +
> > 1 file changed, 52 insertions(+)
> >
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index 984e70e5fc..8cee5ffbd1 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -798,6 +798,48 @@ static int __init allocate_shared_memory(struct
> domain *d,
> > return ret;
> > }
> >
> > +static int __init guest_physmap_add_shm(struct domain *od, struct domain
> *bd,
> > +unsigned long o_gfn,
> > +unsigned long b_gfn,
> > +unsigned long nr_gfns) {
> > +struct page_info **pages = NULL;
> > +p2m_type_t p2mt, t;
> > +int ret = 0;
>
> You don't need to initialize ret.
>
> > +
> > +pages = xmalloc_array(struct page_info *, nr_gfns);
> > +if ( !pages )
> > +return -ENOMEM;
> > +
> > +/*
> > + * Take reference of statically shared pages from owner domain.
> > + * Reference will be released when destroying shared memory region.
> > + */
> > +ret = get_pages_from_gfn(od, o_gfn, nr_gfns, pages, &p2mt, P2M_ALLOC);
> > +if ( ret )
> > +{
> > +ret = -EINVAL;
> > +goto fail_pages;
> > +}
> > +
> > +if ( p2m_is_ram(p2mt) )
> > +t = (p2mt == p2m_ram_rw) ? p2m_map_foreign_rw :
> p2m_map_foreign_ro;
> > +else
> > +{
> > +ret = -EINVAL;
> > +goto fail_pages;
>
> Where would we release the references?
>
> > +}
> > +
> > +/* Set up guest foreign map. */
> > +ret = guest_physmap_add_pages(bd, _gfn(b_gfn), page_to_mfn(pages[0]),
> > + nr_gfns, t);
>
> A few things:
>- The beginning of the code assumes that the MFN may be discontiguous in
> the physical memory. But here, you are assuming they are contiguous.
> If you want the latter, then you should check the MFNs are contiguous.
> That said, I am not sure if this restriction is really necessary.
>
Oh,, caught me, it never occurred to me that the pages in array `pages` could
be discontinuous.
I definitely want the latter. This function is always dealing with a memory
region
(contiguous MFNS) each time. So maybe a loop to check
page_to_mfn(pages[i]) == mfn_add(smfn, i) is needed.
>- IIRC, guest_physmap_add_pages() doesn't revert the mappings. So you
> need to revert it in case of failure.
>
Yes, both failure points you are referring to are requiring reverting the
mappings.
> Cheers,
>
> --
> Julien Grall
RE: [PATCH v1 06/13] xen/arm: set up shared memory foreign mapping for borrower domain
Hi
> -Original Message-
> From: Julien Grall
> Sent: Saturday, April 9, 2022 5:31 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org
> Cc: nd ; Stefano Stabellini ; Bertrand
> Marquis ; Volodymyr Babchuk
>
> Subject: Re: [PATCH v1 06/13] xen/arm: set up shared memory foreign
> mapping for borrower domain
>
> Hi,
>
> On 08/04/2022 23:59, Julien Grall wrote:
> > On 11/03/2022 06:11, Penny Zheng wrote:
> >> From: Penny Zheng
> >>
> >> This commits introduces a new helper guest_physmap_add_shm to set up
> >> shared memory foreign mapping for borrower domain.
> >>
> >> Firstly it should get and take reference of statically shared pages
> >> from owner dom_shared. Then it will setup P2M foreign memory map of
> >> these statically shared pages for borrower domain.
> >>
> >> This commits only considers owner domain is the default dom_shared,
> >> the other scenario will be covered in the following patches.
> >>
> >> Signed-off-by: Penny Zheng
> >> ---
> >> Â xen/arch/arm/domain_build.c | 52
> >> +
> >> Â 1 file changed, 52 insertions(+)
> >>
> >> diff --git a/xen/arch/arm/domain_build.c
> >> b/xen/arch/arm/domain_build.c index 984e70e5fc..8cee5ffbd1 100644
> >> --- a/xen/arch/arm/domain_build.c
> >> +++ b/xen/arch/arm/domain_build.c
> >> @@ -798,6 +798,48 @@ static int __init allocate_shared_memory(struct
> >> domain *d,
> >> Â return ret;
> >> Â }
> >> +static int __init guest_physmap_add_shm(struct domain *od, struct
> >> domain *bd,
> >> +Â Â Â unsigned long o_gfn,
> >> +Â Â Â unsigned long b_gfn,
> >> +Â Â Â unsigned long nr_gfns) {
> >> +Â Â Â struct page_info **pages = NULL;
> >> +Â Â Â p2m_type_t p2mt, t;
> >> +Â Â Â int ret = 0;
> >
> > You don't need to initialize ret.
> >
> >> +
> >> +Â Â Â pages = xmalloc_array(struct page_info *, nr_gfns);
> >> +Â Â Â if ( !pages )
> >> +Â Â Â return -ENOMEM;
> >> +
> >> +Â Â Â /*
> >> + * Take reference of statically shared pages from owner domain.
> >> + * Reference will be released when destroying shared memory region.
> >> + */
> >> +Â Â Â ret = get_pages_from_gfn(od, o_gfn, nr_gfns, pages, &p2mt,
> >> P2M_ALLOC);
> >> +Â Â Â if ( ret )
> >> +Â Â Â {
> >> +Â Â Â ret = -EINVAL;
> >> +Â Â Â goto fail_pages;
> >> +Â Â Â }
> >> +
> >> +Â Â Â if ( p2m_is_ram(p2mt) )
> >> +Â Â Â t = (p2mt == p2m_ram_rw) ? p2m_map_foreign_rw :
> >> p2m_map_foreign_ro;
> >> +Â Â Â else
> >> +Â Â Â {
> >> +Â Â Â ret = -EINVAL;
> >> +Â Â Â goto fail_pages;
> >
> > Where would we release the references?
> >
> >> +Â Â Â }
> >> +
> >> +Â Â Â /* Set up guest foreign map. */
> >> +Â Â Â ret = guest_physmap_add_pages(bd, _gfn(b_gfn),
> >> page_to_mfn(pages[0]),
> >> +Â nr_gfns, t);
> >
> > A few things:
> > Â - The beginning of the code assumes that the MFN may be
> > discontiguous in the physical memory. But here, you are assuming they are
> contiguous.
> > If you want the latter, then you should check the MFNs are contiguous.
> > That said, I am not sure if this restriction is really necessary.
> >
> > Â - IIRC, guest_physmap_add_pages() doesn't revert the mappings. So
> > you need to revert it in case of failure.
>
>
> There is another issue here. guest_physmap_add_pages() may use superpage
> mapping. The P2M code is currently assuming the foreing mapping will be
> using L3 mapping (4KB).
>
> Do you need to use superpage mapping here?
>
Right now, there is no user case on my side needing superpage mapping.
> Cheers,
>
> --
> Julien Grall
RE: [PATCH v1 08/13] xen/arm: destroy static shared memory when de-construct domain
Hi, julien
> -Original Message-
> From: Julien Grall
> Sent: Saturday, April 9, 2022 5:26 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org
> Cc: nd ; Stefano Stabellini ; Bertrand
> Marquis ; Volodymyr Babchuk
>
> Subject: Re: [PATCH v1 08/13] xen/arm: destroy static shared memory when
> de-construct domain
>
> Hi Penny,
>
> On 11/03/2022 06:11, Penny Zheng wrote:
> > From: Penny Zheng
> >
> > This commit introduces a new helper destroy_domain_shm to destroy
> > static shared memory at domain de-construction.
> >
> > This patch only considers the scenario where the owner domain is the
> > default dom_shared, for user-defined owner domain, it will be covered
> > in the following patches.
> >
> > Since all domains are borrower domains, we could simply remove guest
> > P2M foreign mapping of statically shared memory region and drop the
> > reference added at guest_physmap_add_shm.
> >
> > Signed-off-by: Penny Zheng
> > ---
> > xen/arch/arm/domain.c | 48
> +++
> > 1 file changed, 48 insertions(+)
> >
> > diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index
> > 1ff1df5d3f..f0bfd67fe5 100644
> > --- a/xen/arch/arm/domain.c
> > +++ b/xen/arch/arm/domain.c
> > @@ -34,6 +34,7 @@
> > #include
> > #include
> > #include
> > +#include
> > #include
> > #include
> > #include
> > @@ -993,6 +994,48 @@ static int relinquish_memory(struct domain *d,
> struct page_list_head *list)
> > return ret;
> > }
> >
> > +#ifdef CONFIG_STATIC_SHM
> > +static int domain_destroy_shm(struct domain *d) {
> > +int ret = 0;
> > +unsigned long i = 0UL, j;
> > +
> > +if ( d->arch.shm_mem == NULL )
> > +return ret;
>
> You already return the value here. So...
>
> > +else
>
> ... the else is pointless.
>
> > +{
> > +for ( ; i < d->arch.shm_mem->nr_banks; i++ )
> > +{
> > +unsigned long nr_gfns = PFN_DOWN(d->arch.shm_mem-
> >bank[i].size);
> > +gfn_t gfn = gaddr_to_gfn(d->arch.shm_mem->bank[i].start);
> > +
> > +for ( j = 0; j < nr_gfns; j++ )
> > +{
> > +mfn_t mfn;
> > +
> > +mfn = gfn_to_mfn(d, gfn_add(gfn, j));
>
> A domain is allowed to modify its P2M. So there are no guarantee that the
> GFN will still point to the shared memory. This will allow the guest...
>
> > +if ( !mfn_valid(mfn) )
> > +{
> > +dprintk(XENLOG_ERR,
> > +"Domain %pd page number %lx invalid.\n",
> > +d, gfn_x(gfn) + i);
> > +return -EINVAL;
>
> ... to actively prevent destruction.
>
> > +}
>
>
> > +
> > +ret = guest_physmap_remove_page(d, gfn_add(gfn, j), mfn,
> > 0);
> > +if ( ret )
> > +return ret;
> > +
> > +/* Drop the reference. */
> > +put_page(mfn_to_page(mfn));
>
> guest_physmap_remove_page() will already drop the reference taken for the
> foreign mapping. I couldn't find any other reference taken, what is the
> put_page() for?
>
> Also, as per above we don't know whether this is a page from the shared page.
> So we can't blindly call put_page().
>
> However, I don't think we need any specific code here. We can rely on
> relinquish_p2m_mappings() to drop any reference. If there is an extra one for
> shared mappings, then we should update p2m_put_l3_page().
>
True, true. Thanks for pointing this out!
In p2m_put_l3_page, it has already called put_page() for foreign mapping,
definitely no extra one here!
> Cheers,
>
> --
> Julien Grall
RE: [PATCH v2 5/6] xen/arm: unpopulate memory when domain is static
Hi jan
> -Original Message-
> From: Jan Beulich
> Sent: Tuesday, April 19, 2022 5:11 PM
> To: Penny Zheng
> Cc: Wei Chen ; Stefano Stabellini
> ; Julien Grall ; Bertrand Marquis
> ; Volodymyr Babchuk
> ; Andrew Cooper
> ; George Dunlap ;
> Wei Liu ; xen-devel@lists.xenproject.org
> Subject: Re: [PATCH v2 5/6] xen/arm: unpopulate memory when domain is
> static
>
> On 18.04.2022 14:22, Penny Zheng wrote:
> > --- a/xen/arch/arm/include/asm/mm.h
> > +++ b/xen/arch/arm/include/asm/mm.h
> > @@ -358,6 +358,23 @@ void clear_and_clean_page(struct page_info
> > *page);
> >
> > unsigned int arch_get_dma_bitsize(void);
> >
> > +/*
> > + * Put free pages on the resv page list after having taken them
> > + * off the "normal" page list, when pages from static memory */
> > +#ifdef CONFIG_STATIC_MEMORY
> > +#define arch_free_heap_page(d, pg) {\
> > +if ( (pg)->count_info & PGC_reserved ) \
> > +{ \
> > +page_list_del(pg, page_to_list(d, pg)); \
> > +page_list_add_tail(pg, &(d)->resv_page_list); \
> > +(d)->resv_pages++; \
>
> There's no consumer of this counter, so I'd like to ask that it be introduced
> once a consumer appears.
>
> > +} \
> > +else\
> > +page_list_del(pg, page_to_list(d, pg)); \
>
> Is there a particular reason to have this page_list_del() twice, instead of
> just
> once ahead of the if()?
>
> > +}
>
> Also this entire construct want to be an expression, not a
> (compound) statement. And it probably would better evaluate its parameters
> just once.
>
#define arch_free_heap_page(d, pg) {\
page_list_del(pg, page_to_list(d, pg)); \
if ( (pg)->count_info & PGC_reserved ) \
page_list_add_tail(pg, &(d)->resv_page_list); \
}
I'm trying to refine the arch_free_heap_page() here, but I'm a bit confused
about to let it be an expression, not a compound statement. Do you mean that
you prefer to let the if-clause out of the arch_free_heap_page()?
> Jan
RE: [PATCH v2 5/6] xen/arm: unpopulate memory when domain is static
Hi, jan
> -Original Message-
> From: Jan Beulich
> Sent: Monday, April 25, 2022 4:01 PM
> To: Penny Zheng
> Cc: Wei Chen ; Stefano Stabellini
> ; Julien Grall ; Bertrand Marquis
> ; Volodymyr Babchuk
> ; Andrew Cooper
> ; George Dunlap ;
> Wei Liu ; xen-devel@lists.xenproject.org
> Subject: Re: [PATCH v2 5/6] xen/arm: unpopulate memory when domain is
> static
>
> On 25.04.2022 08:34, Penny Zheng wrote:
> >> -Original Message-
> >> From: Jan Beulich
> >> Sent: Tuesday, April 19, 2022 5:11 PM
> >>
> >> On 18.04.2022 14:22, Penny Zheng wrote:
> >>> --- a/xen/arch/arm/include/asm/mm.h
> >>> +++ b/xen/arch/arm/include/asm/mm.h
> >>> @@ -358,6 +358,23 @@ void clear_and_clean_page(struct page_info
> >>> *page);
> >>>
> >>> unsigned int arch_get_dma_bitsize(void);
> >>>
> >>> +/*
> >>> + * Put free pages on the resv page list after having taken them
> >>> + * off the "normal" page list, when pages from static memory */
> >>> +#ifdef CONFIG_STATIC_MEMORY
> >>> +#define arch_free_heap_page(d, pg) {\
> >>> +if ( (pg)->count_info & PGC_reserved ) \
> >>> +{ \
> >>> +page_list_del(pg, page_to_list(d, pg)); \
> >>> +page_list_add_tail(pg, &(d)->resv_page_list); \
> >>> +(d)->resv_pages++; \
> >>
> >> There's no consumer of this counter, so I'd like to ask that it be
> >> introduced once a consumer appears.
> >>
> >>> +} \
> >>> +else\
> >>> +page_list_del(pg, page_to_list(d, pg)); \
> >>
> >> Is there a particular reason to have this page_list_del() twice,
> >> instead of just once ahead of the if()?
> >>
> >>> +}
> >>
> >> Also this entire construct want to be an expression, not a
> >> (compound) statement. And it probably would better evaluate its
> >> parameters just once.
> >>
> >
> > #define arch_free_heap_page(d, pg) {\
> > page_list_del(pg, page_to_list(d, pg)); \
> > if ( (pg)->count_info & PGC_reserved ) \
> > page_list_add_tail(pg, &(d)->resv_page_list); \
> > }
> >
> > I'm trying to refine the arch_free_heap_page() here, but I'm a bit
> > confused about to let it be an expression, not a compound statement.
> > Do you mean that you prefer to let the if-clause out of the
> arch_free_heap_page()?
>
> No. You want to put parentheses around the braces, using a gcc extension we
> make extensive use of throughout the code base.
>
Oh, oh, thanks!
put parentheses around the braces, then that's what you said about make it
be an expression
> Jan
RE: [PATCH 03/11] xen/arm: introduce 1:1 direct-map for domUs
Hi Julien So sorry for taking so long to respond. Just being back from the long National Day holidays. 😉 > -Original Message- > From: Julien Grall > Sent: Thursday, September 23, 2021 6:36 PM > To: Penny Zheng ; xen-devel@lists.xenproject.org; > sstabell...@kernel.org > Cc: Bertrand Marquis ; Wei Chen > > Subject: Re: [PATCH 03/11] xen/arm: introduce 1:1 direct-map for domUs > > > > On 23/09/2021 08:11, Penny Zheng wrote: > > From: Stefano Stabellini > > > > Cases where domU needs 1:1 direct-map memory map: > > "1:1" and "direct-map" means pretty much the same. Given that the property > is name "direct-map", then I would drop "1:1". > > >* IOMMU not present in the system. > >* IOMMU disabled if it doesn't cover a specific device and all the > > guests are trusted. Thinking a mixed scenario, where a few devices > > with IOMMU and a few without, then guest DMA security still could not be > totally guaranteed. > > So users may want to disable the IOMMU, to at least gain some > > performance improvement from IOMMU disabled. > >* IOMMU disabled as a workaround when it doesn't have enough > bandwidth. > > To be specific, in a few extreme situation, when multiple devices do > > DMA concurrently, these requests may exceed IOMMU's transmission > capacity. > >* IOMMU disabled when it adds too much latency on DMA. For example, > > TLB may be missing in some IOMMU hardware, which may bring latency in > > DMA progress, so users may want to disable it in some realtime scenario. > > > > *WARNING: > > Users should be aware that it is not always secure to assign a device > > without IOMMU protection. > > When the device is not protected by the IOMMU, the administrator > > should make sure that: > > 1. The device is assigned to a trusted guest. > > 2. Users have additional security mechanism on the platform. > > The two requirements are only necessary for device that are DMA-capable. > For device that can't do DMA, it will likely be fine to assign to non-trusted > guest. > > > > > Given that with direct-map, the IOMMU could be used but it is not required. > > I can't parse it. > Now when doing device assignments, IOMMU is forced to be enabled. And since we are introducing direct-map here, I think that maybe even if IOMMU is missing/disabled, direct-map on trust guests could also make it work. Maybe I should rephrase it to " When doing device assignments and IOMMU is missing or disabled, direct-map shall be used on trust guests. " > > This commit avoids setting XEN_DOMCTL_CDF_iommu when the IOMMU is > > disabled and direct_map is requested. > > > > Since, for now, 1:1 direct-map is only supported when domain on Static > > I think "Since" seems unnecessary. > > > Allocation, that is, "xen.static-mem" is defined in the domain > > configuration. > > > > This commit also re-implements allocate_static_memory to allocate > > static memory as guest RAM for 1:1 direct-map domain. > > > > Signed-off-by: Stefano Stabellini > > Signed-off-by: Penny Zheng > > --- > > docs/misc/arm/device-tree/booting.txt | 9 ++ > > xen/arch/arm/domain_build.c | 117 +- > > 2 files changed, 85 insertions(+), 41 deletions(-) > > > > diff --git a/docs/misc/arm/device-tree/booting.txt > > b/docs/misc/arm/device-tree/booting.txt > > index 44cd9e1a9a..3d637c747e 100644 > > --- a/docs/misc/arm/device-tree/booting.txt > > +++ b/docs/misc/arm/device-tree/booting.txt > > @@ -164,6 +164,15 @@ with the following properties: > > Both #address-cells and #size-cells need to be specified because > > both sub-nodes (described shortly) have reg properties. > > > > +- direct-map > > + > > +Optional for Domain on Static Allocation. > > +An empty property to request the memory of the domain to be 1:1 > > +direct-map(guest physical address == physical address). > > +WARNING: Users must be aware of this risk, that guests having access > > +to hardware with DMA capacity must be trusted, or it could use the > > +DMA engine to access any other memory area. > > The WARNING is only applicable if the device is not protected by an IOMMU. > So this should be clarified because one may want still want to use the direct- > map (e.g. because the OS relies on the host layout) and have IOMMU enabled. > > > + > > Under the "xen,domain" compatible node, one or more sub-nodes ar
RE: [PATCH 09/11] xen/arm: if 1:1 direct-map domain use native UART address and IRQ number for vPL011
Hi Julien
> -Original Message-
> From: Julien Grall
> Sent: Thursday, September 23, 2021 7:14 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Bertrand Marquis ; Wei Chen
>
> Subject: Re: [PATCH 09/11] xen/arm: if 1:1 direct-map domain use native
> UART address and IRQ number for vPL011
>
>
>
> On 23/09/2021 08:11, Penny Zheng wrote:
> > From: Stefano Stabellini
> >
> > We always use a fix address to map the vPL011 to domains. The address
> > could be a problem for domains that are directly mapped.
> >
> > So, for domains that are directly mapped, reuse the address of the
> > physical UART on the platform to avoid potential clashes.
> >
> > Do the same for the virtual IRQ number: instead of always using
> > GUEST_VPL011_SPI, try to reuse the physical SPI number if possible.
> >
> > Signed-off-by: Stefano Stabellini
> > Signed-off-by: Penny Zheng
> > ---
> > xen/arch/arm/domain_build.c | 34 +++---
> > xen/arch/arm/vpl011.c| 34 +++---
> > xen/include/asm-arm/vpl011.h | 2 ++
> > 3 files changed, 56 insertions(+), 14 deletions(-)
> >
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index 120f1ae575..c92e510ae7 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -30,6 +30,7 @@
> >
> > #include
> > #include
> > +#include
> >
> > static unsigned int __initdata opt_dom0_max_vcpus;
> > integer_param("dom0_max_vcpus", opt_dom0_max_vcpus); @@ -1942,8
> > +1943,11 @@ static int __init make_vpl011_uart_node(struct kernel_info
> *kinfo)
> > gic_interrupt_t intr;
> > __be32 reg[GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS];
> > __be32 *cells;
> > +struct domain *d = kinfo->d;
> > +char buf[27];
> >
> > -res = fdt_begin_node(fdt, "sbsa-uart@"__stringify(GUEST_PL011_BASE));
> > +snprintf(buf, sizeof(buf), "sbsa-uart@%"PRIx64, d-
> >arch.vpl011.base_addr);
> > +res = fdt_begin_node(fdt, buf);
> > if ( res )
> > return res;
> >
> > @@ -1953,14 +1957,14 @@ static int __init make_vpl011_uart_node(struct
> > kernel_info *kinfo)
> >
> > cells = ®[0];
> > dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS,
> > - GUEST_ROOT_SIZE_CELLS, GUEST_PL011_BASE,
> > + GUEST_ROOT_SIZE_CELLS,
> > + d->arch.vpl011.base_addr,
> > GUEST_PL011_SIZE);
> >
> > res = fdt_property(fdt, "reg", reg, sizeof(reg));
> > if ( res )
> > return res;
> >
> > -set_interrupt(intr, GUEST_VPL011_SPI, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
> > +set_interrupt(intr, d->arch.vpl011.virq, 0xf,
> > + DT_IRQ_TYPE_LEVEL_HIGH);
> >
> > res = fdt_property(fdt, "interrupts", intr, sizeof (intr));
> > if ( res )
> > @@ -2670,6 +2674,13 @@ static int __init construct_domU(struct domain
> *d,
> > else
> > allocate_static_memory(d, &kinfo, node);
> >
> > +/*
> > + * Initialization before creating its device
> > + * tree node in prepare_dtb_domU.
> > + */
>
> I think it would be better to explain *why* this needs to be done before.
>
> > +if ( kinfo.vpl011 )
> > +rc = domain_vpl011_init(d, NULL);
> > +
> > rc = prepare_dtb_domU(d, &kinfo);
> > if ( rc < 0 )
> > return rc;
> > @@ -2678,9 +2689,6 @@ static int __init construct_domU(struct domain
> *d,
> > if ( rc < 0 )
> > return rc;
> >
> > -if ( kinfo.vpl011 )
> > -rc = domain_vpl011_init(d, NULL);
> > -
> > return rc;
> > }
> >
> > @@ -2723,15 +2731,27 @@ void __init create_domUs(void)
> >
> > if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
> > {
> > +unsigned int vpl011_virq = GUEST_VPL011_SPI;
>
> Coding style: Add a newline here.
>
> > d_cfg.arch.nr_spis = gic_number_lines() - 32;
> >
> > +/*
> > + * The VPL011 virq is GUEST_VPL011_SPI, unless direct-map in
> > + * set, in which case we'll try to match the hardware.
> > + *
> > + * Typically, d-&
RE: [PATCH 10/11] xen/arm: device assignment on 1:1 direct-map domain
Hi Julien > -Original Message- > From: Julien Grall > Sent: Thursday, September 23, 2021 7:27 PM > To: Penny Zheng ; xen-devel@lists.xenproject.org; > sstabell...@kernel.org > Cc: Bertrand Marquis ; Wei Chen > > Subject: Re: [PATCH 10/11] xen/arm: device assignment on 1:1 direct-map > domain > > Hi, > > On 23/09/2021 08:11, Penny Zheng wrote: > > User could do device passthrough, with > > "xen,force-assign-without-iommu" in the device tree snippet, on > > trusted guest through 1:1 direct-map, if IOMMU absent or disabled on > hardware. > > At the moment, it would be possible to passthrough a non-DMA capable > device with direct-mapping. After this patch, this is going to be forbidden. > > > > > In order to achieve that, this patch adds 1:1 direct-map check and > > disables iommu-related action. > > > > Signed-off-by: Penny Zheng > > --- > > xen/arch/arm/domain_build.c | 12 > > 1 file changed, 8 insertions(+), 4 deletions(-) > > > > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c > > index c92e510ae7..9a9d2522b7 100644 > > --- a/xen/arch/arm/domain_build.c > > +++ b/xen/arch/arm/domain_build.c > > @@ -2070,14 +2070,18 @@ static int __init > handle_passthrough_prop(struct kernel_info *kinfo, > > if ( res < 0 ) > > return res; > > > > +/* > > + * If xen_force, we allow assignment of devices without IOMMU > protection. > > + * And if IOMMU is disabled or absent, 1:1 direct-map is necessary > + > */ > > +if ( xen_force && is_domain_direct_mapped(kinfo->d) && > > + !dt_device_is_protected(node) ) > > dt_device_is_protected() will be always false unless the device is protected > behing an SMMU using the legacy binding. So I don't think this is correct to > move this check ahead. In fact.. > > > +return 0; > > + > > res = iommu_add_dt_device(node); > > ... the call should already be a NOP when the IOMMU is disabled or the > device is not behind an IOMMU. So can you explain what you are trying to > prevent here? > If the IOMMU is disabled, iommu_add_dt_device will return 1 as errno. So we could not make it to the xen_force check... So I tried to move all IOMMU action behind xen_force check. Now, device assignment without IOMMU protection is only applicable on direct-map domains, so this commit also adds is_domain_direct_mapped check together with xen_force check. > > if ( res < 0 ) > > return res; > > > > -/* If xen_force, we allow assignment of devices without IOMMU > protection. */ > > -if ( xen_force && !dt_device_is_protected(node) ) > > -return 0; > > - > > return iommu_assign_dt_device(kinfo->d, node); > > } > > > > > > Cheers, > > -- > Julien Grall
RE: [PATCH 10/11] xen/arm: device assignment on 1:1 direct-map domain
Hi julien
> -Original Message-
> From: Julien Grall
> Sent: Monday, October 11, 2021 7:14 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Bertrand Marquis ; Wei Chen
>
> Subject: Re: [PATCH 10/11] xen/arm: device assignment on 1:1 direct-map
> domain
>
>
>
> On 09/10/2021 10:40, Penny Zheng wrote:
> > Hi Julien
>
> Hi Penny,
>
> >
> >> -Original Message-
> >> From: Julien Grall
> >> Sent: Thursday, September 23, 2021 7:27 PM
> >> To: Penny Zheng ;
> >> xen-devel@lists.xenproject.org; sstabell...@kernel.org
> >> Cc: Bertrand Marquis ; Wei Chen
> >>
> >> Subject: Re: [PATCH 10/11] xen/arm: device assignment on 1:1
> >> direct-map domain
> >>
> >> Hi,
> >>
> >> On 23/09/2021 08:11, Penny Zheng wrote:
> >>> User could do device passthrough, with
> >>> "xen,force-assign-without-iommu" in the device tree snippet, on
> >>> trusted guest through 1:1 direct-map, if IOMMU absent or disabled on
> >> hardware.
> >>
> >> At the moment, it would be possible to passthrough a non-DMA capable
> >> device with direct-mapping. After this patch, this is going to be
> >> forbidden.
> >>
> >>>
> >>> In order to achieve that, this patch adds 1:1 direct-map check and
> >>> disables iommu-related action.
> >>>
> >>> Signed-off-by: Penny Zheng
> >>> ---
> >>>xen/arch/arm/domain_build.c | 12
> >>>1 file changed, 8 insertions(+), 4 deletions(-)
> >>>
> >>> diff --git a/xen/arch/arm/domain_build.c
> >>> b/xen/arch/arm/domain_build.c index c92e510ae7..9a9d2522b7 100644
> >>> --- a/xen/arch/arm/domain_build.c
> >>> +++ b/xen/arch/arm/domain_build.c
> >>> @@ -2070,14 +2070,18 @@ static int __init
> >> handle_passthrough_prop(struct kernel_info *kinfo,
> >>>if ( res < 0 )
> >>>return res;
> >>>
> >>> +/*
> >>> + * If xen_force, we allow assignment of devices without IOMMU
> >> protection.
> >>> + * And if IOMMU is disabled or absent, 1:1 direct-map is
> >>> + necessary > +
> >> */
> >>> +if ( xen_force && is_domain_direct_mapped(kinfo->d) &&
> >>> + !dt_device_is_protected(node) )
> >>
> >> dt_device_is_protected() will be always false unless the device is
> >> protected behing an SMMU using the legacy binding. So I don't think
> >> this is correct to move this check ahead. In fact..
> >>
> >>> +return 0;
> >>> +
> >>>res = iommu_add_dt_device(node);
> >>
> >> ... the call should already be a NOP when the IOMMU is disabled or
> >> the device is not behind an IOMMU. So can you explain what you are
> >> trying to prevent here?
> >>
> >
> > If the IOMMU is disabled, iommu_add_dt_device will return 1 as errno.
> > So we could not make it to the xen_force check...
>
> I disagree. The check is:
>
> if ( res < 0 )
>return res;
>
> Given that res is 1, we wouldn't return and move to check whether the
> assignment can be done.
>
> >
> > So I tried to move all IOMMU action behind xen_force check.
> >
> > Now, device assignment without IOMMU protection is only applicable on
> > direct-map domains,
>
> It is fine to assign a non-DMA capable device without direct-mapping. So why
> do you want to add this restriction?
I understand.
If it is fine to assign a non-DMA capable device without direct-mapping, the
former
commit containing the following changes needs fix.
if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
{
if ( iommu_enabled )
d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
else if ( d_cfg.flags & XEN_DOMCTL_INTERNAL_directmap )
warning_add("Please be sure of having trusted guests, when
doing device assignment without IOMMU protection\n");
else
panic("Assign a device but IOMMU and direct-map are all
disabled\n");
}
Definitely no panic when IOMMU and direct-map are all disabled.
>
> Cheers,
>
> --
> Julien Grall
RE: [PATCH 09/11] xen/arm: if 1:1 direct-map domain use native UART address and IRQ number for vPL011
Hi Julien
> -Original Message-
> From: Julien Grall
> Sent: Monday, October 11, 2021 6:49 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Bertrand Marquis ; Wei Chen
>
> Subject: Re: [PATCH 09/11] xen/arm: if 1:1 direct-map domain use native UART
> address and IRQ number for vPL011
>
> On 09/10/2021 09:47, Penny Zheng wrote:
> > Hi Julien
>
> Hi Penny,
>
> >> -Original Message-
> >> From: Julien Grall
> >> Sent: Thursday, September 23, 2021 7:14 PM
> >> To: Penny Zheng ;
> >> xen-devel@lists.xenproject.org; sstabell...@kernel.org
> >> Cc: Bertrand Marquis ; Wei Chen
> >>
> >> Subject: Re: [PATCH 09/11] xen/arm: if 1:1 direct-map domain use
> >> native UART address and IRQ number for vPL011
> >>
> >>
> >>
> >> On 23/09/2021 08:11, Penny Zheng wrote:
> >>> From: Stefano Stabellini
> >>>
> >>> We always use a fix address to map the vPL011 to domains. The
> >>> address could be a problem for domains that are directly mapped.
> >>>
> >>> So, for domains that are directly mapped, reuse the address of the
> >>> physical UART on the platform to avoid potential clashes.
> >>>
> >>> Do the same for the virtual IRQ number: instead of always using
> >>> GUEST_VPL011_SPI, try to reuse the physical SPI number if possible.
> >>>
> >>> Signed-off-by: Stefano Stabellini
> >>> Signed-off-by: Penny Zheng
> >>> ---
> >>>xen/arch/arm/domain_build.c | 34 +++-
> --
> >>>xen/arch/arm/vpl011.c| 34 +++---
> >>>xen/include/asm-arm/vpl011.h | 2 ++
> >>>3 files changed, 56 insertions(+), 14 deletions(-)
> >>>
> >>> diff --git a/xen/arch/arm/domain_build.c
> >>> b/xen/arch/arm/domain_build.c index 120f1ae575..c92e510ae7 100644
> >>> --- a/xen/arch/arm/domain_build.c
> >>> +++ b/xen/arch/arm/domain_build.c
> >>> @@ -30,6 +30,7 @@
> >>>
> >>>#include
> >>>#include
> >>> +#include
> >>>
> >>>static unsigned int __initdata opt_dom0_max_vcpus;
> >>>integer_param("dom0_max_vcpus", opt_dom0_max_vcpus); @@ -
> 1942,8
> >>> +1943,11 @@ static int __init make_vpl011_uart_node(struct
> >>> +kernel_info
> >> *kinfo)
> >>>gic_interrupt_t intr;
> >>>__be32 reg[GUEST_ROOT_ADDRESS_CELLS +
> GUEST_ROOT_SIZE_CELLS];
> >>>__be32 *cells;
> >>> +struct domain *d = kinfo->d;
> >>> +char buf[27];
> >>>
> >>> -res = fdt_begin_node(fdt, "sbsa-
> uart@"__stringify(GUEST_PL011_BASE));
> >>> +snprintf(buf, sizeof(buf), "sbsa-uart@%"PRIx64, d-
> >>> arch.vpl011.base_addr);
> >>> +res = fdt_begin_node(fdt, buf);
> >>>if ( res )
> >>>return res;
> >>>
> >>> @@ -1953,14 +1957,14 @@ static int __init
> >>> make_vpl011_uart_node(struct kernel_info *kinfo)
> >>>
> >>>cells = ®[0];
> >>>dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS,
> >>> - GUEST_ROOT_SIZE_CELLS, GUEST_PL011_BASE,
> >>> + GUEST_ROOT_SIZE_CELLS,
> >>> + d->arch.vpl011.base_addr,
> >>> GUEST_PL011_SIZE);
> >>>
> >>>res = fdt_property(fdt, "reg", reg, sizeof(reg));
> >>>if ( res )
> >>>return res;
> >>>
> >>> -set_interrupt(intr, GUEST_VPL011_SPI, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
> >>> +set_interrupt(intr, d->arch.vpl011.virq, 0xf,
> >>> + DT_IRQ_TYPE_LEVEL_HIGH);
> >>>
> >>>res = fdt_property(fdt, "interrupts", intr, sizeof (intr));
> >>>if ( res )
> >>> @@ -2670,6 +2674,13 @@ static int __init construct_domU(struct
> >>> domain
> >> *d,
> >>>else
> >>>allocate_static_memory(d, &kinfo, node);
> >>>
> >>> +/*
> >>> + * Initialization before creating its device
> >>> + * tree node in prepare_dtb_domU.
> >>> + */
> >>
> >> I think it would be better t
RE: [PATCH 10/11] xen/arm: device assignment on 1:1 direct-map domain
Hi Julien
> -Original Message-
> From: Julien Grall
> Sent: Monday, October 11, 2021 7:14 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Bertrand Marquis ; Wei Chen
>
> Subject: Re: [PATCH 10/11] xen/arm: device assignment on 1:1 direct-map
> domain
>
>
>
> On 09/10/2021 10:40, Penny Zheng wrote:
> > Hi Julien
>
> Hi Penny,
>
> >
> >> -Original Message-
> >> From: Julien Grall
> >> Sent: Thursday, September 23, 2021 7:27 PM
> >> To: Penny Zheng ;
> >> xen-devel@lists.xenproject.org; sstabell...@kernel.org
> >> Cc: Bertrand Marquis ; Wei Chen
> >>
> >> Subject: Re: [PATCH 10/11] xen/arm: device assignment on 1:1
> >> direct-map domain
> >>
> >> Hi,
> >>
> >> On 23/09/2021 08:11, Penny Zheng wrote:
> >>> User could do device passthrough, with
> >>> "xen,force-assign-without-iommu" in the device tree snippet, on
> >>> trusted guest through 1:1 direct-map, if IOMMU absent or disabled on
> >> hardware.
> >>
> >> At the moment, it would be possible to passthrough a non-DMA capable
> >> device with direct-mapping. After this patch, this is going to be
> >> forbidden.
> >>
> >>>
> >>> In order to achieve that, this patch adds 1:1 direct-map check and
> >>> disables iommu-related action.
> >>>
> >>> Signed-off-by: Penny Zheng
> >>> ---
> >>>xen/arch/arm/domain_build.c | 12
> >>>1 file changed, 8 insertions(+), 4 deletions(-)
> >>>
> >>> diff --git a/xen/arch/arm/domain_build.c
> >>> b/xen/arch/arm/domain_build.c index c92e510ae7..9a9d2522b7 100644
> >>> --- a/xen/arch/arm/domain_build.c
> >>> +++ b/xen/arch/arm/domain_build.c
> >>> @@ -2070,14 +2070,18 @@ static int __init
> >> handle_passthrough_prop(struct kernel_info *kinfo,
> >>>if ( res < 0 )
> >>>return res;
> >>>
> >>> +/*
> >>> + * If xen_force, we allow assignment of devices without IOMMU
> >> protection.
> >>> + * And if IOMMU is disabled or absent, 1:1 direct-map is
> >>> + necessary > +
> >> */
> >>> +if ( xen_force && is_domain_direct_mapped(kinfo->d) &&
> >>> + !dt_device_is_protected(node) )
> >>
> >> dt_device_is_protected() will be always false unless the device is
> >> protected behing an SMMU using the legacy binding. So I don't think
> >> this is correct to move this check ahead. In fact..
> >>
> >>> +return 0;
> >>> +
> >>>res = iommu_add_dt_device(node);
> >>
> >> ... the call should already be a NOP when the IOMMU is disabled or
> >> the device is not behind an IOMMU. So can you explain what you are
> >> trying to prevent here?
> >>
> >
> > If the IOMMU is disabled, iommu_add_dt_device will return 1 as errno.
> > So we could not make it to the xen_force check...
>
> I disagree. The check is:
>
> if ( res < 0 )
>return res;
>
> Given that res is 1, we wouldn't return and move to check whether the
> assignment can be done.
>
> >
> > So I tried to move all IOMMU action behind xen_force check.
> >
> > Now, device assignment without IOMMU protection is only applicable on
> > direct-map domains,
>
> It is fine to assign a non-DMA capable device without direct-mapping. So why
> do you want to add this restriction?
>
When constructing direct-map-v2, found that, in xen/arch/arm/domain_build.c
if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
And this flag XEN_DOMCTL_CDF_iommu determines whether iommu is enabled.
In ./xen/include/xen/sched.h
static always_inline bool is_iommu_enabled(const struct domain *d)
{
return evaluate_nospec(d->options & XEN_DOMCTL_CDF_iommu);
}
That is, even if we assign a non-DMA capable device, we request the platform to
be
iommu enabled.
> Cheers,
>
> --
> Julien Grall
RE: [PATCH 10/11] xen/arm: device assignment on 1:1 direct-map domain
> -Original Message-
> From: Penny Zheng
> Sent: Wednesday, October 13, 2021 3:44 PM
> To: Julien Grall ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Bertrand Marquis ; Wei Chen
>
> Subject: RE: [PATCH 10/11] xen/arm: device assignment on 1:1 direct-map
> domain
>
> Hi Julien
>
> > -Original Message-
> > From: Julien Grall
> > Sent: Monday, October 11, 2021 7:14 PM
> > To: Penny Zheng ; xen-devel@lists.xenproject.org;
> > sstabell...@kernel.org
> > Cc: Bertrand Marquis ; Wei Chen
> >
> > Subject: Re: [PATCH 10/11] xen/arm: device assignment on 1:1
> > direct-map domain
> >
> >
> >
> > On 09/10/2021 10:40, Penny Zheng wrote:
> > > Hi Julien
> >
> > Hi Penny,
> >
> > >
> > >> -Original Message-
> > >> From: Julien Grall
> > >> Sent: Thursday, September 23, 2021 7:27 PM
> > >> To: Penny Zheng ;
> > >> xen-devel@lists.xenproject.org; sstabell...@kernel.org
> > >> Cc: Bertrand Marquis ; Wei Chen
> > >>
> > >> Subject: Re: [PATCH 10/11] xen/arm: device assignment on 1:1
> > >> direct-map domain
> > >>
> > >> Hi,
> > >>
> > >> On 23/09/2021 08:11, Penny Zheng wrote:
> > >>> User could do device passthrough, with
> > >>> "xen,force-assign-without-iommu" in the device tree snippet, on
> > >>> trusted guest through 1:1 direct-map, if IOMMU absent or disabled
> > >>> on
> > >> hardware.
> > >>
> > >> At the moment, it would be possible to passthrough a non-DMA
> > >> capable device with direct-mapping. After this patch, this is going to be
> forbidden.
> > >>
> > >>>
> > >>> In order to achieve that, this patch adds 1:1 direct-map check and
> > >>> disables iommu-related action.
> > >>>
> > >>> Signed-off-by: Penny Zheng
> > >>> ---
> > >>>xen/arch/arm/domain_build.c | 12
> > >>>1 file changed, 8 insertions(+), 4 deletions(-)
> > >>>
> > >>> diff --git a/xen/arch/arm/domain_build.c
> > >>> b/xen/arch/arm/domain_build.c index c92e510ae7..9a9d2522b7 100644
> > >>> --- a/xen/arch/arm/domain_build.c
> > >>> +++ b/xen/arch/arm/domain_build.c
> > >>> @@ -2070,14 +2070,18 @@ static int __init
> > >> handle_passthrough_prop(struct kernel_info *kinfo,
> > >>>if ( res < 0 )
> > >>>return res;
> > >>>
> > >>> +/*
> > >>> + * If xen_force, we allow assignment of devices without IOMMU
> > >> protection.
> > >>> + * And if IOMMU is disabled or absent, 1:1 direct-map is
> > >>> + necessary > +
> > >> */
> > >>> +if ( xen_force && is_domain_direct_mapped(kinfo->d) &&
> > >>> + !dt_device_is_protected(node) )
> > >>
> > >> dt_device_is_protected() will be always false unless the device is
> > >> protected behing an SMMU using the legacy binding. So I don't think
> > >> this is correct to move this check ahead. In fact..
> > >>
> > >>> +return 0;
> > >>> +
> > >>>res = iommu_add_dt_device(node);
> > >>
> > >> ... the call should already be a NOP when the IOMMU is disabled or
> > >> the device is not behind an IOMMU. So can you explain what you are
> > >> trying to prevent here?
> > >>
> > >
> > > If the IOMMU is disabled, iommu_add_dt_device will return 1 as errno.
> > > So we could not make it to the xen_force check...
> >
> > I disagree. The check is:
> >
> > if ( res < 0 )
> >return res;
> >
> > Given that res is 1, we wouldn't return and move to check whether the
> > assignment can be done.
> >
> > >
> > > So I tried to move all IOMMU action behind xen_force check.
> > >
> > > Now, device assignment without IOMMU protection is only applicable
> > > on direct-map domains,
> >
> > It is fine to assign a non-DMA capable device without direct-mapping.
> > So why do you want to add this restriction?
> >
>
> When constructing direct-map-v2, found that, in
> xen/arch/arm/domain_build.c
>
> if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
> d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
>
> And this flag XEN_DOMCTL_CDF_iommu determines whether iommu is
> enabled.
>
> In ./xen/include/xen/sched.h
>
> static always_inline bool is_iommu_enabled(const struct domain *d) {
> return evaluate_nospec(d->options & XEN_DOMCTL_CDF_iommu); }
>
> That is, even if we assign a non-DMA capable device, we request the platform
> to be iommu enabled.
>
I intend to change it to
if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
{
if ( iommu_enabled )
d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
else if ( d_cfg.flags & XEN_DOMCTL_CDF_directmap )
warning_add("Please be sure of having trusted guests, when
doing device assignment without IOMMU protection\n");
}
> > Cheers,
> >
> > --
> > Julien Grall
RE: [PATCH 09/11] xen/arm: if 1:1 direct-map domain use native UART address and IRQ number for vPL011
Hi Julien
> -Original Message-
> From: Julien Grall
> Sent: Thursday, October 14, 2021 2:01 AM
> To: Penny Zheng ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Bertrand Marquis ; Wei Chen
>
> Subject: Re: [PATCH 09/11] xen/arm: if 1:1 direct-map domain use native UART
> address and IRQ number for vPL011
>
> On 12/10/2021 03:42, Penny Zheng wrote:
> > Hi Julien
>
> Hi Penny,
>
> >> -Original Message-
> >> From: Julien Grall
> >> Sent: Monday, October 11, 2021 6:49 PM
> >> To: Penny Zheng ;
> >> xen-devel@lists.xenproject.org; sstabell...@kernel.org
> >> Cc: Bertrand Marquis ; Wei Chen
> >>
> >> Subject: Re: [PATCH 09/11] xen/arm: if 1:1 direct-map domain use
> >> native UART address and IRQ number for vPL011
> >>
> >> On 09/10/2021 09:47, Penny Zheng wrote:
> >>> Hi Julien
> >>
> >> Hi Penny,
> >>
> >>>> -Original Message-
> >>>> From: Julien Grall
> >>>> Sent: Thursday, September 23, 2021 7:14 PM
> >>>> To: Penny Zheng ;
> >>>> xen-devel@lists.xenproject.org; sstabell...@kernel.org
> >>>> Cc: Bertrand Marquis ; Wei Chen
> >>>>
> >>>> Subject: Re: [PATCH 09/11] xen/arm: if 1:1 direct-map domain use
> >>>> native UART address and IRQ number for vPL011
> >>>>
> >>>>
> >>>>
> >>>> On 23/09/2021 08:11, Penny Zheng wrote:
> >>>>> From: Stefano Stabellini
> >>>>>
> >>>>> We always use a fix address to map the vPL011 to domains. The
> >>>>> address could be a problem for domains that are directly mapped.
> >>>>>
> >>>>> So, for domains that are directly mapped, reuse the address of the
> >>>>> physical UART on the platform to avoid potential clashes.
> >>>>>
> >>>>> Do the same for the virtual IRQ number: instead of always using
> >>>>> GUEST_VPL011_SPI, try to reuse the physical SPI number if possible.
> >>>>>
> >>>>> Signed-off-by: Stefano Stabellini
> >>>>> Signed-off-by: Penny Zheng
> >>>>> ---
> >>>>> xen/arch/arm/domain_build.c | 34
> >>>>> +++-
> >> --
> >>>>> xen/arch/arm/vpl011.c| 34 +++---
> >>>>> xen/include/asm-arm/vpl011.h | 2 ++
> >>>>> 3 files changed, 56 insertions(+), 14 deletions(-)
> >>>>>
> >>>>> diff --git a/xen/arch/arm/domain_build.c
> >>>>> b/xen/arch/arm/domain_build.c index 120f1ae575..c92e510ae7 100644
> >>>>> --- a/xen/arch/arm/domain_build.c
> >>>>> +++ b/xen/arch/arm/domain_build.c
> >>>>> @@ -30,6 +30,7 @@
> >>>>>
> >>>>> #include
> >>>>> #include
> >>>>> +#include
> >>>>>
> >>>>> static unsigned int __initdata opt_dom0_max_vcpus;
> >>>>> integer_param("dom0_max_vcpus", opt_dom0_max_vcpus); @@ -
> >> 1942,8
> >>>>> +1943,11 @@ static int __init make_vpl011_uart_node(struct
> >>>>> +kernel_info
> >>>> *kinfo)
> >>>>> gic_interrupt_t intr;
> >>>>> __be32 reg[GUEST_ROOT_ADDRESS_CELLS +
> >> GUEST_ROOT_SIZE_CELLS];
> >>>>> __be32 *cells;
> >>>>> +struct domain *d = kinfo->d;
> >>>>> +char buf[27];
> >>>>>
> >>>>> -res = fdt_begin_node(fdt, "sbsa-
> >> uart@"__stringify(GUEST_PL011_BASE));
> >>>>> +snprintf(buf, sizeof(buf), "sbsa-uart@%"PRIx64, d-
> >>>>> arch.vpl011.base_addr);
> >>>>> +res = fdt_begin_node(fdt, buf);
> >>>>> if ( res )
> >>>>> return res;
> >>>>>
> >>>>> @@ -1953,14 +1957,14 @@ static int __init
> >>>>> make_vpl011_uart_node(struct kernel_info *kinfo)
> >>>>>
> >>>>> cells = ®[0];
> >>>>> dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS,
> >>>>> - GUEST_ROOT_SIZE_CELLS, GUEST_PL011_BASE,
> >>>>&
[PATCH v2 1/6] xen: introduce XEN_DOMCTL_CDF_directmap
From: Stefano Stabellini
This commit introduces a new arm-specific flag XEN_DOMCTL_CDF_directmap to
specify that this domain should have its memory directly mapped
(guest physical address == physical address) at domain creation.
Refine is_domain_direct_mapped to check whether the flag
XEN_DOMCTL_CDF_directmap is set.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
---
CC: andrew.coop...@citrix.com
CC: jbeul...@suse.com
CC: George Dunlap
CC: Ian Jackson
CC: Wei Liu
CC: "Roger Pau Monné"
---
xen/arch/arm/domain.c| 3 ++-
xen/arch/arm/domain_build.c | 4 +++-
xen/common/domain.c | 3 ++-
xen/include/asm-arm/domain.h | 4 ++--
xen/include/public/domctl.h | 4 +++-
5 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index eef0661beb..8cee1c6349 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -628,7 +628,8 @@ int arch_sanitise_domain_config(struct
xen_domctl_createdomain *config)
{
unsigned int max_vcpus;
unsigned int flags_required = (XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap);
-unsigned int flags_optional = (XEN_DOMCTL_CDF_iommu | XEN_DOMCTL_CDF_vpmu);
+unsigned int flags_optional = (XEN_DOMCTL_CDF_iommu | XEN_DOMCTL_CDF_vpmu |
+ XEN_DOMCTL_CDF_directmap);
if ( (config->flags & ~flags_optional) != flags_required )
{
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 0167731ab0..37e2d62d47 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3069,8 +3069,10 @@ static int __init construct_dom0(struct domain *d)
void __init create_dom0(void)
{
struct domain *dom0;
+/* DOM0 has always its memory directly mapped. */
struct xen_domctl_createdomain dom0_cfg = {
-.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,
+.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
+ XEN_DOMCTL_CDF_directmap,
.max_evtchn_port = -1,
.max_grant_frames = gnttab_dom0_frames(),
.max_maptrack_frames = -1,
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 8b53c49d1e..7a6131db74 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -486,7 +486,8 @@ static int sanitise_domain_config(struct
xen_domctl_createdomain *config)
~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off |
XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu |
- XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu) )
+ XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu |
+ XEN_DOMCTL_CDF_directmap) )
{
dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
return -EINVAL;
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 14e575288f..fc42c6a310 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -29,8 +29,8 @@ enum domain_type {
#define is_64bit_domain(d) (0)
#endif
-/* The hardware domain has always its memory direct mapped. */
-#define is_domain_direct_mapped(d) is_hardware_domain(d)
+/* Check if domain is direct-map memory map. */
+#define is_domain_direct_mapped(d) (d->options & XEN_DOMCTL_CDF_directmap)
struct vtimer {
struct vcpu *v;
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 238384b5ae..b505a0db51 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -72,9 +72,11 @@ struct xen_domctl_createdomain {
#define XEN_DOMCTL_CDF_nested_virt(1U << _XEN_DOMCTL_CDF_nested_virt)
/* Should we expose the vPMU to the guest? */
#define XEN_DOMCTL_CDF_vpmu (1U << 7)
+/* If this domain has its memory directly mapped? (ARM only) */
+#define XEN_DOMCTL_CDF_directmap (1U << 8)
/* Max XEN_DOMCTL_CDF_* constant. Used for ABI checking. */
-#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_vpmu
+#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_directmap
uint32_t flags;
--
2.25.1
[PATCH v2 0/6] direct-map memory map
Cases where domU needs direct-map memory map: * IOMMU not present in the system. * IOMMU disabled if it doesn't cover a specific device and all the guests are trusted. Thinking a mixed scenario, where a few devices with IOMMU and a few without, then guest DMA security still could not be totally guaranteed. So users may want to disable the IOMMU, to at least gain some performance improvement from IOMMU disabled. * IOMMU disabled as a workaround when it doesn't have enough bandwidth. To be specific, in a few extreme situation, when multiple devices do DMA concurrently, these requests may exceed IOMMU's transmission capacity. * IOMMU disabled when it adds too much latency on DMA. For example, TLB may be missing in some IOMMU hardware, which may bring latency in DMA progress, so users may want to disable it in some realtime scenario. *WARNING: Users should be aware that it is not always secure to assign a DMA-capable device without IOMMU protection. When the device is not protected by the IOMMU, the administrator should make sure that: 1. The device is assigned to a trusted guest. 2. Users have additional security mechanism on the platform. Requesting direct-map memory mapping for the domain, when IOMMU is absent from the system or it is disabled (status = "disabled" in device tree), In which case, "direct-map" property is added under the appropriate domain node. Right now, direct-map is only supported when domain on Static Allocation, that is, "xen,static-mem" is also necessary in the domain configuration. Looking into related [design link]( https://lists.xenproject.org/archives/html/xen-devel/2021-05/msg00882.html) for more details. The whole design is about Static Allocation and direct-map, and this Patch Serie only covers parts of it, which are direct-map memory map. Other features will be delievered through different patch series. See https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00855.html for Domain on Static Allocation. This patch serie is based on https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00822.html --- v2 changes: - remove the introduce of internal flag - Refine is_domain_direct_mapped to check whether the flag XEN_DOMCTL_CDF_directmap is set - reword "1:1 direct-map" to just "direct-map" - split the common codes into two helpers: parse_static_mem_prop and acquire_static_memory_bank to deduce complexity. - introduce a new helper allocate_static_memory_11 for allocating static memory for direct-map guests - remove panic action since it is fine to assign a non-DMA capable device when IOMMU and direct-map both off - remove redistributor accessor - introduce new helper "is_domain_use_host_layout()" - explain why vpl011 initialization before creating its device tree node - error out if the domain is direct-mapped and the IRQ is not found - harden the code and add a check/comment when the hardware UART region is smaller than CUEST_VPL011_SIZE. Stefano Stabellini (6): xen: introduce XEN_DOMCTL_CDF_directmap xen/arm: introduce direct-map for domUs xen/arm: if direct-map domain use native addresses for GICv2 xen/arm: if direct-map domain use native addresses for GICv3 xen/arm: if direct-map domain use native UART address and IRQ number for vPL011 xen/docs: add a document to explain how to do passthrough without IOMMU docs/misc/arm/device-tree/booting.txt | 10 + docs/misc/arm/passthrough-noiommu.txt | 54 + xen/arch/arm/domain.c | 3 +- xen/arch/arm/domain_build.c | 316 -- xen/arch/arm/vgic-v2.c| 26 ++- xen/arch/arm/vgic-v3.c| 20 +- xen/arch/arm/vgic/vgic-v2.c | 27 ++- xen/arch/arm/vpl011.c | 54 - xen/common/domain.c | 3 +- xen/include/asm-arm/domain.h | 7 +- xen/include/asm-arm/new_vgic.h| 10 + xen/include/asm-arm/vgic.h| 12 +- xen/include/asm-arm/vpl011.h | 2 + xen/include/public/domctl.h | 4 +- 14 files changed, 449 insertions(+), 99 deletions(-) create mode 100644 docs/misc/arm/passthrough-noiommu.txt -- 2.25.1
[PATCH v2 2/6] xen/arm: introduce direct-map for domUs
From: Stefano Stabellini
Cases where domU needs direct-map memory map:
* IOMMU not present in the system.
* IOMMU disabled if it doesn't cover a specific device and all the guests
are trusted. Thinking a mixed scenario, where a few devices with IOMMU and
a few without, then guest DMA security still could not be totally guaranteed.
So users may want to disable the IOMMU, to at least gain some performance
improvement from IOMMU disabled.
* IOMMU disabled as a workaround when it doesn't have enough bandwidth.
To be specific, in a few extreme situation, when multiple devices do DMA
concurrently, these requests may exceed IOMMU's transmission capacity.
* IOMMU disabled when it adds too much latency on DMA. For example,
TLB may be missing in some IOMMU hardware, which may bring latency in DMA
progress, so users may want to disable it in some realtime scenario.
* Guest OS relies on the host memory layout-capablei
*WARNING:
Users should be aware that it is not always secure to assign a DMA-capable
device without IOMMU protection.
The administrator should make sure that:
1. The device is assigned to a trusted guest.
2. Users have additional security mechanism on the platform.
This commit also avoids setting XEN_DOMCTL_CDF_iommu when the IOMMU is
absent/disabled.
For now, direct-map is only supported when domain on Static Allocation, that is,
"xen.static-mem" must be also defined in the domain configuration.
This commit also introduces a new helper allocate_static_memory_11 to allocate
static memory as guest RAM for direct-map domain.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
---
docs/misc/arm/device-tree/booting.txt | 10 ++
xen/arch/arm/domain_build.c | 215 --
2 files changed, 179 insertions(+), 46 deletions(-)
diff --git a/docs/misc/arm/device-tree/booting.txt
b/docs/misc/arm/device-tree/booting.txt
index 71895663a4..297f8fa0c8 100644
--- a/docs/misc/arm/device-tree/booting.txt
+++ b/docs/misc/arm/device-tree/booting.txt
@@ -182,6 +182,16 @@ with the following properties:
Both #address-cells and #size-cells need to be specified because
both sub-nodes (described shortly) have reg properties.
+- direct-map
+
+Optional for Domain on Static Allocation.
+An empty property to request the memory of the domain to be
+direct-map (guest physical address == physical address).
+WARNING:
+Users must be aware of this risk, when doing DMA-capable device assignment,
+direct-map guest must be trusted or have additional security mechanism,
+otherwise it could use the DMA engine to access any other memory area.
+
Under the "xen,domain" compatible node, one or more sub-nodes are present
for the DomU kernel and ramdisk.
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 37e2d62d47..d9118e5bc1 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -492,8 +492,14 @@ static bool __init append_static_memory_to_bank(struct
domain *d,
{
int res;
unsigned int nr_pages = PFN_DOWN(size);
-/* Infer next GFN. */
-gfn_t sgfn = gaddr_to_gfn(bank->start + bank->size);
+gfn_t sgfn;
+
+if ( !is_domain_direct_mapped(d) )
+/* Infer next GFN when GFN != MFN. */
+sgfn = gaddr_to_gfn(bank->start + bank->size);
+else
+sgfn = gaddr_to_gfn(mfn_to_maddr(smfn));
+
res = guest_physmap_add_pages(d, sgfn, smfn, nr_pages);
if ( res )
@@ -507,12 +513,69 @@ static bool __init append_static_memory_to_bank(struct
domain *d,
return true;
}
-/* Allocate memory from static memory as RAM for one specific domain d. */
+static int __init acquire_static_memory_bank(struct domain *d,
+ const __be32 **cell,
+ u32 addr_cells, u32 size_cells,
+ paddr_t *pbase, paddr_t *psize)
+{
+int res = 0;
+
+device_tree_get_reg(cell, addr_cells, size_cells, pbase, psize);
+ASSERT(IS_ALIGNED(*pbase, PAGE_SIZE) && IS_ALIGNED(*psize, PAGE_SIZE));
+if ( PFN_DOWN(*psize) > UINT_MAX )
+{
+printk(XENLOG_ERR "%pd: static memory size too large: %#"PRIpaddr,
+ d, *psize);
+return -EINVAL;
+}
+
+res = acquire_domstatic_pages(d, maddr_to_mfn(*pbase), PFN_DOWN(*psize),
0);
+if ( res )
+printk(XENLOG_ERR
+ "%pd: failed to acquire static memory: %d.\n", d, res);
+
+return res;
+}
+
+static int __init parse_static_mem_prop(const struct dt_device_node *node,
+u32 *addr_cells, u32 *size_cells,
+int *length, const __be32 **cell)
+{
+const struct dt_property *prop;
+
+prop = dt_find_property(node, "xen,static-mem", NULL);
+if ( !dt_property_read_u32(node, "#xen,st
[PATCH v2 3/6] xen/arm: if direct-map domain use native addresses for GICv2
From: Stefano Stabellini
Today we use native addresses to map the GICv2 for Dom0 and fixed
addresses for DomUs.
This patch changes the behavior so that native addresses are used for
all domains that are direct-map memory map.
NEW VGIC has different naming schemes, like referring distributor base
address as vgic_dist_base, other than the dbase. So this patch also introduces
vgic_dist_base/vgic_cpu_base accessor to access correct distributor base
address/cpu interface base address on varied scenarios,
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c| 10 +++---
xen/arch/arm/vgic-v2.c | 26 +-
xen/arch/arm/vgic/vgic-v2.c| 27 ++-
xen/include/asm-arm/new_vgic.h | 10 ++
xen/include/asm-arm/vgic.h | 12 +++-
5 files changed, 71 insertions(+), 14 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index d9118e5bc1..6cd03e4d0f 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2207,8 +2207,12 @@ static int __init make_gicv2_domU_node(struct
kernel_info *kinfo)
int res = 0;
__be32 reg[(GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) * 2];
__be32 *cells;
+struct domain *d = kinfo->d;
+char buf[38];
-res = fdt_begin_node(fdt,
"interrupt-controller@"__stringify(GUEST_GICD_BASE));
+snprintf(buf, sizeof(buf), "interrupt-controller@%"PRIx64,
+ vgic_dist_base(&d->arch.vgic));
+res = fdt_begin_node(fdt, buf);
if ( res )
return res;
@@ -2230,9 +2234,9 @@ static int __init make_gicv2_domU_node(struct kernel_info
*kinfo)
cells = ®[0];
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICD_BASE, GUEST_GICD_SIZE);
+ vgic_dist_base(&d->arch.vgic), GUEST_GICD_SIZE);
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICC_BASE, GUEST_GICC_SIZE);
+ vgic_cpu_base(&d->arch.vgic), GUEST_GICC_SIZE);
res = fdt_property(fdt, "reg", reg, sizeof(reg));
if (res)
diff --git a/xen/arch/arm/vgic-v2.c b/xen/arch/arm/vgic-v2.c
index b2da886adc..a8cf8173d0 100644
--- a/xen/arch/arm/vgic-v2.c
+++ b/xen/arch/arm/vgic-v2.c
@@ -652,7 +652,7 @@ static int vgic_v2_vcpu_init(struct vcpu *v)
static int vgic_v2_domain_init(struct domain *d)
{
int ret;
-paddr_t cbase, csize;
+paddr_t csize;
paddr_t vbase;
/*
@@ -669,10 +669,26 @@ static int vgic_v2_domain_init(struct domain *d)
* Note that we assume the size of the CPU interface is always
* aligned to PAGE_SIZE.
*/
-cbase = vgic_v2_hw.cbase;
+d->arch.vgic.cbase = vgic_v2_hw.cbase;
csize = vgic_v2_hw.csize;
vbase = vgic_v2_hw.vbase;
}
+else if ( is_domain_direct_mapped(d) )
+{
+/*
+ * For non-dom0 direct_mapped guests we only map a 8kB CPU
+ * interface but we make sure it is at a location occupied by
+ * the physical GIC in the host device tree.
+ *
+ * We need to add an offset to the virtual CPU interface base
+ * address when the GIC is aliased to get a 8kB contiguous
+ * region.
+ */
+d->arch.vgic.dbase = vgic_v2_hw.dbase;
+d->arch.vgic.cbase = vgic_v2_hw.cbase + vgic_v2_hw.aliased_offset;
+csize = GUEST_GICC_SIZE;
+vbase = vgic_v2_hw.vbase + vgic_v2_hw.aliased_offset;
+}
else
{
d->arch.vgic.dbase = GUEST_GICD_BASE;
@@ -683,7 +699,7 @@ static int vgic_v2_domain_init(struct domain *d)
* region.
*/
BUILD_BUG_ON(GUEST_GICC_SIZE != SZ_8K);
-cbase = GUEST_GICC_BASE;
+d->arch.vgic.cbase = GUEST_GICC_BASE;
csize = GUEST_GICC_SIZE;
vbase = vgic_v2_hw.vbase + vgic_v2_hw.aliased_offset;
}
@@ -692,8 +708,8 @@ static int vgic_v2_domain_init(struct domain *d)
* Map the gic virtual cpu interface in the gic cpu interface
* region of the guest.
*/
-ret = map_mmio_regions(d, gaddr_to_gfn(cbase), csize / PAGE_SIZE,
- maddr_to_mfn(vbase));
+ret = map_mmio_regions(d, gaddr_to_gfn(d->arch.vgic.cbase),
+ csize / PAGE_SIZE, maddr_to_mfn(vbase));
if ( ret )
return ret;
diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
index b5ba4ace87..ce1f6e4373 100644
--- a/xen/arch/arm/vgic/vgic-v2.c
+++ b/xen/arch/arm/vgic/vgic-v2.c
@@ -258,7 +258,7 @@ void vgic_v2_enable(struct vcpu *vcpu)
int vgic_v2_map_resources(struct domain *d)
{
struct vgic_dist *dist = &d->arch.vgic;
-paddr_t cbase, csize;
+paddr_t csize;
paddr_t vbase;
int ret;
@@ -276,10 +276,27 @@ int vgic
[PATCH v2 5/6] xen/arm: if direct-map domain use native UART address and IRQ number for vPL011
From: Stefano Stabellini
We always use a fix address to map the vPL011 to domains. The address
could be a problem for direct-map domains.
So, for domains that are directly mapped, reuse the address of the
physical UART on the platform to avoid potential clashes.
Do the same for the virtual IRQ number: instead of always using
GUEST_VPL011_SPI, try to reuse the physical SPI number if possible.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c | 41 ++-
xen/arch/arm/vpl011.c| 54 +++-
xen/include/asm-arm/vpl011.h | 2 ++
3 files changed, 83 insertions(+), 14 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 7e0ee07e06..f3e87709f6 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -30,6 +30,7 @@
#include
#include
+#include
static unsigned int __initdata opt_dom0_max_vcpus;
integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
@@ -2350,8 +2351,11 @@ static int __init make_vpl011_uart_node(struct
kernel_info *kinfo)
gic_interrupt_t intr;
__be32 reg[GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS];
__be32 *cells;
+struct domain *d = kinfo->d;
+char buf[27];
-res = fdt_begin_node(fdt, "sbsa-uart@"__stringify(GUEST_PL011_BASE));
+snprintf(buf, sizeof(buf), "sbsa-uart@%"PRIx64, d->arch.vpl011.base_addr);
+res = fdt_begin_node(fdt, buf);
if ( res )
return res;
@@ -2361,14 +2365,14 @@ static int __init make_vpl011_uart_node(struct
kernel_info *kinfo)
cells = ®[0];
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS,
- GUEST_ROOT_SIZE_CELLS, GUEST_PL011_BASE,
+ GUEST_ROOT_SIZE_CELLS, d->arch.vpl011.base_addr,
GUEST_PL011_SIZE);
res = fdt_property(fdt, "reg", reg, sizeof(reg));
if ( res )
return res;
-set_interrupt(intr, GUEST_VPL011_SPI, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
+set_interrupt(intr, d->arch.vpl011.virq, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
res = fdt_property(fdt, "interrupts", intr, sizeof (intr));
if ( res )
@@ -3083,6 +3087,14 @@ static int __init construct_domU(struct domain *d,
allocate_static_memory(d, &kinfo, node);
}
+/*
+ * Base address and irq number are needed when creating vpl011 device
+ * tree node in prepare_dtb_domU, so initialization on related variables
+ * shall be dealt firstly.
+ */
+if ( kinfo.vpl011 )
+rc = domain_vpl011_init(d, NULL);
+
rc = prepare_dtb_domU(d, &kinfo);
if ( rc < 0 )
return rc;
@@ -3091,9 +3103,6 @@ static int __init construct_domU(struct domain *d,
if ( rc < 0 )
return rc;
-if ( kinfo.vpl011 )
-rc = domain_vpl011_init(d, NULL);
-
return rc;
}
@@ -3132,15 +3141,33 @@ void __init create_domUs(void)
if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
{
+unsigned int vpl011_virq = GUEST_VPL011_SPI;
+
d_cfg.arch.nr_spis = gic_number_lines() - 32;
+/*
+ * The VPL011 virq is GUEST_VPL011_SPI, unless direct-map in
+ * set, in which case we'll try to match the hardware.
+ *
+ * Typically, d->arch.vpl011.virq has the vpl011 irq number
+ * but at this point of the boot sequence it is not
+ * initialized yet.
+ */
+if ( d_cfg.flags & XEN_DOMCTL_CDF_directmap )
+{
+vpl011_virq = serial_irq(SERHND_DTUART);
+if ( vpl011_virq < 0 )
+panic("Error getting IRQ number for this serial port %d\n",
+ SERHND_DTUART);
+}
+
/*
* vpl011 uses one emulated SPI. If vpl011 is requested, make
* sure that we allocate enough SPIs for it.
*/
if ( dt_property_read_bool(node, "vpl011") )
d_cfg.arch.nr_spis = MAX(d_cfg.arch.nr_spis,
- GUEST_VPL011_SPI - 32 + 1);
+ vpl011_virq - 32 + 1);
}
/*
diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
index 895f436cc4..2de59e584d 100644
--- a/xen/arch/arm/vpl011.c
+++ b/xen/arch/arm/vpl011.c
@@ -29,6 +29,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -71,11 +72,11 @@ static void vpl011_update_interrupt_status(struct domain *d)
* status bit has been set since the last time.
*/
if ( uartmis & ~vpl011->shadow_uartmis )
-vgic_inject_irq(d, NULL, GUEST_VPL011_SPI, true);
+vgic_inject_irq(d, NULL, vpl011->virq, true);
vpl011->shadow_uar
[PATCH v2 6/6] xen/docs: add a document to explain how to do passthrough without IOMMU
From: Stefano Stabellini
Make sure to start with a WARNING about security.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
---
docs/misc/arm/passthrough-noiommu.txt | 54 +++
1 file changed, 54 insertions(+)
create mode 100644 docs/misc/arm/passthrough-noiommu.txt
diff --git a/docs/misc/arm/passthrough-noiommu.txt
b/docs/misc/arm/passthrough-noiommu.txt
new file mode 100644
index 00..61aeb8a5cd
--- /dev/null
+++ b/docs/misc/arm/passthrough-noiommu.txt
@@ -0,0 +1,54 @@
+Request Device Assignment without IOMMU support
+===
+
+*WARNING:
+Users should be aware that it is not always secure to assign a device without
+IOMMU protection.
+When the device is not protected by the IOMMU, the administrator should make
+sure that:
+ 1. The device is assigned to a trusted guest.
+ 2. Users have additional security mechanism on the platform.
+
+
+This document assumes that the IOMMU is absent from the system or it is
+disabled (status = "disabled" in device tree).
+
+
+Add xen,force-assign-without-iommu; to the device tree snippet:
+
+ethernet: ethernet@ff0e {
+ compatible = "cdns,zynqmp-gem";
+ xen,path = "/amba/ethernet@ff0e";
+ xen,reg = <0x0 0xff0e 0x1000 0x0 0xff0e>;
+ xen,force-assign-without-iommu;
+};
+
+Request 1:1 memory mapping for the domain on static allocation
+==
+
+Add a direct-map property under the appropriate /chosen/domU node which
+is also statically allocated with physical memory ranges through
+xen,static-mem property as its guest RAM.
+
+Below is an example on how to specify the 1:1 memory mapping for the domain
+on static allocation in the device-tree:
+
+/ {
+ chosen {
+ domU1 {
+ compatible = "xen,domain";
+ #address-cells = <0x2>;
+ #size-cells = <0x2>;
+ cpus = <2>;
+ memory = <0x0 0x8>;
+ #xen,static-mem-address-cells = <0x1>;
+ #xen,static-mem-size-cells = <0x1>;
+ xen,static-mem = <0x3000 0x2000>;
+ direct-map;
+ ...
+ };
+ };
+};
+
+Besides reserving a 512MB region starting at the host physical address
+0x3000 to DomU1, it also requests 1:1 memory mapping.
--
2.25.1
[PATCH v2 4/6] xen/arm: if direct-map domain use native addresses for GICv3
From: Stefano Stabellini
Today we use native addresses to map the GICv3 for Dom0 and fixed
addresses for DomUs.
This patch changes the behavior so that native addresses are used for
all direct-map domains(including Dom0).
Considering that dom0 may not always be directly mapped in the future,
this patch introduces a new helper "is_domain_use_host_layout()" that
wraps both two check "is_domain_direct_mapped(d) || is_hardware_domain(d)"
for more flexible useage.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
---
xen/arch/arm/domain_build.c | 46 +++-
xen/arch/arm/vgic-v3.c | 20 +---
xen/include/asm-arm/domain.h | 3 +++
3 files changed, 50 insertions(+), 19 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 6cd03e4d0f..7e0ee07e06 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2255,16 +2255,20 @@ static int __init make_gicv2_domU_node(struct
kernel_info *kinfo)
return res;
}
+#ifdef CONFIG_ARM_64
static int __init make_gicv3_domU_node(struct kernel_info *kinfo)
{
void *fdt = kinfo->fdt;
int res = 0;
-__be32 reg[(GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) * 2];
+__be32 *reg;
__be32 *cells;
+struct domain *d = kinfo->d;
+char buf[38];
+unsigned int i, len = 0;
-res = fdt_begin_node(fdt,
"interrupt-controller@"__stringify(GUEST_GICV3_GICD_BASE));
-if ( res )
-return res;
+snprintf(buf, sizeof(buf), "interrupt-controller@%"PRIx64,
+ vgic_dist_base(&d->arch.vgic));
+res = fdt_begin_node(fdt, buf);
res = fdt_property_cell(fdt, "#address-cells", 0);
if ( res )
@@ -2282,35 +2286,55 @@ static int __init make_gicv3_domU_node(struct
kernel_info *kinfo)
if ( res )
return res;
+/* reg specifies all re-distributors and Distributor. */
+len = (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) *
+ (d->arch.vgic.nr_regions + 1) * sizeof(__be32);
+reg = xmalloc_bytes(len);
+if ( reg == NULL )
+return -ENOMEM;
+
cells = ®[0];
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICV3_GICD_BASE, GUEST_GICV3_GICD_SIZE);
-dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICV3_GICR0_BASE, GUEST_GICV3_GICR0_SIZE);
+ vgic_dist_base(&d->arch.vgic), GUEST_GICV3_GICD_SIZE);
-res = fdt_property(fdt, "reg", reg, sizeof(reg));
+for ( i = 0;
+ i < d->arch.vgic.nr_regions;
+ i++, cells += (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) )
+{
+dt_child_set_range(&cells,
+ GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
+ d->arch.vgic.rdist_regions[i].base,
+ d->arch.vgic.rdist_regions[i].size);
+}
+
+res = fdt_property(fdt, "reg", reg, len);
if (res)
-return res;
+goto out;
res = fdt_property_cell(fdt, "linux,phandle", kinfo->phandle_gic);
if (res)
-return res;
+goto out;
res = fdt_property_cell(fdt, "phandle", kinfo->phandle_gic);
if (res)
-return res;
+goto out;
res = fdt_end_node(fdt);
+ out:
+xfree(reg);
return res;
}
+#endif
static int __init make_gic_domU_node(struct kernel_info *kinfo)
{
switch ( kinfo->d->arch.vgic.version )
{
+#ifdef CONFIG_ARM_64
case GIC_V3:
return make_gicv3_domU_node(kinfo);
+#endif
case GIC_V2:
return make_gicv2_domU_node(kinfo);
default:
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index cb5a70c42e..70168ca1ac 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -1641,14 +1641,15 @@ static inline unsigned int
vgic_v3_max_rdist_count(struct domain *d)
* Normally there is only one GICv3 redistributor region.
* The GICv3 DT binding provisions for multiple regions, since there are
* platforms out there which need those (multi-socket systems).
- * For Dom0 we have to live with the MMIO layout the hardware provides,
- * so we have to copy the multiple regions - as the first region may not
- * provide enough space to hold all redistributors we need.
+ * For direct-map domain(including dom0), we have to live with the MMIO
+ * layout the hardware provides, so we have to copy the multiple regions
+ * - as the first region may not provide enough space to hold all
+ * redistributors we need.
* However DomU get a constructed memory map, so we can go with
* the architected single redistributor region.
*/
-return is_hardware_domain
RE: [PATCH v2 1/6] xen: introduce XEN_DOMCTL_CDF_directmap
Hi Jan
> -Original Message-
> From: Jan Beulich
> Sent: Friday, October 15, 2021 4:47 PM
> To: Penny Zheng
> Cc: Wei Chen ; Bertrand Marquis
> ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org; jul...@xen.org
> Subject: Re: [PATCH v2 1/6] xen: introduce XEN_DOMCTL_CDF_directmap
>
> On 15.10.2021 05:09, Penny Zheng wrote:
> > From: Stefano Stabellini
> >
> > This commit introduces a new arm-specific flag
> > XEN_DOMCTL_CDF_directmap to specify that this domain should have its
> > memory directly mapped (guest physical address == physical address) at
> domain creation.
> >
> > Refine is_domain_direct_mapped to check whether the flag
> > XEN_DOMCTL_CDF_directmap is set.
> >
> > Signed-off-by: Stefano Stabellini
> > Signed-off-by: Penny Zheng
> > ---
> > CC: andrew.coop...@citrix.com
> > CC: jbeul...@suse.com
> > CC: George Dunlap
> > CC: Ian Jackson
> > CC: Wei Liu
> > CC: "Roger Pau Monné"
> > ---
>
> Please have here a brief log of changes in the new version, to aid reviewers.
>
Sure.
> > xen/arch/arm/domain.c| 3 ++-
> > xen/arch/arm/domain_build.c | 4 +++-
> > xen/common/domain.c | 3 ++-
> > xen/include/asm-arm/domain.h | 4 ++-- xen/include/public/domctl.h |
> > 4 +++-
> > 5 files changed, 12 insertions(+), 6 deletions(-)
>
> You clearly had to re-base over the XEN_DOMCTL_CDF_vpmu addition. I think
> just like that change (which I'd expect you to have looked at while doing the
> re-base) you also need to at least fiddle with OCaml's domain_create_flag, to
> keep the ABI check there happy.
>
The patch serie is based on the staging branch with an extra commit "
Revert "xen/domctl: Introduce XEN_DOMCTL_CDF_vpci flag", which
Is already been pushed to community for review.(
https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00822.html)
> > --- a/xen/include/public/domctl.h
> > +++ b/xen/include/public/domctl.h
> > @@ -72,9 +72,11 @@ struct xen_domctl_createdomain {
> > #define XEN_DOMCTL_CDF_nested_virt(1U <<
> _XEN_DOMCTL_CDF_nested_virt)
> > /* Should we expose the vPMU to the guest? */
> > #define XEN_DOMCTL_CDF_vpmu (1U << 7)
> > +/* If this domain has its memory directly mapped? (ARM only) */
> > +#define XEN_DOMCTL_CDF_directmap (1U << 8)
>
> The comment doesn't read well; how about "Should domain memory be
> directly mapped?" That's if a comment here is really needed in the first
> place. I
> also don't think "Arm only" should be here - this may go stale. What I'm
> missing in this regard is rejecting of the flag in x86'es
> arch_sanitise_domain_config() (or by whichever other means).
>
> Jan
RE: [PATCH v2 1/6] xen: introduce XEN_DOMCTL_CDF_directmap
Hi julien
> -Original Message-
> From: Julien Grall
> Sent: Friday, October 15, 2021 4:57 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Wei Chen ; Bertrand Marquis
>
> Subject: Re: [PATCH v2 1/6] xen: introduce XEN_DOMCTL_CDF_directmap
>
> Hi Penny,
>
> On 15/10/2021 04:09, Penny Zheng wrote:
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index 0167731ab0..37e2d62d47 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -3069,8 +3069,10 @@ static int __init construct_dom0(struct domain *d)
> > void __init create_dom0(void)
> > {
> > struct domain *dom0;
> > +/* DOM0 has always its memory directly mapped. */
> > struct xen_domctl_createdomain dom0_cfg = {
> > -.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,
> > +.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
> > + XEN_DOMCTL_CDF_directmap,
> > .max_evtchn_port = -1,
> > .max_grant_frames = gnttab_dom0_frames(),
> > .max_maptrack_frames = -1,
> > diff --git a/xen/common/domain.c b/xen/common/domain.c index
> > 8b53c49d1e..7a6131db74 100644
> > --- a/xen/common/domain.c
> > +++ b/xen/common/domain.c
> > @@ -486,7 +486,8 @@ static int sanitise_domain_config(struct
> xen_domctl_createdomain *config)
> >~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
> > XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off |
> > XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu |
> > - XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu) )
> > + XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu |
> > + XEN_DOMCTL_CDF_directmap) )
> > {
> > dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
> > return -EINVAL;
> > diff --git a/xen/include/asm-arm/domain.h
> > b/xen/include/asm-arm/domain.h index 14e575288f..fc42c6a310 100644
> > --- a/xen/include/asm-arm/domain.h
> > +++ b/xen/include/asm-arm/domain.h
> > @@ -29,8 +29,8 @@ enum domain_type {
> > #define is_64bit_domain(d) (0)
> > #endif
> >
> > -/* The hardware domain has always its memory direct mapped. */
> > -#define is_domain_direct_mapped(d) is_hardware_domain(d)
> > +/* Check if domain is direct-map memory map. */ #define
> > +is_domain_direct_mapped(d) (d->options & XEN_DOMCTL_CDF_directmap)
> >
> > struct vtimer {
> > struct vcpu *v;
> > diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
> > index 238384b5ae..b505a0db51 100644
> > --- a/xen/include/public/domctl.h
> > +++ b/xen/include/public/domctl.h
> > @@ -72,9 +72,11 @@ struct xen_domctl_createdomain {
> > #define XEN_DOMCTL_CDF_nested_virt(1U <<
> _XEN_DOMCTL_CDF_nested_virt)
> > /* Should we expose the vPMU to the guest? */
> > #define XEN_DOMCTL_CDF_vpmu (1U << 7)
> > +/* If this domain has its memory directly mapped? (ARM only) */
> > +#define XEN_DOMCTL_CDF_directmap (1U << 8)
> >
> > /* Max XEN_DOMCTL_CDF_* constant. Used for ABI checking. */
> > -#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_vpmu
> > +#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_directmap
>
> In the previous version, this flag was only settable for domain created by
> Xen.
> Now this is also settable by the toolstack. I don't think the toolstack can
> sensibly use this flag (at least in the current state).
>
> So can you explain why this flag is exposed to the toolstack?
Oh, I misunderstood the previous discussion on internal usage a bit, sorry.
And I will make it back to xen/include/xen/domain.h
>
> Cheers,
>
> --
> Julien Grall
RE: [PATCH v2 4/6] xen/arm: if direct-map domain use native addresses for GICv3
Hi Julien
> -Original Message-
> From: Julien Grall
> Sent: Wednesday, October 20, 2021 7:11 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Wei Chen ; Bertrand Marquis
>
> Subject: Re: [PATCH v2 4/6] xen/arm: if direct-map domain use native
> addresses for GICv3
>
> Hi Penny,
>
> On 15/10/2021 04:09, Penny Zheng wrote:
> > From: Stefano Stabellini
> >
> > Today we use native addresses to map the GICv3 for Dom0 and fixed
> > addresses for DomUs.
> >
> > This patch changes the behavior so that native addresses are used for
> > all direct-map domains(including Dom0).
> >
> > Considering that dom0 may not always be directly mapped in the future,
> > this patch introduces a new helper "is_domain_use_host_layout()" that
> > wraps both two check "is_domain_direct_mapped(d) ||
> is_hardware_domain(d)"
> > for more flexible useage.
>
> Typo: s/useage/usage/
>
> >
> > Signed-off-by: Stefano Stabellini
> > Signed-off-by: Penny Zheng
> > ---
> > xen/arch/arm/domain_build.c | 46 +++-
> > xen/arch/arm/vgic-v3.c | 20 +---
> > xen/include/asm-arm/domain.h | 3 +++
> > 3 files changed, 50 insertions(+), 19 deletions(-)
> >
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index 6cd03e4d0f..7e0ee07e06 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -2255,16 +2255,20 @@ static int __init make_gicv2_domU_node(struct
> kernel_info *kinfo)
> > return res;
> > }
> >
> > +#ifdef CONFIG_ARM_64
>
> The code below is specific to the GICv3 (and not 64-bit). So this should be
> gated with CONFIG_GICV3.
>
> Personally, I would have gated the code in a separate patch. But I am fine if
> this is added in this patch so long this is mentionned in the commit message.
>
> > static int __init make_gicv3_domU_node(struct kernel_info *kinfo)
> > {
> > void *fdt = kinfo->fdt;
> > int res = 0;
> > -__be32 reg[(GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) *
> 2];
> > +__be32 *reg;
> > __be32 *cells;
> > +struct domain *d = kinfo->d;
>
> AFAICT, 'd' is not going to be modified. So please add const.
>
> > +char buf[38];
>
> Please explain how 38 was found. For an example, see the comment on top of
> 'buf' in make_memory_node().
>
> > +unsigned int i, len = 0;
> >
> > -res = fdt_begin_node(fdt, "interrupt-
> controller@"__stringify(GUEST_GICV3_GICD_BASE));
> > -if ( res )
> > -return res;
> > +snprintf(buf, sizeof(buf), "interrupt-controller@%"PRIx64,
> > + vgic_dist_base(&d->arch.vgic));
> > +res = fdt_begin_node(fdt, buf);
>
> You set res but it gets overwritten just below. Were you meant to check the
> return value?
>
> >
> > res = fdt_property_cell(fdt, "#address-cells", 0);
> > if ( res )
> > @@ -2282,35 +2286,55 @@ static int __init make_gicv3_domU_node(struct
> kernel_info *kinfo)
> > if ( res )
> > return res;
> >
> > +/* reg specifies all re-distributors and Distributor. */
> > +len = (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) *
> > + (d->arch.vgic.nr_regions + 1) * sizeof(__be32);
> > +reg = xmalloc_bytes(len);
> > +if ( reg == NULL )
> > +return -ENOMEM;
> > +
> > cells = ®[0];
> > dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS,
> GUEST_ROOT_SIZE_CELLS,
> > - GUEST_GICV3_GICD_BASE, GUEST_GICV3_GICD_SIZE);
> > -dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS,
> GUEST_ROOT_SIZE_CELLS,
> > - GUEST_GICV3_GICR0_BASE, GUEST_GICV3_GICR0_SIZE);
> > + vgic_dist_base(&d->arch.vgic),
> > + GUEST_GICV3_GICD_SIZE);
> >
> > -res = fdt_property(fdt, "reg", reg, sizeof(reg));
> > +for ( i = 0;
> > + i < d->arch.vgic.nr_regions;
> > + i++, cells += (GUEST_ROOT_ADDRESS_CELLS +
> > + GUEST_ROOT_SIZE_CELLS) )
>
> dt_child_set_range() will already update cells to the next ones. So this needs
> to be dropped.
>
You're so right!!! I checked the code and it will already points to the next
ones
> I was expecting this to be caugt during test. So did you test this series with
> GICv3?
RE: static-mem preventing dom0 from booting
Hi Stefano
> -Original Message-
> From: Stefano Stabellini
> Sent: Friday, November 5, 2021 9:36 AM
> To: Penny Zheng
> Cc: sstabell...@kernel.org; xen-devel@lists.xenproject.org
> Subject: static-mem preventing dom0 from booting
>
> Hi Penny,
>
> I am trying to test the static-mem feature in Xen 4.16 but I am having some
> issues. I can boot a dom0less domU with static-mem configured correctly, but
> when I do so, dom0 doesn't boot any longer.
>
Hmm, In our first intention, dom0less is a mandatory option for static
allocation.
domU on static allocation shall boot when dom0 doesn't boot there.
We think that, nevertheless, dom0 memory range is allocated by Xen
automatically,
and it leads to the unpredictability. Static allocation through device tree
configuration
prefers the total static environment to avoid unpredictability.
> In the same configuration, if I remove the static-mem related lines from the
> domU node on device tree, both dom0 and domU boot successfully.
>
> I am sure the dom0 memory range, allocated by Xen automatically, and the
> domU memory range, hand-picked by me, do not clash as you can see from the
> boot logs (appended).
>
> Am I missing anything in the configuration? I am using a ZCU102 board, the
> memory node on the host device tree is:
>
Are you suggesting one scenario where dom0 with static-mem domU?
Hmmm, one quick thought, it may not be working with vpl011 emulation.
When dom0 exists, vpl011 will take dom0 as backend, which requests event
channel, xen store, etc, involved to communicate. but static-mem domU
without CONFIG_XEN shall not handle it.
> memory {
> device_type = "memory";
> reg = <0x00 0x00 0x00 0x7ff0 0x08 0x00 0x00 0x8000>;
> };
>
> I am attaching my u-boot boot.scr script with the device tree modification to
> add static memory.
>
> Many thanks for your help!
>
> Cheers,
>
> Stefano
>
>
> (XEN) Xen version 4.16-rc (sstabellini@) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04)
> 9.3.0) debug=y Thu Nov 4 18:28:30 PDT 2021
> (XEN) Latest ChangeSet: Mon Nov 1 12:36:26 2021 + git:9f8434d778-dirty
> (XEN) Processor: 410fd034: "ARM Limited", variant: 0x0, part
> 0xd03,rev 0x4
> (XEN) 64-bit Execution:
> (XEN) Processor Features: 1100
> (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
> (XEN) Extensions: FloatingPoint AdvancedSIMD
> (XEN) Debug Features: 10305106
> (XEN) Auxiliary Features:
> (XEN) Memory Model Features: 1122
> (XEN) ISA Features: 00011120
> (XEN) 32-bit Execution:
> (XEN) Processor Features: 1231:00011011
> (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
> (XEN) Extensions: GenericTimer Security
> (XEN) Debug Features: 03010066
> (XEN) Auxiliary Features:
> (XEN) Memory Model Features: 10101105 4000
> (XEN) 0126 02102211
> (XEN) ISA Features: 02101110 13112111
> 21232042
> (XEN) 01112131 00011142 00011121
> (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 65000 KHz
> (XEN) GICv2 initialization:
> (XEN) gic_dist_addr=f901
> (XEN) gic_cpu_addr=f902
> (XEN) gic_hyp_addr=f904
> (XEN) gic_vcpu_addr=f906
> (XEN) gic_maintenance_irq=25
> (XEN) GICv2: Adjusting CPU interface base to 0xf902f000
> (XEN) GICv2: 192 lines, 4 cpus, secure (IID ).
> (XEN) XSM Framework v1.0.1 initialized
> (XEN) Initialising XSM SILO mode
> (XEN) Using scheduler: null Scheduler (null)
> (XEN) Initializing null scheduler
> (XEN) WARNING: This is experimental software in development.
> (XEN) Use at your own risk.
> (XEN) Allocated console ring of 16 KiB.
> (XEN) Bringing up CPU1
> (XEN) Bringing up CPU2
> (XEN) Bringing up CPU3
> (XEN) Brought up 4 CPUs
> (XEN) I/O virtualisation enabled
> (XEN) - Dom0 mode: Relaxed
> (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
> (XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558
> (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Loading d0 kernel from boot module @ 00e0
> (XEN) Loading ramdisk from boot module @ 0220
> (XEN) Allocating 1:1 mappings totalling 1024MB for dom0:
> (XEN) BANK[0] 0x002000-0x006000 (1024MB)
> (XEN) Grant table range: 0x000560-0x000
RE: static-mem preventing dom0 from booting
Hi stefano
> -Original Message-
> From: Penny Zheng
> Sent: Friday, November 5, 2021 11:23 AM
> To: Stefano Stabellini
> Cc: xen-devel@lists.xenproject.org; Wei Chen
> Subject: RE: static-mem preventing dom0 from booting
>
> Hi Stefano
>
> > -Original Message-
> > From: Stefano Stabellini
> > Sent: Friday, November 5, 2021 9:36 AM
> > To: Penny Zheng
> > Cc: sstabell...@kernel.org; xen-devel@lists.xenproject.org
> > Subject: static-mem preventing dom0 from booting
> >
> > Hi Penny,
> >
> > I am trying to test the static-mem feature in Xen 4.16 but I am having
> > some issues. I can boot a dom0less domU with static-mem configured
> > correctly, but when I do so, dom0 doesn't boot any longer.
> >
>
> Hmm, In our first intention, dom0less is a mandatory option for static
> allocation.
> domU on static allocation shall boot when dom0 doesn't boot there.
>
> We think that, nevertheless, dom0 memory range is allocated by Xen
> automatically, and it leads to the unpredictability. Static allocation through
> device tree configuration prefers the total static environment to avoid
> unpredictability.
>
> > In the same configuration, if I remove the static-mem related lines
> > from the domU node on device tree, both dom0 and domU boot successfully.
> >
> > I am sure the dom0 memory range, allocated by Xen automatically, and
> > the domU memory range, hand-picked by me, do not clash as you can see
> > from the boot logs (appended).
> >
> > Am I missing anything in the configuration? I am using a ZCU102 board,
> > the memory node on the host device tree is:
> >
>
> Are you suggesting one scenario where dom0 with static-mem domU?
>
> Hmmm, one quick thought, it may not be working with vpl011 emulation.
> When dom0 exists, vpl011 will take dom0 as backend, which requests event
> channel, xen store, etc, involved to communicate. but static-mem domU
> without CONFIG_XEN shall not handle it.
>
Digging on this, and my misunderstanding. Domain created through device tree
shall use xen as backend, not dom0. I guess that only the one created by xl is
using
dom0 as backend.
Nevertheless, if this is your scenario, where dom0 and static-mem domU(NO
CONFIG_XEN) co-exists,
I'll test for you.
> > memory {
> > device_type = "memory";
> > reg = <0x00 0x00 0x00 0x7ff0 0x08 0x00 0x00 0x8000>;
> > };
> >
> > I am attaching my u-boot boot.scr script with the device tree
> > modification to add static memory.
> >
> > Many thanks for your help!
> >
> > Cheers,
> >
> > Stefano
> >
> >
> > (XEN) Xen version 4.16-rc (sstabellini@) (gcc (Ubuntu
> > 9.3.0-17ubuntu1~20.04)
> > 9.3.0) debug=y Thu Nov 4 18:28:30 PDT 2021
> > (XEN) Latest ChangeSet: Mon Nov 1 12:36:26 2021 +
> > git:9f8434d778-dirty
> > (XEN) Processor: 410fd034: "ARM Limited", variant: 0x0, part
> > 0xd03,rev 0x4
> > (XEN) 64-bit Execution:
> > (XEN) Processor Features: 1100
> > (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
> > (XEN) Extensions: FloatingPoint AdvancedSIMD
> > (XEN) Debug Features: 10305106
> > (XEN) Auxiliary Features:
> > (XEN) Memory Model Features: 1122
> > (XEN) ISA Features: 00011120
> > (XEN) 32-bit Execution:
> > (XEN) Processor Features: 1231:00011011
> > (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
> > (XEN) Extensions: GenericTimer Security
> > (XEN) Debug Features: 03010066
> > (XEN) Auxiliary Features:
> > (XEN) Memory Model Features: 10101105 4000
> > (XEN) 0126 02102211
> > (XEN) ISA Features: 02101110 13112111
> > 21232042
> > (XEN) 01112131 00011142 00011121
> > (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 65000 KHz
> > (XEN) GICv2 initialization:
> > (XEN) gic_dist_addr=f901
> > (XEN) gic_cpu_addr=f902
> > (XEN) gic_hyp_addr=f904
> > (XEN) gic_vcpu_addr=f906
> > (XEN) gic_maintenance_irq=25
> > (XEN) GICv2: Adjusting CPU interface base to 0xf902f000
> > (XEN) GICv2: 192 lines, 4 cpus, secure (IID ).
> > (XEN) XSM Fra
RE: static-mem preventing dom0 from booting
Hi Stefano
> -Original Message-
> From: Stefano Stabellini
> Sent: Saturday, November 6, 2021 7:03 AM
> To: Stefano Stabellini
> Cc: Penny Zheng ; xen-devel@lists.xenproject.org;
> Wei Chen ; Bertrand Marquis
>
> Subject: RE: static-mem preventing dom0 from booting
>
> On Fri, 5 Nov 2021, Stefano Stabellini wrote:
> > The scenario is extremely simple; you can see the full device tree
> > configuration in the attachment to my previous email.
> >
> > - dom0
> > - dom0less domU with static-mem
> >
> > That's it! So basically it is just a normal dom0 + dom0less domU
> > configuration, which already works fine, where I added static-mem to
> > the domU and suddenly dom0 (not the domU!) stopped working.
>
Got it. Sorry, I missed the scenario you are talking about... I simply think
what dom0less
means that dom0 is absent... ;/
> I did some more debugging today and I found the problem. The issue is that
> static-mem regions are added to the list of reserved_mem. However,
> reserved_mem is automatically assigned to Dom0 by default at the bottom of
> xen/arch/arm/domain_build.c:handle_node, see the second call to
> make_memory_node. Really, we shouldn't give to dom0 static-mem ranges
> meant for other domUs. E.g. the following change is sufficient to solve the
> problem:
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 88a79247cb..dc609c4f0e 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -891,6 +891,9 @@ static int __init make_memory_node(const struct
> domain *d,
> u64 start = mem->bank[i].start;
> u64 size = mem->bank[i].size;
>
> +if ( mem->bank[i].xen_domain )
> +continue;
> +
> dt_dprintk(" Bank %d: %#"PRIx64"->%#"PRIx64"\n",
> i, start, start + size);
>
> However, maybe a better fix would be to purge reserved_mem of any
> xen_domain items before calling make_memory_node.
>
>
> I found one additional issue regarding is_domain_direct_mapped which
> doesn't return true for static-mem domains. I think we need to add a
> direct_map bool to arch_domain and set it for both dom0 and static-mem
> dom0less domUs, so that we can change the implementation of
> is_domain_direct_mapped to:
>
> #define is_domain_direct_mapped(d) (d->arch.direct_map)
Yeah, I already pushed a patch serie regarding direct-map to community for
review,
and it is also based on your old direct-map serie.
Today, I may push direct-map version 3 to community for review~~~ If you're
free, plz take a look.
Cheers,
Penny Zheng
RE: [PATCH v2 1/6] xen: introduce XEN_DOMCTL_CDF_directmap
Hi Julien
> -Original Message-
> From: Julien Grall
> Sent: Friday, October 15, 2021 4:57 PM
> To: Penny Zheng ; xen-devel@lists.xenproject.org;
> sstabell...@kernel.org
> Cc: Wei Chen ; Bertrand Marquis
>
> Subject: Re: [PATCH v2 1/6] xen: introduce XEN_DOMCTL_CDF_directmap
>
> Hi Penny,
>
> On 15/10/2021 04:09, Penny Zheng wrote:
> > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> > index 0167731ab0..37e2d62d47 100644
> > --- a/xen/arch/arm/domain_build.c
> > +++ b/xen/arch/arm/domain_build.c
> > @@ -3069,8 +3069,10 @@ static int __init construct_dom0(struct domain *d)
> > void __init create_dom0(void)
> > {
> > struct domain *dom0;
> > +/* DOM0 has always its memory directly mapped. */
> > struct xen_domctl_createdomain dom0_cfg = {
> > -.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,
> > +.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
> > + XEN_DOMCTL_CDF_directmap,
> > .max_evtchn_port = -1,
> > .max_grant_frames = gnttab_dom0_frames(),
> > .max_maptrack_frames = -1,
> > diff --git a/xen/common/domain.c b/xen/common/domain.c index
> > 8b53c49d1e..7a6131db74 100644
> > --- a/xen/common/domain.c
> > +++ b/xen/common/domain.c
> > @@ -486,7 +486,8 @@ static int sanitise_domain_config(struct
> xen_domctl_createdomain *config)
> >~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
> > XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off |
> > XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu |
> > - XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu) )
> > + XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu |
> > + XEN_DOMCTL_CDF_directmap) )
> > {
> > dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
> > return -EINVAL;
> > diff --git a/xen/include/asm-arm/domain.h
> > b/xen/include/asm-arm/domain.h index 14e575288f..fc42c6a310 100644
> > --- a/xen/include/asm-arm/domain.h
> > +++ b/xen/include/asm-arm/domain.h
> > @@ -29,8 +29,8 @@ enum domain_type {
> > #define is_64bit_domain(d) (0)
> > #endif
> >
> > -/* The hardware domain has always its memory direct mapped. */
> > -#define is_domain_direct_mapped(d) is_hardware_domain(d)
> > +/* Check if domain is direct-map memory map. */ #define
> > +is_domain_direct_mapped(d) (d->options & XEN_DOMCTL_CDF_directmap)
> >
> > struct vtimer {
> > struct vcpu *v;
> > diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
> > index 238384b5ae..b505a0db51 100644
> > --- a/xen/include/public/domctl.h
> > +++ b/xen/include/public/domctl.h
> > @@ -72,9 +72,11 @@ struct xen_domctl_createdomain {
> > #define XEN_DOMCTL_CDF_nested_virt(1U <<
> _XEN_DOMCTL_CDF_nested_virt)
> > /* Should we expose the vPMU to the guest? */
> > #define XEN_DOMCTL_CDF_vpmu (1U << 7)
> > +/* If this domain has its memory directly mapped? (ARM only) */
> > +#define XEN_DOMCTL_CDF_directmap (1U << 8)
> >
> > /* Max XEN_DOMCTL_CDF_* constant. Used for ABI checking. */
> > -#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_vpmu
> > +#define XEN_DOMCTL_CDF_MAX XEN_DOMCTL_CDF_directmap
>
> In the previous version, this flag was only settable for domain created by
> Xen.
> Now this is also settable by the toolstack. I don't think the toolstack can
> sensibly use this flag (at least in the current state).
>
> So can you explain why this flag is exposed to the toolstack?
>
if I moved XEN_DOMCTL_CDF_directmap to the domain.h, and let it hold the
bit 8, in case later another developer wants to introduce a new flag to
accidently
hold 8 bit too, I would like to add some explanatory comments here and maybe
rename
the XEN_DOMCTL_CDF_directmap to XEN_DOMCTL_CDF_Internal_directmap, wdyt?
> Cheers,
>
> --
Cheers
Penny Zheng
> Julien Grall
RE: [PATCH v3] xen/arm: don't assign domU static-mem to dom0 as reserved-memory
Hi Stefano
> -Original Message-
> From: Stefano Stabellini
> Sent: Thursday, November 11, 2021 4:18 AM
> To: jul...@xen.org
> Cc: sstabell...@kernel.org; Penny Zheng ; Bertrand
> Marquis ; Wei Chen ;
> i...@xenproject.org; volodymyr_babc...@epam.com; xen-
> de...@lists.xenproject.org; Stefano Stabellini
> Subject: [PATCH v3] xen/arm: don't assign domU static-mem to dom0 as
> reserved-memory
>
> From: Stefano Stabellini
>
> DomUs static-mem ranges are added to the reserved_mem array for
> accounting, but they shouldn't be assigned to dom0 as the other regular
> reserved-memory ranges in device tree.
>
> In make_memory_nodes, fix the error by skipping banks with xen_domain set
> to true in the reserved-memory array. Also make sure to use the first valid
> (!xen_domain) start address for the memory node name.
>
> Fixes: 41c031ff437b ("xen/arm: introduce domain on Static Allocation")
> Signed-off-by: Stefano Stabellini
> Release-Acked-by: Ian Jackson
Thx again for the fixing up.
Reviewed-by: Penny Zheng
> ---
> Changes in v3:
> - move BUG_ON inside the loop
>
> Changes in v2:
> - improve commit message
> - improve in-code comment
> - update nr_cells appropriately
> ---
> xen/arch/arm/domain_build.c | 23 +--
> 1 file changed, 17 insertions(+), 6 deletions(-)
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 9e92b640cd..dafbc13962 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -862,21 +862,25 @@ static int __init make_memory_node(const struct
> domain *d, {
> int res, i;
> int reg_size = addrcells + sizecells;
> -int nr_cells = reg_size * mem->nr_banks;
> +int nr_cells = 0;
> /* Placeholder for memory@ + a 64-bit number + \0 */
> char buf[24];
> __be32 reg[NR_MEM_BANKS * 4 /* Worst case addrcells + sizecells */];
> __be32 *cells;
>
> -BUG_ON(nr_cells >= ARRAY_SIZE(reg));
> if ( mem->nr_banks == 0 )
> return -ENOENT;
>
> -dt_dprintk("Create memory node (reg size %d, nr cells %d)\n",
> - reg_size, nr_cells);
> +/* find first memory range not bound to a Xen domain */
> +for ( i = 0; i < mem->nr_banks && mem->bank[i].xen_domain; i++ )
> +;
> +if ( i == mem->nr_banks )
> +return 0;
> +
> +dt_dprintk("Create memory node\n");
>
> /* ePAPR 3.4 */
> -snprintf(buf, sizeof(buf), "memory@%"PRIx64, mem->bank[0].start);
> +snprintf(buf, sizeof(buf), "memory@%"PRIx64, mem->bank[i].start);
> res = fdt_begin_node(fdt, buf);
> if ( res )
> return res;
> @@ -886,17 +890,24 @@ static int __init make_memory_node(const struct
> domain *d,
> return res;
>
> cells = ®[0];
> -for ( i = 0 ; i < mem->nr_banks; i++ )
> +for ( ; i < mem->nr_banks; i++ )
> {
> u64 start = mem->bank[i].start;
> u64 size = mem->bank[i].size;
>
> +if ( mem->bank[i].xen_domain )
> +continue;
> +
> dt_dprintk(" Bank %d: %#"PRIx64"->%#"PRIx64"\n",
> i, start, start + size);
>
> +nr_cells += reg_size;
> +BUG_ON(nr_cells >= ARRAY_SIZE(reg));
> dt_child_set_range(&cells, addrcells, sizecells, start, size);
> }
>
> +dt_dprintk("(reg size %d, nr cells %d)\n", reg_size, nr_cells);
> +
> res = fdt_property(fdt, "reg", reg, nr_cells * sizeof(*reg));
> if ( res )
> return res;
> --
> 2.25.1
--
Cheers,
Penny Zheng
[PATCH v3 02/10] xen/arm: avoid setting XEN_DOMCTL_CDF_iommu when IOMMU off
From: Stefano Stabellini
This commit avoids setting XEN_DOMCTL_CDF_iommu when the IOMMU is
absent/disabled, otherwise xen will fail later when handling
device assignment.
Signed-off-by: Penny Zheng
Signed-off-by: Stefano Stabellini
---
v3 changes:
- new commit, split from the original "[PATCH v2 2/6] xen/arm: introduce
direct-map for domUs"
---
xen/arch/arm/domain_build.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 664c88ebe4..7a063f62fe 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2996,7 +2996,8 @@ void __init create_domUs(void)
panic("Missing property 'cpus' for domain %s\n",
dt_node_name(node));
-if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
+if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") &&
+ iommu_enabled )
d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
--
2.25.1
[PATCH v3 00/10] direct-map memory map
Cases where domU needs direct-map memory map: * IOMMU not present in the system. * IOMMU disabled if it doesn't cover a specific device and all the guests are trusted. Thinking a mixed scenario, where a few devices with IOMMU and a few without, then guest DMA security still could not be totally guaranteed. So users may want to disable the IOMMU, to at least gain some performance improvement from IOMMU disabled. * IOMMU disabled as a workaround when it doesn't have enough bandwidth. To be specific, in a few extreme situation, when multiple devices do DMA concurrently, these requests may exceed IOMMU's transmission capacity. * IOMMU disabled when it adds too much latency on DMA. For example, TLB may be missing in some IOMMU hardware, which may bring latency in DMA progress, so users may want to disable it in some realtime scenario. * Guest OS relies on the host memory layout "direct-map" property shall be added under the appropriate domain node, when users requesting direct-map memory mapping for the domain. Right now, direct-map is only supported when domain on Static Allocation, that is, "xen,static-mem" is also necessary in the domain configuration. Looking into related [design link]( https://lists.xenproject.org/archives/html/xen-devel/2021-05/msg00882.html) for more details. The whole design is about Static Allocation and direct-map, and this Patch Serie only covers parts of it, which are direct-map memory map. Other features will be delievered through different patch series. See https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00855.html for Domain on Static Allocation. This patch serie is based on https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00822.html --- v3 changes: - move flag XEN_DOMCTL_CDF_INTERNAL_directmap back to xen/include/xen/domain.h, to let it be only available for domain created by XEN. - name it with extra "INTERNAL" and add comments to warn developers not to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag. - reject this flag in x86'es arch_sanitise_domain_config() - add ASSERT_UNREACHABLE to catch any misuse in allocate_static_memory() and allocate_static_memory_11() - add another check of validating flag XEN_DOMCTL_CDF_INTERNAL_directmap only when CONFIG_STATIC_MEMORY is set. - simply map the CPU interface at the GPA vgic_v2_hw.cbase - drop 'cells += (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS)' - rename 'is_domain_use_host_layout()' to 'domain_use_host_layout()' --- v2 changes: - remove the introduce of internal flag - Refine is_domain_direct_mapped to check whether the flag XEN_DOMCTL_CDF_directmap is set - reword "1:1 direct-map" to just "direct-map" - split the common codes into two helpers: parse_static_mem_prop and acquire_static_memory_bank to deduce complexity. - introduce a new helper allocate_static_memory_11 for allocating static memory for direct-map guests - remove panic action since it is fine to assign a non-DMA capable device when IOMMU and direct-map both off - remove redistributor accessor - introduce new helper "is_domain_use_host_layout()" - explain why vpl011 initialization before creating its device tree node - error out if the domain is direct-mapped and the IRQ is not found - harden the code and add a check/comment when the hardware UART region is smaller than CUEST_VPL011_SIZE. Penny Zheng (4): xen/arm: introduce new helper parse_static_mem_prop and ... xen/arm: introduce direct-map for domUs xen/arm: add ASSERT_UNREACHABLE in allocate_static_memory xen/arm: gate make_gicv3_domU_node with CONFIG_GICV3 Stefano Stabellini (6): xen: introduce XEN_DOMCTL_CDF_INTERNAL_directmap xen/arm: avoid setting XEN_DOMCTL_CDF_iommu when IOMMU off xen/arm: if direct-map domain use native addresses for GICv2 xen/arm: if direct-map domain use native addresses for GICv3 xen/arm: if direct-map domain use native UART address and IRQ ... xen/docs: Document how to do passthrough without IOMMU docs/misc/arm/device-tree/booting.txt | 6 + docs/misc/arm/passthrough-noiommu.txt | 52 + xen/arch/arm/domain.c | 3 +- xen/arch/arm/domain_build.c | 310 +- xen/arch/arm/vgic-v2.c| 31 ++- xen/arch/arm/vgic-v3.c| 29 ++- xen/arch/arm/vgic/vgic-v2.c | 31 ++- xen/arch/arm/vpl011.c | 60 - xen/arch/x86/domain.c | 6 + xen/common/domain.c | 3 +- xen/include/asm-arm/domain.h | 11 +- xen/include/asm-arm/new_vgic.h| 10 + xen/include/asm-arm/vgic.h| 11 + xen/include/asm-arm/vpl011.h | 2 + xen/include/public/domctl.h | 4 + xen/include/xen/domain.h | 3 + 16 files changed, 471 insertions(+), 101 deletions(-) create mode 100644 docs/misc/arm/passthrough-noiommu.txt -- 2.25.1
[PATCH v3 05/10] xen/arm: add ASSERT_UNREACHABLE in allocate_static_memory
Helper allocate_static_memory is not meant to be reachable when built with
!CONFIG_STATIC_MEMORY, so this commit adds ASSERT_UNREACHABLE in it to catch
potential misuse.
Signed-off-by: Penny Zheng
---
v3 changes:
- new commit
---
xen/arch/arm/domain_build.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 97a5b5dedd..b6fde74d74 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -759,6 +759,7 @@ static void __init allocate_static_memory(struct domain *d,
struct kernel_info *kinfo,
const struct dt_device_node *node)
{
+ASSERT_UNREACHABLE();
}
static void __init allocate_static_memory_11(struct domain *d,
--
2.25.1
[PATCH v3 01/10] xen: introduce XEN_DOMCTL_CDF_INTERNAL_directmap
From: Stefano Stabellini
This commit introduces a new arm-specific flag
XEN_DOMCTL_CDF_INTERNAL_directmap to specify that a domain should
have its memory direct-map(guest physical address == physical address)
at domain creation.
Since this flag is only available for domain created by XEN, not exposed
to the toolstack, we name it with extra "INTERNAL" to distinguish from
other public XEN_DOMCTL_CDF_xxx flags, and add comments to warn developers
not to accidently use its bitfield when introducing new
XEN_DOMCTL_CDF_xxx flag.
Refine is_domain_direct_mapped to check whether the flag
XEN_DOMCTL_CDF_INTERNAL_directmap is set.
Signed-off-by: Penny Zheng
Signed-off-by: Stefano Stabellini
---
CC: andrew.coop...@citrix.com
CC: jbeul...@suse.com
CC: George Dunlap
CC: Ian Jackson
CC: Wei Liu
CC: "Roger Pau Monné"
---
v2 changes
- remove the introduce of internal flag
- remove flag direct_map since we already store this flag in d->options
- Refine is_domain_direct_mapped to check whether the flag
XEN_DOMCTL_CDF_directmap is set
- reword "1:1 direct-map" to just "direct-map"
---
v3 changes
- move flag back to xen/include/xen/domain.h, to let it be only available for
domain created by XEN.
- name it with extra "INTERNAL" and add comments to warn developers not
to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag.
- reject this flag in x86'es arch_sanitise_domain_config()
---
xen/arch/arm/domain.c| 3 ++-
xen/arch/arm/domain_build.c | 4 +++-
xen/arch/x86/domain.c| 6 ++
xen/common/domain.c | 3 ++-
xen/include/asm-arm/domain.h | 4 ++--
xen/include/public/domctl.h | 4
xen/include/xen/domain.h | 3 +++
7 files changed, 22 insertions(+), 5 deletions(-)
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 96e1b23550..d77265c03f 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -629,7 +629,8 @@ int arch_sanitise_domain_config(struct
xen_domctl_createdomain *config)
{
unsigned int max_vcpus;
unsigned int flags_required = (XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap);
-unsigned int flags_optional = (XEN_DOMCTL_CDF_iommu | XEN_DOMCTL_CDF_vpmu);
+unsigned int flags_optional = (XEN_DOMCTL_CDF_iommu | XEN_DOMCTL_CDF_vpmu |
+ XEN_DOMCTL_CDF_INTERNAL_directmap);
if ( (config->flags & ~flags_optional) != flags_required )
{
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 19487c79da..664c88ebe4 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3089,8 +3089,10 @@ static int __init construct_dom0(struct domain *d)
void __init create_dom0(void)
{
struct domain *dom0;
+/* DOM0 has always its memory direct-map. */
struct xen_domctl_createdomain dom0_cfg = {
-.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,
+.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
+ XEN_DOMCTL_CDF_INTERNAL_directmap,
.max_evtchn_port = -1,
.max_grant_frames = gnttab_dom0_frames(),
.max_maptrack_frames = -1,
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index ef1812dc14..eba6502218 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -692,6 +692,12 @@ int arch_sanitise_domain_config(struct
xen_domctl_createdomain *config)
return -EINVAL;
}
+if ( config->flags & XEN_DOMCTL_CDF_INTERNAL_directmap )
+{
+dprintk(XENLOG_INFO, "direct-map cannot be enabled yet\n");
+return -EINVAL;
+}
+
return 0;
}
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 56d47dd664..13ac5950bc 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -486,7 +486,8 @@ static int sanitise_domain_config(struct
xen_domctl_createdomain *config)
~(XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap |
XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off |
XEN_DOMCTL_CDF_xs_domain | XEN_DOMCTL_CDF_iommu |
- XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu) )
+ XEN_DOMCTL_CDF_nested_virt | XEN_DOMCTL_CDF_vpmu |
+ XEN_DOMCTL_CDF_INTERNAL_directmap) )
{
dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
return -EINVAL;
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 9b3647587a..4f2c3f09d4 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -29,8 +29,8 @@ enum domain_type {
#define is_64bit_domain(d) (0)
#endif
-/* The hardware domain has always its memory direct mapped. */
-#define is_domain_direct_mapped(d) is_hardware_domain(d)
+#define is_domain_direct_mapped(d) \
+(d->options & XEN_DOMCTL_CDF_INTERNAL_directmap)
struct vtimer {
struct vcpu *v;
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 1c21d4dc75..054e545c9
[PATCH v3 04/10] xen/arm: introduce direct-map for domUs
Cases where domU needs direct-map memory map:
* IOMMU not present in the system.
* IOMMU disabled if it doesn't cover a specific device and all the guests
are trusted. Thinking a mixed scenario, where a few devices with IOMMU and
a few without, then guest DMA security still could not be totally guaranteed.
So users may want to disable the IOMMU, to at least gain some performance
improvement from IOMMU disabled.
* IOMMU disabled as a workaround when it doesn't have enough bandwidth.
To be specific, in a few extreme situation, when multiple devices do DMA
concurrently, these requests may exceed IOMMU's transmission capacity.
* IOMMU disabled when it adds too much latency on DMA. For example,
TLB may be missing in some IOMMU hardware, which may bring latency in DMA
progress, so users may want to disable it in some realtime scenario.
* Guest OS relies on the host memory layout
This commit introduces a new helper allocate_static_memory_11 to allocate
static memory as guest RAM for direct-map domain.
For now, direct-map is only available when statically allocated memory is
used for the domain, that is, "xen,static-mem" must be also defined in the
domain configuration.
Signed-off-by: Penny Zheng
Signed-off-by: Stefano Stabellini
---
v2 changes:
- split the common codes into two helpers: parse_static_mem_prop and
acquire_static_memory_bank to deduce complexity.
- introduce a new helper allocate_static_memory_11 for allocating static
memory for direct-map guests
- remove redundant use "bool direct_map", to be replaced by
d_cfg.flags & XEN_DOMCTL_CDF_directmap
- remove panic action since it is fine to assign a non-DMA capable device when
IOMMU and direct-map both off
---
v3 changes:
- doc refinement
- drop the pointless gbank
- add check of the size of nr_banks shall not exceed NR_MEM_BANKS
- add ASSERT_UNREACHABLE to catch any misuse
- add another check of validating flag XEN_DOMCTL_CDF_INTERNAL_directmap only
when CONFIG_STATIC_MEMORY is set.
---
docs/misc/arm/device-tree/booting.txt | 6 ++
xen/arch/arm/domain_build.c | 105 +-
2 files changed, 108 insertions(+), 3 deletions(-)
diff --git a/docs/misc/arm/device-tree/booting.txt
b/docs/misc/arm/device-tree/booting.txt
index 71895663a4..a94125394e 100644
--- a/docs/misc/arm/device-tree/booting.txt
+++ b/docs/misc/arm/device-tree/booting.txt
@@ -182,6 +182,12 @@ with the following properties:
Both #address-cells and #size-cells need to be specified because
both sub-nodes (described shortly) have reg properties.
+- direct-map
+
+Only available when statically allocated memory is used for the domain.
+An empty property to request the memory of the domain to be
+direct-map (guest physical address == physical address).
+
Under the "xen,domain" compatible node, one or more sub-nodes are present
for the DomU kernel and ramdisk.
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 1dc728e848..97a5b5dedd 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -500,8 +500,13 @@ static bool __init append_static_memory_to_bank(struct
domain *d,
{
int res;
unsigned int nr_pages = PFN_DOWN(size);
-/* Infer next GFN. */
-gfn_t sgfn = gaddr_to_gfn(bank->start + bank->size);
+gfn_t sgfn;
+
+if ( !is_domain_direct_mapped(d) )
+/* Infer next GFN when GFN != MFN. */
+sgfn = gaddr_to_gfn(bank->start + bank->size);
+else
+sgfn = gaddr_to_gfn(mfn_to_maddr(smfn));
res = guest_physmap_add_pages(d, sgfn, smfn, nr_pages);
if ( res )
@@ -674,12 +679,94 @@ static void __init allocate_static_memory(struct domain
*d,
fail:
panic("Failed to allocate requested static memory for domain %pd.", d);
}
+
+/*
+ * Allocate static memory as RAM for one specific domain d.
+ * The static memory will be directly mapped in the guest(Guest Physical
+ * Address == Physical Address).
+ */
+static void __init allocate_static_memory_11(struct domain *d,
+ struct kernel_info *kinfo,
+ const struct dt_device_node *node)
+{
+u32 addr_cells, size_cells, reg_cells;
+unsigned int nr_banks, bank = 0;
+const __be32 *cell;
+u64 tot_size = 0;
+paddr_t pbase, psize;
+mfn_t smfn;
+int length;
+
+if ( parse_static_mem_prop(node, &addr_cells, &size_cells, &length, &cell)
)
+{
+printk(XENLOG_ERR
+ "%pd: failed to parse \"xen,static-mem\" property.\n", d);
+goto fail;
+}
+reg_cells = addr_cells + size_cells;
+nr_banks = length / (reg_cells * sizeof (u32));
+
+if ( nr_banks > NR_MEM_BANKS )
+{
+printk(XENLOG_ERR
+ "%pd: exceed max number of supported guest memory banks.\n", d);
+goto fail;
+}
+
+
[PATCH v3 10/10] xen/docs: Document how to do passthrough without IOMMU
From: Stefano Stabellini
This commit creates a new doc to document how to do passthrough without IOMMU.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
---
docs/misc/arm/passthrough-noiommu.txt | 52 +++
1 file changed, 52 insertions(+)
create mode 100644 docs/misc/arm/passthrough-noiommu.txt
diff --git a/docs/misc/arm/passthrough-noiommu.txt
b/docs/misc/arm/passthrough-noiommu.txt
new file mode 100644
index 00..3e2ef21ad7
--- /dev/null
+++ b/docs/misc/arm/passthrough-noiommu.txt
@@ -0,0 +1,52 @@
+Request Device Assignment without IOMMU support
+===
+
+*WARNING:
+Users should be aware that it is not always secure to assign a device without
+IOMMU protection.
+When the device is not protected by the IOMMU, the administrator should make
+sure that:
+ 1. The device is assigned to a trusted guest.
+ 2. Users have additional security mechanism on the platform.
+
+This document assumes that the IOMMU is absent from the system or it is
+disabled (status = "disabled" in device tree).
+
+Add xen,force-assign-without-iommu; to the device tree snippet:
+
+ethernet: ethernet@ff0e {
+ compatible = "cdns,zynqmp-gem";
+ xen,path = "/amba/ethernet@ff0e";
+ xen,reg = <0x0 0xff0e 0x1000 0x0 0xff0e>;
+ xen,force-assign-without-iommu;
+};
+
+Request 1:1 memory mapping for the domain on static allocation
+==
+
+Add a direct-map property under the appropriate /chosen/domU node which
+is also statically allocated with physical memory ranges through
+xen,static-mem property as its guest RAM.
+
+Below is an example on how to specify the 1:1 memory mapping for the domain
+on static allocation in the device-tree:
+
+/ {
+ chosen {
+ domU1 {
+ compatible = "xen,domain";
+ #address-cells = <0x2>;
+ #size-cells = <0x2>;
+ cpus = <2>;
+ memory = <0x0 0x8>;
+ #xen,static-mem-address-cells = <0x1>;
+ #xen,static-mem-size-cells = <0x1>;
+ xen,static-mem = <0x3000 0x2000>;
+ direct-map;
+ ...
+ };
+ };
+};
+
+Besides reserving a 512MB region starting at the host physical address
+0x3000 to DomU1, it also requests 1:1 memory mapping.
--
2.25.1
[PATCH v3 06/10] xen/arm: if direct-map domain use native addresses for GICv2
From: Stefano Stabellini
Today we use native addresses to map the GICv2 for Dom0 and fixed
addresses for DomUs.
This patch changes the behavior so that native addresses are used for
all domains that are direct-mapped.
NEW VGIC has different naming schemes, like referring distributor base
address as vgic_dist_base, other than the dbase. So this patch also introduces
vgic_dist_base/vgic_cpu_base accessor to access correct distributor base
address/cpu interface base address on varied scenarios,
Signed-off-by: Penny Zheng
Signed-off-by: Stefano Stabellini
---
v2 changes
- combine all changes in patch 4-6 here
---
v3 changes
- refine comment message
- add a comment explaining how the 38 was found of "char buf[38]"
- simply map the CPU interface at the GPA vgic_v2_hw.cbase
- remove a spurious change
---
xen/arch/arm/domain_build.c| 11 ---
xen/arch/arm/vgic-v2.c | 31 ++-
xen/arch/arm/vgic/vgic-v2.c| 31 ++-
xen/include/asm-arm/new_vgic.h | 10 ++
xen/include/asm-arm/vgic.h | 11 +++
5 files changed, 73 insertions(+), 21 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index b6fde74d74..c419a4b2cc 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2230,8 +2230,13 @@ static int __init make_gicv2_domU_node(struct
kernel_info *kinfo)
int res = 0;
__be32 reg[(GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) * 2];
__be32 *cells;
+const struct domain *d = kinfo->d;
+/* Placeholder for interrupt-controller@ + a 64-bit number + \0 */
+char buf[38];
-res = fdt_begin_node(fdt,
"interrupt-controller@"__stringify(GUEST_GICD_BASE));
+snprintf(buf, sizeof(buf), "interrupt-controller@%"PRIx64,
+ vgic_dist_base(&d->arch.vgic));
+res = fdt_begin_node(fdt, buf);
if ( res )
return res;
@@ -2253,9 +2258,9 @@ static int __init make_gicv2_domU_node(struct kernel_info
*kinfo)
cells = ®[0];
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICD_BASE, GUEST_GICD_SIZE);
+ vgic_dist_base(&d->arch.vgic), GUEST_GICD_SIZE);
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICC_BASE, GUEST_GICC_SIZE);
+ vgic_cpu_base(&d->arch.vgic), GUEST_GICC_SIZE);
res = fdt_property(fdt, "reg", reg, sizeof(reg));
if (res)
diff --git a/xen/arch/arm/vgic-v2.c b/xen/arch/arm/vgic-v2.c
index 589c033eda..6f5492e30e 100644
--- a/xen/arch/arm/vgic-v2.c
+++ b/xen/arch/arm/vgic-v2.c
@@ -654,13 +654,10 @@ static int vgic_v2_vcpu_init(struct vcpu *v)
static int vgic_v2_domain_init(struct domain *d)
{
int ret;
-paddr_t cbase, csize;
+paddr_t csize;
paddr_t vbase;
-/*
- * The hardware domain gets the hardware address.
- * Guests get the virtual platform layout.
- */
+/* The hardware domain gets the hardware address. */
if ( is_hardware_domain(d) )
{
d->arch.vgic.dbase = vgic_v2_hw.dbase;
@@ -671,10 +668,26 @@ static int vgic_v2_domain_init(struct domain *d)
* Note that we assume the size of the CPU interface is always
* aligned to PAGE_SIZE.
*/
-cbase = vgic_v2_hw.cbase;
+d->arch.vgic.cbase = vgic_v2_hw.cbase;
csize = vgic_v2_hw.csize;
vbase = vgic_v2_hw.vbase;
}
+else if ( is_domain_direct_mapped(d) )
+{
+/*
+ * For all the direct-mapped domain other than the hardware domain,
+ * we only map a 8kB CPU interface but we make sure it is at a
+ * location occupied by the physical GIC in the host device tree.
+ *
+ * We need to add an offset to the virtual CPU interface base
+ * address when the GIC is aliased to get a 8kB contiguous
+ * region.
+ */
+d->arch.vgic.dbase = vgic_v2_hw.dbase;
+d->arch.vgic.cbase = vgic_v2_hw.cbase;
+csize = GUEST_GICC_SIZE;
+vbase = vgic_v2_hw.vbase + vgic_v2_hw.aliased_offset;
+}
else
{
d->arch.vgic.dbase = GUEST_GICD_BASE;
@@ -685,7 +698,7 @@ static int vgic_v2_domain_init(struct domain *d)
* region.
*/
BUILD_BUG_ON(GUEST_GICC_SIZE != SZ_8K);
-cbase = GUEST_GICC_BASE;
+d->arch.vgic.cbase = GUEST_GICC_BASE;
csize = GUEST_GICC_SIZE;
vbase = vgic_v2_hw.vbase + vgic_v2_hw.aliased_offset;
}
@@ -694,8 +707,8 @@ static int vgic_v2_domain_init(struct domain *d)
* Map the gic virtual cpu interface in the gic cpu interface
* region of the guest.
*/
-ret = map_mmio_regions(d, gaddr_to_gfn(cbase), csize / PAGE_SIZE,
- maddr_to_mfn(vbase));
+ret = map_mmio_
[PATCH v3 07/10] xen/arm: gate make_gicv3_domU_node with CONFIG_GICV3
This commit gates function make_gicv3_domU_node with CONFIG_GICV3, and
also adds ASSERT_UNREACHABLE to catch any misuse.
Signed-off-by: Penny Zheng
---
v3 changes:
- new commit
---
xen/arch/arm/domain_build.c | 7 +++
1 file changed, 7 insertions(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index c419a4b2cc..24f3edf069 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2279,6 +2279,7 @@ static int __init make_gicv2_domU_node(struct kernel_info
*kinfo)
return res;
}
+#ifdef CONFIG_GICV3
static int __init make_gicv3_domU_node(struct kernel_info *kinfo)
{
void *fdt = kinfo->fdt;
@@ -2328,6 +2329,12 @@ static int __init make_gicv3_domU_node(struct
kernel_info *kinfo)
return res;
}
+#else
+static int __init make_gicv3_domU_node(struct kernel_info *kinfo)
+{
+ASSERT_UNREACHABLE();
+}
+#endif
static int __init make_gic_domU_node(struct kernel_info *kinfo)
{
--
2.25.1
[PATCH v3 08/10] xen/arm: if direct-map domain use native addresses for GICv3
From: Stefano Stabellini
Today we use native addresses to map the GICv3 for Dom0 and fixed
addresses for DomUs.
This patch changes the behavior so that native addresses are used for
all domain which is using the host memory layout
Considering that DOM0 may not always be directly mapped in the future,
this patch introduces a new helper "domain_use_host_layout()" that
wraps both two check "is_domain_direct_mapped(d) || is_hardware_domain(d)"
for more flexible usage.
Signed-off-by: Penny Zheng
Signed-off-by: Stefano Stabellini
---
v2 changes:
- remove redistributor accessor
- introduce new helper "is_domain_use_host_layout()"
- comment fix
---
v3 changes:
- the comment on top of 'buf' to explain how 38 was found
- fix res getting overwritten
- drop 'cells += (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS)'
- free 'reg' right way
- fix comment
- rename 'is_domain_use_host_layout()' to 'domain_use_host_layout()'
---
xen/arch/arm/domain_build.c | 37 +++-
xen/arch/arm/vgic-v3.c | 29
xen/include/asm-arm/domain.h | 7 +++
3 files changed, 52 insertions(+), 21 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 24f3edf069..61fd374c5d 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2284,10 +2284,16 @@ static int __init make_gicv3_domU_node(struct
kernel_info *kinfo)
{
void *fdt = kinfo->fdt;
int res = 0;
-__be32 reg[(GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) * 2];
-__be32 *cells;
+__be32 *reg;
+const struct domain *d = kinfo->d;
+/* Placeholder for interrupt-controller@ + a 64-bit number + \0 */
+char buf[38];
+unsigned int i, len = 0;
-res = fdt_begin_node(fdt,
"interrupt-controller@"__stringify(GUEST_GICV3_GICD_BASE));
+snprintf(buf, sizeof(buf), "interrupt-controller@%"PRIx64,
+ vgic_dist_base(&d->arch.vgic));
+
+res = fdt_begin_node(fdt, buf);
if ( res )
return res;
@@ -2307,13 +2313,26 @@ static int __init make_gicv3_domU_node(struct
kernel_info *kinfo)
if ( res )
return res;
-cells = ®[0];
-dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICV3_GICD_BASE, GUEST_GICV3_GICD_SIZE);
-dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICV3_GICR0_BASE, GUEST_GICV3_GICR0_SIZE);
+/* reg specifies all re-distributors and Distributor. */
+len = (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) *
+ (d->arch.vgic.nr_regions + 1) * sizeof(__be32);
+reg = xmalloc_bytes(len);
+if ( reg == NULL )
+return -ENOMEM;
-res = fdt_property(fdt, "reg", reg, sizeof(reg));
+dt_child_set_range(®, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
+ vgic_dist_base(&d->arch.vgic), GUEST_GICV3_GICD_SIZE);
+
+for ( i = 0; i < d->arch.vgic.nr_regions; i++)
+{
+dt_child_set_range(®,
+ GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
+ d->arch.vgic.rdist_regions[i].base,
+ d->arch.vgic.rdist_regions[i].size);
+}
+
+res = fdt_property(fdt, "reg", reg, len);
+xfree(reg);
if (res)
return res;
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index 65bb7991a6..181b66513d 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -1640,14 +1640,15 @@ static inline unsigned int
vgic_v3_max_rdist_count(struct domain *d)
* Normally there is only one GICv3 redistributor region.
* The GICv3 DT binding provisions for multiple regions, since there are
* platforms out there which need those (multi-socket systems).
- * For Dom0 we have to live with the MMIO layout the hardware provides,
- * so we have to copy the multiple regions - as the first region may not
- * provide enough space to hold all redistributors we need.
+ * For domain using the host memory layout, we have to live with the MMIO
+ * layout the hardware provides, so we have to copy the multiple regions
+ * - as the first region may not provide enough space to hold all
+ * redistributors we need.
* However DomU get a constructed memory map, so we can go with
* the architected single redistributor region.
*/
-return is_hardware_domain(d) ? vgic_v3_hw.nr_rdist_regions :
- GUEST_GICV3_RDIST_REGIONS;
+return domain_use_host_layout(d) ? vgic_v3_hw.nr_rdist_regions :
+ GUEST_GICV3_RDIST_REGIONS;
}
static int vgic_v3_domain_init(struct domain *d)
@@ -1669,10 +1670,14 @@ static int vgic_v3_domain_init(struct domain *d)
radix_tre
[PATCH v3 03/10] xen/arm: introduce new helper parse_static_mem_prop and acquire_static_memory_bank
Later, we will introduce allocate_static_memory_11 for allocating static
memory for direct-map domains, and it will share a lot common codes with
the existing allocate_static_memory.
In order not to bring a lot of duplicate codes, and also to make the whole
code more readable, this commit extracts common codes into two new helpers
parse_static_mem_prop and acquire_static_memory_bank.
Signed-off-by: Penny Zheng
---
v3 changes:
- new commit to move the split off of the code outside in a separate patch
---
xen/arch/arm/domain_build.c | 100 +++-
1 file changed, 64 insertions(+), 36 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 7a063f62fe..1dc728e848 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -515,12 +515,69 @@ static bool __init append_static_memory_to_bank(struct
domain *d,
return true;
}
+static mfn_t __init acquire_static_memory_bank(struct domain *d,
+ const __be32 **cell,
+ u32 addr_cells, u32 size_cells,
+ paddr_t *pbase, paddr_t *psize)
+{
+mfn_t smfn;
+int res;
+
+device_tree_get_reg(cell, addr_cells, size_cells, pbase, psize);
+ASSERT(IS_ALIGNED(*pbase, PAGE_SIZE) && IS_ALIGNED(*psize, PAGE_SIZE));
+if ( PFN_DOWN(*psize) > UINT_MAX )
+{
+printk(XENLOG_ERR "%pd: static memory size too large: %#"PRIpaddr,
+ d, *psize);
+return INVALID_MFN;
+}
+
+smfn = maddr_to_mfn(*pbase);
+res = acquire_domstatic_pages(d, smfn, PFN_DOWN(*psize), 0);
+if ( res )
+{
+printk(XENLOG_ERR
+ "%pd: failed to acquire static memory: %d.\n", d, res);
+return INVALID_MFN;
+}
+
+return smfn;
+}
+
+static int __init parse_static_mem_prop(const struct dt_device_node *node,
+u32 *addr_cells, u32 *size_cells,
+int *length, const __be32 **cell)
+{
+const struct dt_property *prop;
+
+prop = dt_find_property(node, "xen,static-mem", NULL);
+if ( !dt_property_read_u32(node, "#xen,static-mem-address-cells",
+ addr_cells) )
+{
+printk(XENLOG_ERR
+ "failed to read \"#xen,static-mem-address-cells\".\n");
+return -EINVAL;
+}
+
+if ( !dt_property_read_u32(node, "#xen,static-mem-size-cells",
+ size_cells) )
+{
+printk(XENLOG_ERR
+ "failed to read \"#xen,static-mem-size-cells\".\n");
+return -EINVAL;
+}
+
+*cell = (const __be32 *)prop->value;
+*length = prop->length;
+
+return 0;
+}
+
/* Allocate memory from static memory as RAM for one specific domain d. */
static void __init allocate_static_memory(struct domain *d,
struct kernel_info *kinfo,
const struct dt_device_node *node)
{
-const struct dt_property *prop;
u32 addr_cells, size_cells, reg_cells;
unsigned int nr_banks, gbank, bank = 0;
const uint64_t rambase[] = GUEST_RAM_BANK_BASES;
@@ -529,24 +586,10 @@ static void __init allocate_static_memory(struct domain
*d,
u64 tot_size = 0;
paddr_t pbase, psize, gsize;
mfn_t smfn;
-int res;
-
-prop = dt_find_property(node, "xen,static-mem", NULL);
-if ( !dt_property_read_u32(node, "#xen,static-mem-address-cells",
- &addr_cells) )
-{
-printk(XENLOG_ERR
- "%pd: failed to read \"#xen,static-mem-address-cells\".\n", d);
-goto fail;
-}
+int length;
-if ( !dt_property_read_u32(node, "#xen,static-mem-size-cells",
- &size_cells) )
-{
-printk(XENLOG_ERR
- "%pd: failed to read \"#xen,static-mem-size-cells\".\n", d);
+if ( parse_static_mem_prop(node, &addr_cells, &size_cells, &length, &cell)
)
goto fail;
-}
reg_cells = addr_cells + size_cells;
/*
@@ -557,29 +600,14 @@ static void __init allocate_static_memory(struct domain
*d,
gbank = 0;
gsize = ramsize[gbank];
kinfo->mem.bank[gbank].start = rambase[gbank];
-
-cell = (const __be32 *)prop->value;
-nr_banks = (prop->length) / (reg_cells * sizeof (u32));
+nr_banks = length / (reg_cells * sizeof (u32));
for ( ; bank < nr_banks; bank++ )
{
-device_tree_get_reg(&cell, addr_cells, size_cells, &pbase, &psize);
-ASSERT(IS_ALIGNED(pbase, PAGE_SIZE) && IS_ALIGNED(psize, PAGE_SIZE));
-
-if ( PFN_DOWN(psize) > UINT_MAX )
-{
-
[PATCH v3 09/10] xen/arm: if direct-map domain use native UART address and IRQ number for vPL011
From: Stefano Stabellini
We always use a fix address to map the vPL011 to domains. The address
could be a problem for direct-map domains.
So, for domains that are directly mapped, reuse the address of the
physical UART on the platform to avoid potential clashes.
Do the same for the virtual IRQ number: instead of always using
GUEST_VPL011_SPI, try to reuse the physical SPI number if possible.
Signed-off-by: Penny Zheng
Signed-off-by: Stefano Stabellini
---
v2 changes:
- explain why vpl011 initialization before creating its device tree node
- error out if the domain is direct-mapped and the IRQ is not found
- harden the code and add a check/comment when the hardware UART region
is smaller than GUEST_VPL011_SIZE.
---
v3 changes:
- explain how the '27' was found for 'buf'
- fix checking before dereferencing
- refine comment message
---
xen/arch/arm/domain_build.c | 42 -
xen/arch/arm/vpl011.c| 60 +++-
xen/include/asm-arm/vpl011.h | 2 ++
3 files changed, 90 insertions(+), 14 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 61fd374c5d..871c7114ae 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -30,6 +30,7 @@
#include
#include
+#include
static unsigned int __initdata opt_dom0_max_vcpus;
integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
@@ -2376,8 +2377,12 @@ static int __init make_vpl011_uart_node(struct
kernel_info *kinfo)
gic_interrupt_t intr;
__be32 reg[GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS];
__be32 *cells;
+struct domain *d = kinfo->d;
+/* Placeholder for sbsa-uart@ + a 64-bit number + \0 */
+char buf[27];
-res = fdt_begin_node(fdt, "sbsa-uart@"__stringify(GUEST_PL011_BASE));
+snprintf(buf, sizeof(buf), "sbsa-uart@%"PRIx64, d->arch.vpl011.base_addr);
+res = fdt_begin_node(fdt, buf);
if ( res )
return res;
@@ -2387,14 +2392,14 @@ static int __init make_vpl011_uart_node(struct
kernel_info *kinfo)
cells = ®[0];
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS,
- GUEST_ROOT_SIZE_CELLS, GUEST_PL011_BASE,
+ GUEST_ROOT_SIZE_CELLS, d->arch.vpl011.base_addr,
GUEST_PL011_SIZE);
res = fdt_property(fdt, "reg", reg, sizeof(reg));
if ( res )
return res;
-set_interrupt(intr, GUEST_VPL011_SPI, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
+set_interrupt(intr, d->arch.vpl011.virq, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
res = fdt_property(fdt, "interrupts", intr, sizeof (intr));
if ( res )
@@ -3109,6 +3114,14 @@ static int __init construct_domU(struct domain *d,
allocate_static_memory(d, &kinfo, node);
}
+/*
+ * Base address and irq number are needed when creating vpl011 device
+ * tree node in prepare_dtb_domU, so initialization on related variables
+ * shall be done first.
+ */
+if ( kinfo.vpl011 )
+rc = domain_vpl011_init(d, NULL);
+
rc = prepare_dtb_domU(d, &kinfo);
if ( rc < 0 )
return rc;
@@ -3117,9 +3130,6 @@ static int __init construct_domU(struct domain *d,
if ( rc < 0 )
return rc;
-if ( kinfo.vpl011 )
-rc = domain_vpl011_init(d, NULL);
-
return rc;
}
@@ -3161,15 +3171,33 @@ void __init create_domUs(void)
if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
{
+unsigned int vpl011_virq = GUEST_VPL011_SPI;
+
d_cfg.arch.nr_spis = gic_number_lines() - 32;
+/*
+ * The VPL011 virq is GUEST_VPL011_SPI, unless direct-map is
+ * set, in which case it'll match the hardware.
+ *
+ * Since here the domain is not totally built, we need to
+ * open-code the logic to find the vIRQ. and the logic here
+ * is consistent with the ones in domain_vpl011_init().
+ */
+if ( d_cfg.flags & XEN_DOMCTL_CDF_INTERNAL_directmap )
+{
+vpl011_virq = serial_irq(SERHND_DTUART);
+if ( vpl011_virq < 0 )
+panic("Error getting IRQ number for this serial port %d\n",
+ SERHND_DTUART);
+}
+
/*
* vpl011 uses one emulated SPI. If vpl011 is requested, make
* sure that we allocate enough SPIs for it.
*/
if ( dt_property_read_bool(node, "vpl011") )
d_cfg.arch.nr_spis = MAX(d_cfg.arch.nr_spis,
- GUEST_VPL011_SPI - 32 + 1);
+ vpl011_virq - 32 + 1);
}
/*
diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
index 895f436cc4..65610bccaf
[PATCH] design: design doc for shared memory on a dom0less system
This commit provides a design doc for static shared memory
on a dom0less system.
Signed-off-by: Penny Zheng
---
design/shm-dom0less.md | 182 +
1 file changed, 182 insertions(+)
create mode 100644 design/shm-dom0less.md
diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md
new file mode 100644
index 000..b46199d
--- /dev/null
+++ b/design/shm-dom0less.md
@@ -0,0 +1,182 @@
+# Static Shared Memory between domains on a dom0less system
+
+This design aims to provide an overview of the new feature: setting up static
+shared memory between domains on a dom0less system, through device tree
+configuration.
+
+The new feature is driven by the need of finding a way to build up
+communication channels on dom0less system, since the legacy ways including
+grant table, etc are all absent there.
+
+It was inspired by the patch serie of "xl/libxl-based shared memory", see
+[1] for more details.
+
+# Static Shared Memory Device Tree Configuration
+
+The static shared memory device tree nodes allow users to statically set up
+shared memory among a group of dom0less DomUs and Dom0, enabling domains
+to do shm-based communication.
+
+- compatible
+
+"xen,domain-shared-memory-v1"
+
+- xen,shm-id
+
+An u32 value represents the unique identifier of the shared memory region.
+User valuing per shared memory region shall follow the ascending order,
+starting from xen,shm-id = <0x0>, to the maximum identifier
+xen,shm-id = <0x126>. The special xen,shm-id = <0x127> is reserved for
+INVALID_SHMID.
+
+- xen,shared-mem
+
+An array takes a physical address, which is the base address of the
+shared memory region in host physical address space, a size, and a guest
+physical address, as the target address of the mapping.
+
+- role(Optional)
+
+A string property specifying the ownership of a shared memory region,
+the value must be one of the following: "owner", or "borrower"
+A shared memory region could be explicitly backed by one domain, which is
+called "owner domain", and all the other domains who are also sharing
+this region are called "borrower domain".
+If not specified, the default value is "borrower" and owner is
+"dom_shared", a system domain.
+
+## Example
+
+chosen {
+#address-cells = <0x1>;
+#size-cells = <0x1>;
+xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";
+
+..
+
+/* this is for Dom0 */
+dom0-shared-mem@1000 {
+compatible = "xen,domain-shared-memory-v1";
+xen,shm-id = <0x0>;
+role = "owner";
+xen,shared-mem = <0x1000 0x1000 0x1000>;
+}
+
+domU1 {
+compatible = "xen,domain";
+#address-cells = <0x1>;
+#size-cells = <0x1>;
+memory = <0 131072>;
+cpus = <2>;
+vpl011;
+
+/*
+ * shared memory region identified as 0x0(xen,shm-id = <0x0>)
+ * shared between dom0.
+ */
+domU1-shared-mem@1000 {
+compatible = "xen,domain-shared-memory-v1";
+xen,shm-id = <0x0>;
+role = "borrower";
+xen,shared-mem = <0x1000 0x1000 0x5000>;
+}
+
+domU1-shared-mem@5000 {
+compatible = "xen,domain-shared-memory-v1";
+xen,shm-id = <0x1>;
+xen,shared-mem = <0x5000 0x2000 0x6000>;
+}
+
+..
+
+};
+
+domU2 {
+compatible = "xen,domain";
+#address-cells = <0x1>;
+#size-cells = <0x1>;
+memory = <0 65536>;
+cpus = <1>;
+
+/*
+ * shared memory region identified as 0x1(xen,shm-id = <0x1>)
+ * shared between domU1.
+ */
+domU2-shared-mem@5000 {
+compatible = "xen,domain-shared-memory-v1";
+xen,shm-id = <0x1>;
+xen,shared-mem = <0x5000 0x2000 0x7000>;
+}
+
+..
+};
+};
+
+It is the example of two static shared memory regions.
+
+In terms of shared memory region identified as 0x0, host physical address
+starting at 0x1000 of 256MB will be reserved to be shared between Dom0
+and DomU1. It will get mapped at 0x1000 in Dom0 guest physical address
+space, and at 0x5000 in DomU1 guest physical address space. Dom0 is the
+owner domain, and domU1 is the borrower domain.
+
+And in terms of shared memory region identified as 0x1, host physical address
+starting at 0x5000 of 512MB will be reserved to be shared between DomU1
+and DomU2. It will get mapped at 0x6000 in DomU1 guest physical address
+space, and at 0x7000 in DomU2 guest physical
[PATCH v5 00/11] direct-map memory map
Cases where domU needs direct-map memory map: * IOMMU not present in the system. * IOMMU disabled if it doesn't cover a specific device and all the guests are trusted. Thinking a mixed scenario, where a few devices with IOMMU and a few without, then guest DMA security still could not be totally guaranteed. So users may want to disable the IOMMU, to at least gain some performance improvement from IOMMU disabled. * IOMMU disabled as a workaround when it doesn't have enough bandwidth. To be specific, in a few extreme situation, when multiple devices do DMA concurrently, these requests may exceed IOMMU's transmission capacity. * IOMMU disabled when it adds too much latency on DMA. For example, TLB may be missing in some IOMMU hardware, which may bring latency in DMA progress, so users may want to disable it in some realtime scenario. * Guest OS relies on the host memory layout "direct-map" property shall be added under the appropriate domain node, when users requesting direct-map memory mapping for the domain. Right now, direct-map is only supported when domain on Static Allocation, that is, "xen,static-mem" is also necessary in the domain configuration. Looking into related [design link]( https://lists.xenproject.org/archives/html/xen-devel/2021-05/msg00882.html) for more details. The whole design is about Static Allocation and direct-map, and this Patch Serie only covers parts of it, which are direct-map memory map. Other features will be delievered through different patch series. See https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00855.html for Domain on Static Allocation. This patch serie is based on https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00822.html\ --- v5 changes: - remove const constraint and strict "static allocation" check - fix coding style --- v4 changes: - introduce internal const CDF_xxx flags for domain creation - introduce internal flag CDF_privileged - introduce new internal flag CDF_directmap - add a directmap flag under struct arch_domain and use it to reimplement is_domain_direct_mapped. - expand arch_domain_create/domain_create to include internal-only parameter "const unsigned int flags" - use mfn_eq() instead, because it is the only value used to indicate there is an error and this is more lightweight than mfn_valid() - rename function allocate_static_memory_11() to assign_static_memory_11() to make clear there is actually no allocation done. Instead we are only mapping pre-defined host regions to pre-defined guest regions. - remove tot_size to directly substract psize from kinfo->unassigned_mem - check kinfo->unassigned_mem doesn't underflow or overflow - remove nested if/else - remove ASSERT_UNREACHABLE() to avoid breaking compilation on prod build with CONFIG_GICV3=n - comment and commit message refinement --- v3 changes: - move flag XEN_DOMCTL_CDF_INTERNAL_directmap back to xen/include/xen/domain.h, to let it be only available for domain created by XEN. - name it with extra "INTERNAL" and add comments to warn developers not to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag. - reject this flag in x86'es arch_sanitise_domain_config() - add ASSERT_UNREACHABLE to catch any misuse in allocate_static_memory() and allocate_static_memory_11() - add another check of validating flag XEN_DOMCTL_CDF_INTERNAL_directmap only when CONFIG_STATIC_MEMORY is set. - simply map the CPU interface at the GPA vgic_v2_hw.cbase - drop 'cells += (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS)' - rename 'is_domain_use_host_layout()' to 'domain_use_host_layout()' --- v2 changes: - remove the introduce of internal flag - Refine is_domain_direct_mapped to check whether the flag XEN_DOMCTL_CDF_directmap is set - reword "1:1 direct-map" to just "direct-map" - split the common codes into two helpers: parse_static_mem_prop and acquire_static_memory_bank to deduce complexity. - introduce a new helper allocate_static_memory_11 for allocating static memory for direct-map guests - remove panic action since it is fine to assign a non-DMA capable device when IOMMU and direct-map both off - remove redistributor accessor - introduce new helper "is_domain_use_host_layout()" - explain why vpl011 initialization before creating its device tree node - error out if the domain is direct-mapped and the IRQ is not found - harden the code and add a check/comment when the hardware UART region is smaller than CUEST_VPL011_SIZE. Penny Zheng (4): xen/arm: introduce new helper parse_static_mem_prop and acquire_static_memory_bank xen/arm: introduce direct-map for domUs xen/arm: add ASSERT_UNREACHABLE in allocate_static_memory xen/arm: gate make_gicv3_domU_node with CONFIG_GICV3 Stefano Stabellini (7): xen: introduce internal CDF_xxx flags for domain creation xen: introduce CDF_directmap xen/a
[PATCH v5 01/11] xen: introduce internal CDF_xxx flags for domain creation
From: Stefano Stabellini
We are passing an internal-only boolean flag at domain creation to
specify whether we want the domain to be privileged (i.e. dom0) or
not. Another flag will be introduced later in this series.
This commit extends original "boolean" to an "unsigned int" covering both
the existing "is_priv" and our new "directmap", which will be introduced later.
To make visible the relationship, we name the respective constants CDF_xxx
(with no XEN_DOMCTL_ prefix) to represent the difference with the public
constants XEN_DOMCTL_CDF_xxx.
Allocate bit 0 as CDF_privileged: whether a domain is privileged or not.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Reviewed-by: Jan Beulich
Tested-by: Stefano Stabellini
---
v4 changes:
- new commit
---
v5 changes
- remove const constraint
---
xen/arch/arm/domain_build.c | 4 ++--
xen/arch/x86/setup.c| 2 +-
xen/common/domain.c | 10 +-
xen/common/sched/core.c | 2 +-
xen/include/xen/domain.h| 4
xen/include/xen/sched.h | 2 +-
6 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 6931c022a2..0fab8604de 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3058,7 +3058,7 @@ void __init create_domUs(void)
* very important to use the pre-increment operator to call
* domain_create() with a domid > 0. (domid == 0 is reserved for Dom0)
*/
-d = domain_create(++max_init_domid, &d_cfg, false);
+d = domain_create(++max_init_domid, &d_cfg, 0);
if ( IS_ERR(d) )
panic("Error creating domain %s\n", dt_node_name(node));
@@ -3160,7 +3160,7 @@ void __init create_dom0(void)
if ( iommu_enabled )
dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
-dom0 = domain_create(0, &dom0_cfg, true);
+dom0 = domain_create(0, &dom0_cfg, CDF_privileged);
if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
panic("Error creating domain 0\n");
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index e716005145..a14271488c 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -789,7 +789,7 @@ static struct domain *__init create_dom0(const module_t
*image,
/* Create initial domain. Not d0 for pvshim. */
domid = get_initial_domain_id();
-d = domain_create(domid, &dom0_cfg, !pv_shim);
+d = domain_create(domid, &dom0_cfg, pv_shim ? 0 : CDF_privileged);
if ( IS_ERR(d) )
panic("Error creating d%u: %ld\n", domid, PTR_ERR(d));
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 2048ebad86..a79103e04a 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -552,7 +552,7 @@ static int sanitise_domain_config(struct
xen_domctl_createdomain *config)
struct domain *domain_create(domid_t domid,
struct xen_domctl_createdomain *config,
- bool is_priv)
+ unsigned int flags)
{
struct domain *d, **pd, *old_hwdom = NULL;
enum { INIT_watchdog = 1u<<1,
@@ -578,7 +578,7 @@ struct domain *domain_create(domid_t domid,
}
/* Sort out our idea of is_control_domain(). */
-d->is_privileged = is_priv;
+d->is_privileged = flags & CDF_privileged;
/* Sort out our idea of is_hardware_domain(). */
if ( domid == 0 || domid == hardware_domid )
@@ -772,7 +772,7 @@ void __init setup_system_domains(void)
* Hidden PCI devices will also be associated with this domain
* (but be [partly] controlled by Dom0 nevertheless).
*/
-dom_xen = domain_create(DOMID_XEN, NULL, false);
+dom_xen = domain_create(DOMID_XEN, NULL, 0);
if ( IS_ERR(dom_xen) )
panic("Failed to create d[XEN]: %ld\n", PTR_ERR(dom_xen));
@@ -782,7 +782,7 @@ void __init setup_system_domains(void)
* array. Mappings occur at the priv of the caller.
* Quarantined PCI devices will be associated with this domain.
*/
-dom_io = domain_create(DOMID_IO, NULL, false);
+dom_io = domain_create(DOMID_IO, NULL, 0);
if ( IS_ERR(dom_io) )
panic("Failed to create d[IO]: %ld\n", PTR_ERR(dom_io));
@@ -791,7 +791,7 @@ void __init setup_system_domains(void)
* Initialise our COW domain.
* This domain owns sharable pages.
*/
-dom_cow = domain_create(DOMID_COW, NULL, false);
+dom_cow = domain_create(DOMID_COW, NULL, 0);
if ( IS_ERR(dom_cow) )
panic("Failed to create d[COW]: %ld\n", PTR_ERR(dom_cow));
#endif
diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c
index 8f4b1ca10d..f5c819349b 100644
--- a/xen/common/sched/core.c
+++ b/xen/common/sched/core.c
@@ -3021,7 +3021,7 @@ void __init scheduler_init(void)
sched_ratelimit_us = SCHED_DEFAULT_RATELIMIT_US
[PATCH v5 04/11] xen/arm: introduce new helper parse_static_mem_prop and acquire_static_memory_bank
Later, we will introduce assign_static_memory_11 for allocating static
memory for direct-map domains, and it will share a lot common codes with
the existing allocate_static_memory.
In order not to bring a lot of duplicate codes, and also to make the whole
code more readable, this commit extracts common codes into two new helpers
parse_static_mem_prop and acquire_static_memory_bank.
Signed-off-by: Penny Zheng
Reviewed-by: Stefano Stabellini
Tested-by: Stefano Stabellini
---
v3 changes:
- new commit, split from the original "[PATCH v2 2/6] xen/arm: introduce
direct-map for domUs"
---
v4 changes
- explain briefly in the commit message why we want to do device assignment
without IOMMU.
---
v5 changes
- fix coding style
---
xen/arch/arm/domain_build.c | 100 +++-
1 file changed, 64 insertions(+), 36 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index c1e8c99f64..e61d2d53ba 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -509,12 +509,69 @@ static bool __init append_static_memory_to_bank(struct
domain *d,
return true;
}
+static mfn_t __init acquire_static_memory_bank(struct domain *d,
+ const __be32 **cell,
+ u32 addr_cells, u32 size_cells,
+ paddr_t *pbase, paddr_t *psize)
+{
+mfn_t smfn;
+int res;
+
+device_tree_get_reg(cell, addr_cells, size_cells, pbase, psize);
+ASSERT(IS_ALIGNED(*pbase, PAGE_SIZE) && IS_ALIGNED(*psize, PAGE_SIZE));
+if ( PFN_DOWN(*psize) > UINT_MAX )
+{
+printk(XENLOG_ERR "%pd: static memory size too large: %#"PRIpaddr,
+ d, *psize);
+return INVALID_MFN;
+}
+
+smfn = maddr_to_mfn(*pbase);
+res = acquire_domstatic_pages(d, smfn, PFN_DOWN(*psize), 0);
+if ( res )
+{
+printk(XENLOG_ERR
+ "%pd: failed to acquire static memory: %d.\n", d, res);
+return INVALID_MFN;
+}
+
+return smfn;
+}
+
+static int __init parse_static_mem_prop(const struct dt_device_node *node,
+u32 *addr_cells, u32 *size_cells,
+int *length, const __be32 **cell)
+{
+const struct dt_property *prop;
+
+prop = dt_find_property(node, "xen,static-mem", NULL);
+if ( !dt_property_read_u32(node, "#xen,static-mem-address-cells",
+ addr_cells) )
+{
+printk(XENLOG_ERR
+ "failed to read \"#xen,static-mem-address-cells\".\n");
+return -EINVAL;
+}
+
+if ( !dt_property_read_u32(node, "#xen,static-mem-size-cells",
+ size_cells) )
+{
+printk(XENLOG_ERR
+ "failed to read \"#xen,static-mem-size-cells\".\n");
+return -EINVAL;
+}
+
+*cell = (const __be32 *)prop->value;
+*length = prop->length;
+
+return 0;
+}
+
/* Allocate memory from static memory as RAM for one specific domain d. */
static void __init allocate_static_memory(struct domain *d,
struct kernel_info *kinfo,
const struct dt_device_node *node)
{
-const struct dt_property *prop;
u32 addr_cells, size_cells, reg_cells;
unsigned int nr_banks, gbank, bank = 0;
const uint64_t rambase[] = GUEST_RAM_BANK_BASES;
@@ -523,24 +580,10 @@ static void __init allocate_static_memory(struct domain
*d,
u64 tot_size = 0;
paddr_t pbase, psize, gsize;
mfn_t smfn;
-int res;
-
-prop = dt_find_property(node, "xen,static-mem", NULL);
-if ( !dt_property_read_u32(node, "#xen,static-mem-address-cells",
- &addr_cells) )
-{
-printk(XENLOG_ERR
- "%pd: failed to read \"#xen,static-mem-address-cells\".\n", d);
-goto fail;
-}
+int length;
-if ( !dt_property_read_u32(node, "#xen,static-mem-size-cells",
- &size_cells) )
-{
-printk(XENLOG_ERR
- "%pd: failed to read \"#xen,static-mem-size-cells\".\n", d);
+if ( parse_static_mem_prop(node, &addr_cells, &size_cells, &length, &cell)
)
goto fail;
-}
reg_cells = addr_cells + size_cells;
/*
@@ -551,29 +594,14 @@ static void __init allocate_static_memory(struct domain
*d,
gbank = 0;
gsize = ramsize[gbank];
kinfo->mem.bank[gbank].start = rambase[gbank];
-
-cell = (const __be32 *)prop->value;
-nr_banks = (prop->length) / (reg_cells * sizeof (u32));
+nr_banks = length / (reg_cells * sizeof (u32));
for ( ; bank < n
[PATCH v5 02/11] xen: introduce CDF_directmap
From: Stefano Stabellini
This commit introduces a new arm-specific flag CDF_directmap to specify
that a domain should have its memory direct-map(guest physical address
== physical address) at domain creation.
Also, add a directmap flag under struct arch_domain and use it to
reimplement is_domain_direct_mapped.
For now, direct-map is only available when statically allocated memory is
used for the domain, that is, "xen,static-mem" must be also defined in the
domain configuration.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Acked-by: Jan Beulich
Tested-by: Stefano Stabellini
---
CC: andrew.coop...@citrix.com
CC: jbeul...@suse.com
CC: George Dunlap
CC: Ian Jackson
CC: Wei Liu
CC: "Roger Pau Monné"
---
v2 changes
- remove the introduce of internal flag
- remove flag direct_map since we already store this flag in d->options
- Refine is_domain_direct_mapped to check whether the flag
XEN_DOMCTL_CDF_directmap is set
- reword "1:1 direct-map" to just "direct-map"
---
v3 changes
- move flag back to xen/include/xen/domain.h, to let it be only available for
domain created by XEN.
- name it with extra "INTERNAL" and add comments to warn developers not
to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag.
- reject this flag in x86'es arch_sanitise_domain_config()
---
v4 changes
- introduce new internal flag CDF_directmap
- add a directmap flag under struct arch_domain and use it to
reimplement is_domain_direct_mapped.
- expand arch_domain_create to include internal-only parameter "const unsigned
int flags"
---
v5 changes
- remove const constraint
- strict "static allocation" check
---
docs/misc/arm/device-tree/booting.txt | 6 ++
xen/arch/arm/domain.c | 5 -
xen/arch/arm/domain_build.c | 14 --
xen/arch/arm/include/asm/domain.h | 5 +++--
xen/arch/x86/domain.c | 3 ++-
xen/common/domain.c | 2 +-
xen/include/xen/domain.h | 5 -
7 files changed, 32 insertions(+), 8 deletions(-)
diff --git a/docs/misc/arm/device-tree/booting.txt
b/docs/misc/arm/device-tree/booting.txt
index 71895663a4..a94125394e 100644
--- a/docs/misc/arm/device-tree/booting.txt
+++ b/docs/misc/arm/device-tree/booting.txt
@@ -182,6 +182,12 @@ with the following properties:
Both #address-cells and #size-cells need to be specified because
both sub-nodes (described shortly) have reg properties.
+- direct-map
+
+Only available when statically allocated memory is used for the domain.
+An empty property to request the memory of the domain to be
+direct-map (guest physical address == physical address).
+
Under the "xen,domain" compatible node, one or more sub-nodes are present
for the DomU kernel and ramdisk.
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 92a6c509e5..8110c1df86 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -692,7 +692,8 @@ int arch_sanitise_domain_config(struct
xen_domctl_createdomain *config)
}
int arch_domain_create(struct domain *d,
- struct xen_domctl_createdomain *config)
+ struct xen_domctl_createdomain *config,
+ unsigned int flags)
{
int rc, count = 0;
@@ -708,6 +709,8 @@ int arch_domain_create(struct domain *d,
ioreq_domain_init(d);
#endif
+d->arch.directmap = flags & CDF_directmap;
+
/* p2m_init relies on some value initialized by the IOMMU subsystem */
if ( (rc = iommu_domain_init(d, config->iommu_opts)) != 0 )
goto fail;
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 0fab8604de..6467e8ee32 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3029,10 +3029,20 @@ void __init create_domUs(void)
.max_maptrack_frames = -1,
.grant_opts = XEN_DOMCTL_GRANT_version(opt_gnttab_max_version),
};
+unsigned int flags = 0U;
if ( !dt_device_is_compatible(node, "xen,domain") )
continue;
+if ( dt_property_read_bool(node, "direct-map") )
+{
+if ( !IS_ENABLED(CONFIG_STATIC_MEMORY) || !dt_find_property(node,
"xen,static-mem", NULL) )
+panic("direct-map is not valid for domain %s without static
allocation.\n",
+ dt_node_name(node));
+
+flags |= CDF_directmap;
+}
+
if ( !dt_property_read_u32(node, "cpus", &d_cfg.max_vcpus) )
panic("Missing property 'cpus' for domain %s\n",
dt_node_name(node));
@@ -3058,7 +3068,7 @@ void __init create_domUs(void)
* very important to use the pre-increment operator to call
* domain_create() with a domid > 0. (domid == 0 is reserved for Dom0)
[PATCH v5 06/11] xen/arm: add ASSERT_UNREACHABLE in allocate_static_memory
Helper allocate_static_memory is not meant to be reachable when built with
!CONFIG_STATIC_MEMORY, so this commit adds ASSERT_UNREACHABLE in it to catch
potential misuse.
Signed-off-by: Penny Zheng
Acked-by: Julien Grall
Tested-by: Stefano Stabellini
---
v3 changes:
- new commit
---
v4 changes:
- nothing changed
---
v5 changes:
- nothing changed
---
xen/arch/arm/domain_build.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index ec29bd302c..52f256de9c 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -755,6 +755,7 @@ static void __init allocate_static_memory(struct domain *d,
struct kernel_info *kinfo,
const struct dt_device_node *node)
{
+ASSERT_UNREACHABLE();
}
static void __init assign_static_memory_11(struct domain *d,
--
2.25.1
[PATCH v5 08/11] xen/arm: gate make_gicv3_domU_node with CONFIG_GICV3
This commit gates function make_gicv3_domU_node with CONFIG_GICV3.
Signed-off-by: Penny Zheng
Acked-by: Stefano Stabellini
Tested-by: Stefano Stabellini
---
v4 changes:
- remove ASSERT_UNREACHABLE() to avoid breaking compilation on prod build with
CONFIG_GICV3=n
---
v5 changes:
- nothing changed
---
xen/arch/arm/domain_build.c | 4
1 file changed, 4 insertions(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 87391adde6..a01dc60b55 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2322,6 +2322,7 @@ static int __init make_gicv2_domU_node(struct kernel_info
*kinfo)
return res;
}
+#ifdef CONFIG_GICV3
static int __init make_gicv3_domU_node(struct kernel_info *kinfo)
{
void *fdt = kinfo->fdt;
@@ -2371,13 +2372,16 @@ static int __init make_gicv3_domU_node(struct
kernel_info *kinfo)
return res;
}
+#endif
static int __init make_gic_domU_node(struct kernel_info *kinfo)
{
switch ( kinfo->d->arch.vgic.version )
{
+#ifdef CONFIG_GICV3
case GIC_V3:
return make_gicv3_domU_node(kinfo);
+#endif
case GIC_V2:
return make_gicv2_domU_node(kinfo);
default:
--
2.25.1
[PATCH v5 07/11] xen/arm: if direct-map domain use native addresses for GICv2
From: Stefano Stabellini
Today we use native addresses to map the GICv2 for Dom0 and fixed
addresses for DomUs.
This patch changes the behavior so that native addresses are used for
all domains that are direct-mapped.
NEW VGIC has different naming schemes, like referring distributor base
address as vgic_dist_base, other than the dbase. So this patch also introduces
vgic_dist_base/vgic_cpu_base accessor to access correct distributor base
address/cpu interface base address on varied scenarios,
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Tested-by: Stefano Stabellini
---
v2 changes
- combine all changes in patch 4-6 here
---
v3 changes
- refine comment message
- add a comment explaining how the 38 was found of "char buf[38]"
- simply map the CPU interface at the GPA vgic_v2_hw.cbase
- remove a spurious change
---
v4 changes:
- refine comment to let it be a summary of the if/else if/else.
---
v5 changes:
- nothing changed
---
xen/arch/arm/domain_build.c | 11 +++---
xen/arch/arm/include/asm/new_vgic.h | 10 +
xen/arch/arm/include/asm/vgic.h | 11 ++
xen/arch/arm/vgic-v2.c | 34 +++--
xen/arch/arm/vgic/vgic-v2.c | 34 +++--
5 files changed, 83 insertions(+), 17 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 52f256de9c..87391adde6 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2273,8 +2273,13 @@ static int __init make_gicv2_domU_node(struct
kernel_info *kinfo)
int res = 0;
__be32 reg[(GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) * 2];
__be32 *cells;
+const struct domain *d = kinfo->d;
+/* Placeholder for interrupt-controller@ + a 64-bit number + \0 */
+char buf[38];
-res = fdt_begin_node(fdt,
"interrupt-controller@"__stringify(GUEST_GICD_BASE));
+snprintf(buf, sizeof(buf), "interrupt-controller@%"PRIx64,
+ vgic_dist_base(&d->arch.vgic));
+res = fdt_begin_node(fdt, buf);
if ( res )
return res;
@@ -2296,9 +2301,9 @@ static int __init make_gicv2_domU_node(struct kernel_info
*kinfo)
cells = ®[0];
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICD_BASE, GUEST_GICD_SIZE);
+ vgic_dist_base(&d->arch.vgic), GUEST_GICD_SIZE);
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICC_BASE, GUEST_GICC_SIZE);
+ vgic_cpu_base(&d->arch.vgic), GUEST_GICC_SIZE);
res = fdt_property(fdt, "reg", reg, sizeof(reg));
if (res)
diff --git a/xen/arch/arm/include/asm/new_vgic.h
b/xen/arch/arm/include/asm/new_vgic.h
index 97d622bff6..ab57fcd91d 100644
--- a/xen/arch/arm/include/asm/new_vgic.h
+++ b/xen/arch/arm/include/asm/new_vgic.h
@@ -186,6 +186,16 @@ struct vgic_cpu {
uint32_t num_id_bits;
};
+static inline paddr_t vgic_cpu_base(const struct vgic_dist *vgic)
+{
+return vgic->vgic_cpu_base;
+}
+
+static inline paddr_t vgic_dist_base(const struct vgic_dist *vgic)
+{
+return vgic->vgic_dist_base;
+}
+
#endif /* __ASM_ARM_NEW_VGIC_H */
/*
diff --git a/xen/arch/arm/include/asm/vgic.h b/xen/arch/arm/include/asm/vgic.h
index ade427a808..d2a9fc7d83 100644
--- a/xen/arch/arm/include/asm/vgic.h
+++ b/xen/arch/arm/include/asm/vgic.h
@@ -152,6 +152,7 @@ struct vgic_dist {
struct pending_irq *pending_irqs;
/* Base address for guest GIC */
paddr_t dbase; /* Distributor base address */
+paddr_t cbase; /* CPU interface base address */
#ifdef CONFIG_GICV3
/* GIC V3 addressing */
/* List of contiguous occupied by the redistributors */
@@ -271,6 +272,16 @@ static inline int REG_RANK_NR(int b, uint32_t n)
enum gic_sgi_mode;
+static inline paddr_t vgic_cpu_base(const struct vgic_dist *vgic)
+{
+return vgic->cbase;
+}
+
+static inline paddr_t vgic_dist_base(const struct vgic_dist *vgic)
+{
+return vgic->dbase;
+}
+
/*
* Offset of GICD_ with its rank, for GICD_ size with
* -bits-per-interrupt.
diff --git a/xen/arch/arm/vgic-v2.c b/xen/arch/arm/vgic-v2.c
index 589c033eda..b1bd7a46ad 100644
--- a/xen/arch/arm/vgic-v2.c
+++ b/xen/arch/arm/vgic-v2.c
@@ -654,12 +654,16 @@ static int vgic_v2_vcpu_init(struct vcpu *v)
static int vgic_v2_domain_init(struct domain *d)
{
int ret;
-paddr_t cbase, csize;
+paddr_t csize;
paddr_t vbase;
/*
- * The hardware domain gets the hardware address.
- * Guests get the virtual platform layout.
+ * The hardware domain and direct-mapped domain both get the hardware
+ * address.
+ * We have to handle them separately because the hwdom is re-using the
+ * same Device-Tree as the host (see more details below).
+ *
+ * Other domains get the virtual platform layout
[PATCH v5 10/11] xen/arm: if direct-map domain use native UART address and IRQ number for vPL011
From: Stefano Stabellini
We always use a fix address to map the vPL011 to domains. The address
could be a problem for direct-map domains.
So, for domains that are directly mapped, reuse the address of the
physical UART on the platform to avoid potential clashes.
Do the same for the virtual IRQ number: instead of always using
GUEST_VPL011_SPI, try to reuse the physical SPI number if possible.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Tested-by: Stefano Stabellini
---
v2 changes:
- explain why vpl011 initialization before creating its device tree node
- error out if the domain is direct-mapped and the IRQ is not found
- harden the code and add a check/comment when the hardware UART region
is smaller than GUEST_VPL011_SIZE.
---
v3 changes:
- explain how the '27' was found for 'buf'
- fix checking before dereferencing
- refine comment message
---
v4 changes:
- refine comment message
---
v5 changes:
- nothing changed
---
xen/arch/arm/domain_build.c | 44 +++
xen/arch/arm/include/asm/vpl011.h | 2 ++
xen/arch/arm/vpl011.c | 60 +++
3 files changed, 92 insertions(+), 14 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index cff2cb93cc..8be01678de 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -30,6 +30,7 @@
#include
#include
+#include
static unsigned int __initdata opt_dom0_max_vcpus;
integer_param("dom0_max_vcpus", opt_dom0_max_vcpus);
@@ -2415,8 +2416,12 @@ static int __init make_vpl011_uart_node(struct
kernel_info *kinfo)
gic_interrupt_t intr;
__be32 reg[GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS];
__be32 *cells;
+struct domain *d = kinfo->d;
+/* Placeholder for sbsa-uart@ + a 64-bit number + \0 */
+char buf[27];
-res = fdt_begin_node(fdt, "sbsa-uart@"__stringify(GUEST_PL011_BASE));
+snprintf(buf, sizeof(buf), "sbsa-uart@%"PRIx64, d->arch.vpl011.base_addr);
+res = fdt_begin_node(fdt, buf);
if ( res )
return res;
@@ -2426,14 +2431,14 @@ static int __init make_vpl011_uart_node(struct
kernel_info *kinfo)
cells = ®[0];
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS,
- GUEST_ROOT_SIZE_CELLS, GUEST_PL011_BASE,
+ GUEST_ROOT_SIZE_CELLS, d->arch.vpl011.base_addr,
GUEST_PL011_SIZE);
res = fdt_property(fdt, "reg", reg, sizeof(reg));
if ( res )
return res;
-set_interrupt(intr, GUEST_VPL011_SPI, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
+set_interrupt(intr, d->arch.vpl011.virq, 0xf, DT_IRQ_TYPE_LEVEL_HIGH);
res = fdt_property(fdt, "interrupts", intr, sizeof (intr));
if ( res )
@@ -3145,6 +3150,14 @@ static int __init construct_domU(struct domain *d,
else
assign_static_memory_11(d, &kinfo, node);
+/*
+ * Base address and irq number are needed when creating vpl011 device
+ * tree node in prepare_dtb_domU, so initialization on related variables
+ * shall be done first.
+ */
+if ( kinfo.vpl011 )
+rc = domain_vpl011_init(d, NULL);
+
rc = prepare_dtb_domU(d, &kinfo);
if ( rc < 0 )
return rc;
@@ -3153,9 +3166,6 @@ static int __init construct_domU(struct domain *d,
if ( rc < 0 )
return rc;
-if ( kinfo.vpl011 )
-rc = domain_vpl011_init(d, NULL);
-
return rc;
}
@@ -3200,15 +3210,35 @@ void __init create_domUs(void)
if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
{
+unsigned int vpl011_virq = GUEST_VPL011_SPI;
+
d_cfg.arch.nr_spis = gic_number_lines() - 32;
+/*
+ * The VPL011 virq is GUEST_VPL011_SPI, unless direct-map is
+ * set, in which case it'll match the hardware.
+ *
+ * Since the domain is not yet created, we can't use
+ * d->arch.vpl011.irq. So the logic to find the vIRQ has to
+ * be hardcoded.
+ * The logic here shall be consistent with the one in
+ * domain_vpl011_init().
+ */
+if ( flags & CDF_directmap )
+{
+vpl011_virq = serial_irq(SERHND_DTUART);
+if ( vpl011_virq < 0 )
+panic("Error getting IRQ number for this serial port %d\n",
+ SERHND_DTUART);
+}
+
/*
* vpl011 uses one emulated SPI. If vpl011 is requested, make
* sure that we allocate enough SPIs for it.
*/
if ( dt_property_read_bool(node, "vpl011") )
d_cfg.arch.nr_spis = MAX(d_cfg.arch.nr_spis,
- GUEST_VPL011_SPI - 32 + 1);
+
[PATCH v5 03/11] xen/arm: avoid setting XEN_DOMCTL_CDF_iommu when IOMMU off
From: Stefano Stabellini
When IOMMU is absent or shall not be used (trusted domain, performance,
hardware limitation, ..., etc), in which cases this commit avoids setting
XEN_DOMCTL_CDF_iommu to make those user cases possible and prevent failure
later during device assignment.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Tested-by: Stefano Stabellini
---
v3 changes:
- new commit, split from the original "[PATCH v2 2/6] xen/arm: introduce
direct-map for domUs"
---
v4 changes:
- explain briefly in the commit message why we want to do device assignment
without IOMMU.
---
v5 changes:
- nothing changed
---
xen/arch/arm/domain_build.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 6467e8ee32..c1e8c99f64 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3047,7 +3047,8 @@ void __init create_domUs(void)
panic("Missing property 'cpus' for domain %s\n",
dt_node_name(node));
-if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
+if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") &&
+ iommu_enabled )
d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
--
2.25.1
[PATCH v5 05/11] xen/arm: introduce direct-map for domUs
Cases where domU needs direct-map memory map:
* IOMMU not present in the system.
* IOMMU disabled if it doesn't cover a specific device and all the guests
are trusted. Thinking a mixed scenario, where a few devices with IOMMU and
a few without, then guest DMA security still could not be totally guaranteed.
So users may want to disable the IOMMU, to at least gain some performance
improvement from IOMMU disabled.
* IOMMU disabled as a workaround when it doesn't have enough bandwidth.
To be specific, in a few extreme situation, when multiple devices do DMA
concurrently, these requests may exceed IOMMU's transmission capacity.
* IOMMU disabled when it adds too much latency on DMA. For example,
TLB may be missing in some IOMMU hardware, which may bring latency in DMA
progress, so users may want to disable it in some realtime scenario.
* Guest OS relies on the host memory layout
This commit introduces a new helper assign_static_memory_11 to allocate
static memory as guest RAM for direct-map domain.
Signed-off-by: Penny Zheng
Reviewed-by: Stefano Stabellini
Tested-by: Stefano Stabellini
---
v2 changes:
- split the common codes into two helpers: parse_static_mem_prop and
acquire_static_memory_bank to deduce complexity.
- introduce a new helper allocate_static_memory_11 for allocating static
memory for direct-map guests
- remove redundant use "bool direct_map", to be replaced by
d_cfg.flags & XEN_DOMCTL_CDF_directmap
- remove panic action since it is fine to assign a non-DMA capable device when
IOMMU and direct-map both off
---
v3 changes:
- doc refinement
- drop the pointless gbank
- add check of the size of nr_banks shall not exceed NR_MEM_BANKS
- add ASSERT_UNREACHABLE to catch any misuse
- add another check of validating flag XEN_DOMCTL_CDF_INTERNAL_directmap only
when CONFIG_STATIC_MEMORY is set.
---
v4 changes:
- comment refinement
- rename function allocate_static_memory_11() to assign_static_memory_11()
to make clear there is actually no allocation done. Instead we are only
mapping pre-defined host regions to pre-defined guest regions.
- remove tot_size to directly substract psize from kinfo->unassigned_mem
- check kinfo->unassigned_mem doesn't underflow or overflow
- remove nested if/else
- refine "panic" info
---
v5 changes:
- fix coding style
---
xen/arch/arm/domain_build.c | 97 +++--
1 file changed, 94 insertions(+), 3 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index e61d2d53ba..ec29bd302c 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -494,8 +494,17 @@ static bool __init append_static_memory_to_bank(struct
domain *d,
{
int res;
unsigned int nr_pages = PFN_DOWN(size);
-/* Infer next GFN. */
-gfn_t sgfn = gaddr_to_gfn(bank->start + bank->size);
+gfn_t sgfn;
+
+/*
+ * For direct-mapped domain, the GFN match the MFN.
+ * Otherwise, this is inferred on what has already been allocated
+ * in the bank.
+ */
+if ( !is_domain_direct_mapped(d) )
+sgfn = gaddr_to_gfn(bank->start + bank->size);
+else
+sgfn = gaddr_to_gfn(mfn_to_maddr(smfn));
res = guest_physmap_add_pages(d, sgfn, smfn, nr_pages);
if ( res )
@@ -668,12 +677,92 @@ static void __init allocate_static_memory(struct domain
*d,
fail:
panic("Failed to allocate requested static memory for domain %pd.", d);
}
+
+/*
+ * Allocate static memory as RAM for one specific domain d.
+ * The static memory will be directly mapped in the guest(Guest Physical
+ * Address == Physical Address).
+ */
+static void __init assign_static_memory_11(struct domain *d,
+ struct kernel_info *kinfo,
+ const struct dt_device_node *node)
+{
+u32 addr_cells, size_cells, reg_cells;
+unsigned int nr_banks, bank = 0;
+const __be32 *cell;
+paddr_t pbase, psize;
+mfn_t smfn;
+int length;
+
+if ( parse_static_mem_prop(node, &addr_cells, &size_cells, &length, &cell)
)
+{
+printk(XENLOG_ERR
+ "%pd: failed to parse \"xen,static-mem\" property.\n", d);
+goto fail;
+}
+reg_cells = addr_cells + size_cells;
+nr_banks = length / (reg_cells * sizeof(u32));
+
+if ( nr_banks > NR_MEM_BANKS )
+{
+printk(XENLOG_ERR
+ "%pd: exceed max number of supported guest memory banks.\n", d);
+goto fail;
+}
+
+for ( ; bank < nr_banks; bank++ )
+{
+smfn = acquire_static_memory_bank(d, &cell, addr_cells, size_cells,
+ &pbase, &psize);
+if ( mfn_eq(smfn, INVALID_MFN) )
+goto fail;
+
+printk(XENLOG_INFO "%pd: STATIC BANK[%u] %#"PRIpaddr"-%#"PRIpaddr"\n",
+
[PATCH v5 11/11] xen/docs: Document how to do passthrough without IOMMU
From: Stefano Stabellini
This commit creates a new doc to document how to do passthrough without IOMMU.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Tested-by: Stefano Stabellini
---
v2 changes:
- nothing changed
---
v3 changes:
- nothing changed
---
v4 changes:
- nothing changed
---
v5 changes:
- nothing changed
---
docs/misc/arm/passthrough-noiommu.txt | 52 +++
1 file changed, 52 insertions(+)
create mode 100644 docs/misc/arm/passthrough-noiommu.txt
diff --git a/docs/misc/arm/passthrough-noiommu.txt
b/docs/misc/arm/passthrough-noiommu.txt
new file mode 100644
index 00..3e2ef21ad7
--- /dev/null
+++ b/docs/misc/arm/passthrough-noiommu.txt
@@ -0,0 +1,52 @@
+Request Device Assignment without IOMMU support
+===
+
+*WARNING:
+Users should be aware that it is not always secure to assign a device without
+IOMMU protection.
+When the device is not protected by the IOMMU, the administrator should make
+sure that:
+ 1. The device is assigned to a trusted guest.
+ 2. Users have additional security mechanism on the platform.
+
+This document assumes that the IOMMU is absent from the system or it is
+disabled (status = "disabled" in device tree).
+
+Add xen,force-assign-without-iommu; to the device tree snippet:
+
+ethernet: ethernet@ff0e {
+ compatible = "cdns,zynqmp-gem";
+ xen,path = "/amba/ethernet@ff0e";
+ xen,reg = <0x0 0xff0e 0x1000 0x0 0xff0e>;
+ xen,force-assign-without-iommu;
+};
+
+Request 1:1 memory mapping for the domain on static allocation
+==
+
+Add a direct-map property under the appropriate /chosen/domU node which
+is also statically allocated with physical memory ranges through
+xen,static-mem property as its guest RAM.
+
+Below is an example on how to specify the 1:1 memory mapping for the domain
+on static allocation in the device-tree:
+
+/ {
+ chosen {
+ domU1 {
+ compatible = "xen,domain";
+ #address-cells = <0x2>;
+ #size-cells = <0x2>;
+ cpus = <2>;
+ memory = <0x0 0x8>;
+ #xen,static-mem-address-cells = <0x1>;
+ #xen,static-mem-size-cells = <0x1>;
+ xen,static-mem = <0x3000 0x2000>;
+ direct-map;
+ ...
+ };
+ };
+};
+
+Besides reserving a 512MB region starting at the host physical address
+0x3000 to DomU1, it also requests 1:1 memory mapping.
--
2.25.1
[PATCH v5 09/11] xen/arm: if direct-map domain use native addresses for GICv3
From: Stefano Stabellini
Today we use native addresses to map the GICv3 for Dom0 and fixed
addresses for DomUs.
This patch changes the behavior so that native addresses are used for
all domain which is using the host memory layout
Considering that DOM0 may not always be directly mapped in the future,
this patch introduces a new helper "domain_use_host_layout()" that
wraps both two check "is_domain_direct_mapped(d) || is_hardware_domain(d)"
for more flexible usage.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Tested-by: Stefano Stabellini
---
v2 changes:
- remove redistributor accessor
- introduce new helper "is_domain_use_host_layout()"
- comment fix
---
v3 changes:
- the comment on top of 'buf' to explain how 38 was found
- fix res getting overwritten
- drop 'cells += (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS)'
- free 'reg' right way
- fix comment
- rename 'is_domain_use_host_layout()' to 'domain_use_host_layout()'
---
v4 changes:
- refine comment
---
v5 changes:
- nothing changed
---
xen/arch/arm/domain_build.c | 34 +++
xen/arch/arm/include/asm/domain.h | 14 +
xen/arch/arm/vgic-v3.c| 26 ---
3 files changed, 54 insertions(+), 20 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index a01dc60b55..cff2cb93cc 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2327,10 +2327,16 @@ static int __init make_gicv3_domU_node(struct
kernel_info *kinfo)
{
void *fdt = kinfo->fdt;
int res = 0;
-__be32 reg[(GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) * 2];
-__be32 *cells;
+__be32 *reg, *cells;
+const struct domain *d = kinfo->d;
+/* Placeholder for interrupt-controller@ + a 64-bit number + \0 */
+char buf[38];
+unsigned int i, len = 0;
+
+snprintf(buf, sizeof(buf), "interrupt-controller@%"PRIx64,
+ vgic_dist_base(&d->arch.vgic));
-res = fdt_begin_node(fdt,
"interrupt-controller@"__stringify(GUEST_GICV3_GICD_BASE));
+res = fdt_begin_node(fdt, buf);
if ( res )
return res;
@@ -2350,13 +2356,25 @@ static int __init make_gicv3_domU_node(struct
kernel_info *kinfo)
if ( res )
return res;
-cells = ®[0];
-dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICV3_GICD_BASE, GUEST_GICV3_GICD_SIZE);
+/* reg specifies all re-distributors and Distributor. */
+len = (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS) *
+ (d->arch.vgic.nr_regions + 1) * sizeof(__be32);
+reg = xmalloc_bytes(len);
+if ( reg == NULL )
+return -ENOMEM;
+cells = reg;
+
dt_child_set_range(&cells, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
- GUEST_GICV3_GICR0_BASE, GUEST_GICV3_GICR0_SIZE);
+ vgic_dist_base(&d->arch.vgic), GUEST_GICV3_GICD_SIZE);
-res = fdt_property(fdt, "reg", reg, sizeof(reg));
+for ( i = 0; i < d->arch.vgic.nr_regions; i++ )
+dt_child_set_range(&cells,
+ GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS,
+ d->arch.vgic.rdist_regions[i].base,
+ d->arch.vgic.rdist_regions[i].size);
+
+res = fdt_property(fdt, "reg", reg, len);
+xfree(reg);
if (res)
return res;
diff --git a/xen/arch/arm/include/asm/domain.h
b/xen/arch/arm/include/asm/domain.h
index cb37ce89ec..848fec8a0f 100644
--- a/xen/arch/arm/include/asm/domain.h
+++ b/xen/arch/arm/include/asm/domain.h
@@ -31,6 +31,20 @@ enum domain_type {
#define is_domain_direct_mapped(d) (d->arch.directmap)
+/*
+ * Is the domain using the host memory layout?
+ *
+ * Direct-mapped domain will always have the RAM mapped with GFN == MFN.
+ * To avoid any trouble finding space, it is easier to force using the
+ * host memory layout.
+ *
+ * The hardware domain will use the host layout regardless of
+ * direct-mapped because some OS may rely on a specific address ranges
+ * for the devices.
+ */
+#define domain_use_host_layout(d) (is_domain_direct_mapped(d) || \
+ is_hardware_domain(d))
+
struct vtimer {
struct vcpu *v;
int irq;
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index 65bb7991a6..144089a7b6 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -1640,14 +1640,15 @@ static inline unsigned int
vgic_v3_max_rdist_count(struct domain *d)
* Normally there is only one GICv3 redistributor region.
* The GICv3 DT binding provisions for multiple regions, since there are
* platforms out there which need those (multi-socket systems).
- * For Dom0 we have to live with the MMIO layout the hardware p
[PATCH v6 02/11] xen: introduce CDF_directmap
From: Stefano Stabellini
This commit introduces a new arm-specific flag CDF_directmap to specify
that a domain should have its memory direct-map(guest physical address
== host physical address) at domain creation.
Also, add a directmap flag under struct arch_domain and use it to
reimplement is_domain_direct_mapped.
For now, direct-map is only available when statically allocated memory is
used for the domain, that is, "xen,static-mem" must be also defined in the
domain configuration.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Acked-by: Jan Beulich
Tested-by: Stefano Stabellini
---
CC: andrew.coop...@citrix.com
CC: jbeul...@suse.com
CC: George Dunlap
CC: Ian Jackson
CC: Wei Liu
CC: "Roger Pau Monné"
---
v2 changes
- remove the introduce of internal flag
- remove flag direct_map since we already store this flag in d->options
- Refine is_domain_direct_mapped to check whether the flag
XEN_DOMCTL_CDF_directmap is set
- reword "1:1 direct-map" to just "direct-map"
---
v3 changes
- move flag back to xen/include/xen/domain.h, to let it be only available for
domain created by XEN.
- name it with extra "INTERNAL" and add comments to warn developers not
to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag.
- reject this flag in x86'es arch_sanitise_domain_config()
---
v4 changes
- introduce new internal flag CDF_directmap
- add a directmap flag under struct arch_domain and use it to
reimplement is_domain_direct_mapped.
- expand arch_domain_create to include internal-only parameter "const unsigned
int flags"
---
v5 changes
- remove const constraint
---
v6 changes
- comment and coding style fix
- protect CDF_directmap with #ifdef CONFIG_ARM
---
docs/misc/arm/device-tree/booting.txt | 6 ++
xen/arch/arm/domain.c | 5 -
xen/arch/arm/domain_build.c | 14 --
xen/arch/arm/include/asm/domain.h | 5 +++--
xen/arch/x86/domain.c | 3 ++-
xen/common/domain.c | 2 +-
xen/include/xen/domain.h | 7 ++-
7 files changed, 34 insertions(+), 8 deletions(-)
diff --git a/docs/misc/arm/device-tree/booting.txt
b/docs/misc/arm/device-tree/booting.txt
index 71895663a4..a94125394e 100644
--- a/docs/misc/arm/device-tree/booting.txt
+++ b/docs/misc/arm/device-tree/booting.txt
@@ -182,6 +182,12 @@ with the following properties:
Both #address-cells and #size-cells need to be specified because
both sub-nodes (described shortly) have reg properties.
+- direct-map
+
+Only available when statically allocated memory is used for the domain.
+An empty property to request the memory of the domain to be
+direct-map (guest physical address == physical address).
+
Under the "xen,domain" compatible node, one or more sub-nodes are present
for the DomU kernel and ramdisk.
diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 92a6c509e5..8110c1df86 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -692,7 +692,8 @@ int arch_sanitise_domain_config(struct
xen_domctl_createdomain *config)
}
int arch_domain_create(struct domain *d,
- struct xen_domctl_createdomain *config)
+ struct xen_domctl_createdomain *config,
+ unsigned int flags)
{
int rc, count = 0;
@@ -708,6 +709,8 @@ int arch_domain_create(struct domain *d,
ioreq_domain_init(d);
#endif
+d->arch.directmap = flags & CDF_directmap;
+
/* p2m_init relies on some value initialized by the IOMMU subsystem */
if ( (rc = iommu_domain_init(d, config->iommu_opts)) != 0 )
goto fail;
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 0fab8604de..6467e8ee32 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3029,10 +3029,20 @@ void __init create_domUs(void)
.max_maptrack_frames = -1,
.grant_opts = XEN_DOMCTL_GRANT_version(opt_gnttab_max_version),
};
+unsigned int flags = 0U;
if ( !dt_device_is_compatible(node, "xen,domain") )
continue;
+if ( dt_property_read_bool(node, "direct-map") )
+{
+if ( !IS_ENABLED(CONFIG_STATIC_MEMORY) || !dt_find_property(node,
"xen,static-mem", NULL) )
+panic("direct-map is not valid for domain %s without static
allocation.\n",
+ dt_node_name(node));
+
+flags |= CDF_directmap;
+}
+
if ( !dt_property_read_u32(node, "cpus", &d_cfg.max_vcpus) )
panic("Missing property 'cpus' for domain %s\n",
dt_node_name(node));
@@ -3058,7 +3068,7 @@ void __init create_domUs(void)
* very important to use the pre-increment operator to call
* domain_create() with a domi
[PATCH v6 01/11] xen: introduce internal CDF_xxx flags for domain creation
From: Stefano Stabellini
We are passing an internal-only boolean flag at domain creation to
specify whether we want the domain to be privileged (i.e. dom0) or
not. Another flag will be introduced later in this series.
This commit extends original "boolean" to an "unsigned int" covering both
the existing "is_priv" and our new "directmap", which will be introduced later.
To make visible the relationship, we name the respective constants CDF_xxx
(with no XEN_DOMCTL_ prefix) to represent the difference with the public
constants XEN_DOMCTL_CDF_xxx.
Allocate bit 0 as CDF_privileged: whether a domain is privileged or not.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Reviewed-by: Jan Beulich
Reviewed-by: Julien Grall
Tested-by: Stefano Stabellini
---
v4 changes:
- new commit
---
v5 changes
- remove const constraint
---
v6 changes
- no changes
---
xen/arch/arm/domain_build.c | 4 ++--
xen/arch/x86/setup.c| 2 +-
xen/common/domain.c | 10 +-
xen/common/sched/core.c | 2 +-
xen/include/xen/domain.h| 4
xen/include/xen/sched.h | 2 +-
6 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 6931c022a2..0fab8604de 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3058,7 +3058,7 @@ void __init create_domUs(void)
* very important to use the pre-increment operator to call
* domain_create() with a domid > 0. (domid == 0 is reserved for Dom0)
*/
-d = domain_create(++max_init_domid, &d_cfg, false);
+d = domain_create(++max_init_domid, &d_cfg, 0);
if ( IS_ERR(d) )
panic("Error creating domain %s\n", dt_node_name(node));
@@ -3160,7 +3160,7 @@ void __init create_dom0(void)
if ( iommu_enabled )
dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
-dom0 = domain_create(0, &dom0_cfg, true);
+dom0 = domain_create(0, &dom0_cfg, CDF_privileged);
if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
panic("Error creating domain 0\n");
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 115f8f6517..624b53ded4 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -789,7 +789,7 @@ static struct domain *__init create_dom0(const module_t
*image,
/* Create initial domain. Not d0 for pvshim. */
domid = get_initial_domain_id();
-d = domain_create(domid, &dom0_cfg, !pv_shim);
+d = domain_create(domid, &dom0_cfg, pv_shim ? 0 : CDF_privileged);
if ( IS_ERR(d) )
panic("Error creating d%u: %ld\n", domid, PTR_ERR(d));
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 2048ebad86..a79103e04a 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -552,7 +552,7 @@ static int sanitise_domain_config(struct
xen_domctl_createdomain *config)
struct domain *domain_create(domid_t domid,
struct xen_domctl_createdomain *config,
- bool is_priv)
+ unsigned int flags)
{
struct domain *d, **pd, *old_hwdom = NULL;
enum { INIT_watchdog = 1u<<1,
@@ -578,7 +578,7 @@ struct domain *domain_create(domid_t domid,
}
/* Sort out our idea of is_control_domain(). */
-d->is_privileged = is_priv;
+d->is_privileged = flags & CDF_privileged;
/* Sort out our idea of is_hardware_domain(). */
if ( domid == 0 || domid == hardware_domid )
@@ -772,7 +772,7 @@ void __init setup_system_domains(void)
* Hidden PCI devices will also be associated with this domain
* (but be [partly] controlled by Dom0 nevertheless).
*/
-dom_xen = domain_create(DOMID_XEN, NULL, false);
+dom_xen = domain_create(DOMID_XEN, NULL, 0);
if ( IS_ERR(dom_xen) )
panic("Failed to create d[XEN]: %ld\n", PTR_ERR(dom_xen));
@@ -782,7 +782,7 @@ void __init setup_system_domains(void)
* array. Mappings occur at the priv of the caller.
* Quarantined PCI devices will be associated with this domain.
*/
-dom_io = domain_create(DOMID_IO, NULL, false);
+dom_io = domain_create(DOMID_IO, NULL, 0);
if ( IS_ERR(dom_io) )
panic("Failed to create d[IO]: %ld\n", PTR_ERR(dom_io));
@@ -791,7 +791,7 @@ void __init setup_system_domains(void)
* Initialise our COW domain.
* This domain owns sharable pages.
*/
-dom_cow = domain_create(DOMID_COW, NULL, false);
+dom_cow = domain_create(DOMID_COW, NULL, 0);
if ( IS_ERR(dom_cow) )
panic("Failed to create d[COW]: %ld\n", PTR_ERR(dom_cow));
#endif
diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c
index 8f4b1ca10d..f5c819349b 100644
--- a/xen/common/sched/core.c
+++ b/xen/common/sched/core.c
@@ -3021,7 +3021,7 @@ void __init scheduler_init(void)
[PATCH v6 00/11] direct-map memory map
"direct-map" property shall be added under the appropriate domain node, when users requesting direct-map memory mapping for the domain. Right now, direct-map is only supported when domain on Static Allocation, that is, "xen,static-mem" is also necessary in the domain configuration. Looking into related [design link]( https://lists.xenproject.org/archives/html/xen-devel/2021-05/msg00882.html) for more details. The whole design is about Static Allocation and direct-map, and this Patch Serie only covers parts of it, which are direct-map memory map. Other features will be delievered through different patch series. See https://lists.xenproject.org/archives/html/xen-devel/2021-09/msg00855.html for Domain on Static Allocation. This patch serie is based on https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00822.html\ --- v6 changes: - comment, commit message and coding style fix - protect CDF_directmap with #ifdef CONFIG_ARM --- v5 changes: - remove const constraint and strict "static allocation" check - fix coding style --- v4 changes: - introduce internal const CDF_xxx flags for domain creation - introduce internal flag CDF_privileged - introduce new internal flag CDF_directmap - add a directmap flag under struct arch_domain and use it to reimplement is_domain_direct_mapped. - expand arch_domain_create/domain_create to include internal-only parameter "const unsigned int flags" - use mfn_eq() instead, because it is the only value used to indicate there is an error and this is more lightweight than mfn_valid() - rename function allocate_static_memory_11() to assign_static_memory_11() to make clear there is actually no allocation done. Instead we are only mapping pre-defined host regions to pre-defined guest regions. - remove tot_size to directly substract psize from kinfo->unassigned_mem - check kinfo->unassigned_mem doesn't underflow or overflow - remove nested if/else - remove ASSERT_UNREACHABLE() to avoid breaking compilation on prod build with CONFIG_GICV3=n - comment and commit message refinement --- v3 changes: - move flag XEN_DOMCTL_CDF_INTERNAL_directmap back to xen/include/xen/domain.h, to let it be only available for domain created by XEN. - name it with extra "INTERNAL" and add comments to warn developers not to accidently use its bitfield when introducing new XEN_DOMCTL_CDF_xxx flag. - reject this flag in x86'es arch_sanitise_domain_config() - add ASSERT_UNREACHABLE to catch any misuse in allocate_static_memory() and allocate_static_memory_11() - add another check of validating flag XEN_DOMCTL_CDF_INTERNAL_directmap only when CONFIG_STATIC_MEMORY is set. - simply map the CPU interface at the GPA vgic_v2_hw.cbase - drop 'cells += (GUEST_ROOT_ADDRESS_CELLS + GUEST_ROOT_SIZE_CELLS)' - rename 'is_domain_use_host_layout()' to 'domain_use_host_layout()' --- v2 changes: - remove the introduce of internal flag - Refine is_domain_direct_mapped to check whether the flag XEN_DOMCTL_CDF_directmap is set - reword "1:1 direct-map" to just "direct-map" - split the common codes into two helpers: parse_static_mem_prop and acquire_static_memory_bank to deduce complexity. - introduce a new helper allocate_static_memory_11 for allocating static memory for direct-map guests - remove panic action since it is fine to assign a non-DMA capable device when IOMMU and direct-map both off - remove redistributor accessor - introduce new helper "is_domain_use_host_layout()" - explain why vpl011 initialization before creating its device tree node - error out if the domain is direct-mapped and the IRQ is not found - harden the code and add a check/comment when the hardware UART region is smaller than CUEST_VPL011_SIZE. Penny Zheng (4): xen/arm: introduce new helper parse_static_mem_prop and acquire_static_memory_bank xen/arm: introduce direct-map for domUs xen/arm: add ASSERT_UNREACHABLE in allocate_static_memory xen/arm: gate make_gicv3_domU_node with CONFIG_GICV3 Stefano Stabellini (7): xen: introduce internal CDF_xxx flags for domain creation xen: introduce CDF_directmap xen/arm: Allow device-passthrough even the IOMMU is off xen/arm: if direct-map domain use native addresses for GICv2 xen/arm: if direct-map domain use native addresses for GICv3 xen/arm: if direct-map domain use native UART address and IRQ number for vPL011 xen/docs: Document how to do passthrough without IOMMU docs/misc/arm/device-tree/booting.txt | 6 + docs/misc/arm/passthrough-noiommu.txt | 52 + xen/arch/arm/domain.c | 5 +- xen/arch/arm/domain_build.c | 308 +- xen/arch/arm/include/asm/domain.h | 19 +- xen/arch/arm/include/asm/new_vgic.h | 10 + xen/arch/arm/include/asm/vgic.h | 11 + xen/arch/arm/include/asm/vpl011.h | 2 + xen/arch/arm/vgic-v2.c| 34 ++- xen/arch/arm/vgic-v3.c
[PATCH v6 03/11] xen/arm: Allow device-passthrough even the IOMMU is off
From: Stefano Stabellini
At the moment, we are only supporting device-passthrough when Xen has
enabled the IOMMU. There are some use cases where it is not possible to
use the IOMMU (e.g. doesn't exist, hardware limitation, performance) yet
it would be OK to assign a device to trusted domain so long they are
direct-mapped or the device doesn't do DMA.
Note that when the IOMMU is disabled, it will be necessary to add
xen,force-assign-without-iommu for every device that needs to be assigned.
Signed-off-by: Stefano Stabellini
Signed-off-by: Penny Zheng
Tested-by: Stefano Stabellini
---
v3 changes:
- new commit, split from the original "[PATCH v2 2/6] xen/arm: introduce
direct-map for domUs"
---
v4 changes:
- explain briefly in the commit message why we want to do device assignment
without IOMMU.
---
v5 changes:
- nothing changed
---
v6 changes
- commit message refinement
---
xen/arch/arm/domain_build.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 6467e8ee32..c1e8c99f64 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -3047,7 +3047,8 @@ void __init create_domUs(void)
panic("Missing property 'cpus' for domain %s\n",
dt_node_name(node));
-if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
+if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") &&
+ iommu_enabled )
d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
--
2.25.1
[PATCH v6 04/11] xen/arm: introduce new helper parse_static_mem_prop and acquire_static_memory_bank
Later, we will introduce assign_static_memory_11 for allocating static
memory for direct-map domains, and it will share a lot common codes with
the existing allocate_static_memory.
In order not to bring a lot of duplicate codes, and also to make the whole
code more readable, this commit extracts common codes into two new helpers
parse_static_mem_prop and acquire_static_memory_bank.
Signed-off-by: Penny Zheng
Reviewed-by: Stefano Stabellini
Tested-by: Stefano Stabellini
---
v3 changes:
- new commit, split from the original "[PATCH v2 2/6] xen/arm: introduce
direct-map for domUs"
---
v4 changes
- explain briefly in the commit message why we want to do device assignment
without IOMMU.
---
v5 changes
- fix coding style
---
v6 changes
- no changes
---
xen/arch/arm/domain_build.c | 100 +++-
1 file changed, 64 insertions(+), 36 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index c1e8c99f64..e61d2d53ba 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -509,12 +509,69 @@ static bool __init append_static_memory_to_bank(struct
domain *d,
return true;
}
+static mfn_t __init acquire_static_memory_bank(struct domain *d,
+ const __be32 **cell,
+ u32 addr_cells, u32 size_cells,
+ paddr_t *pbase, paddr_t *psize)
+{
+mfn_t smfn;
+int res;
+
+device_tree_get_reg(cell, addr_cells, size_cells, pbase, psize);
+ASSERT(IS_ALIGNED(*pbase, PAGE_SIZE) && IS_ALIGNED(*psize, PAGE_SIZE));
+if ( PFN_DOWN(*psize) > UINT_MAX )
+{
+printk(XENLOG_ERR "%pd: static memory size too large: %#"PRIpaddr,
+ d, *psize);
+return INVALID_MFN;
+}
+
+smfn = maddr_to_mfn(*pbase);
+res = acquire_domstatic_pages(d, smfn, PFN_DOWN(*psize), 0);
+if ( res )
+{
+printk(XENLOG_ERR
+ "%pd: failed to acquire static memory: %d.\n", d, res);
+return INVALID_MFN;
+}
+
+return smfn;
+}
+
+static int __init parse_static_mem_prop(const struct dt_device_node *node,
+u32 *addr_cells, u32 *size_cells,
+int *length, const __be32 **cell)
+{
+const struct dt_property *prop;
+
+prop = dt_find_property(node, "xen,static-mem", NULL);
+if ( !dt_property_read_u32(node, "#xen,static-mem-address-cells",
+ addr_cells) )
+{
+printk(XENLOG_ERR
+ "failed to read \"#xen,static-mem-address-cells\".\n");
+return -EINVAL;
+}
+
+if ( !dt_property_read_u32(node, "#xen,static-mem-size-cells",
+ size_cells) )
+{
+printk(XENLOG_ERR
+ "failed to read \"#xen,static-mem-size-cells\".\n");
+return -EINVAL;
+}
+
+*cell = (const __be32 *)prop->value;
+*length = prop->length;
+
+return 0;
+}
+
/* Allocate memory from static memory as RAM for one specific domain d. */
static void __init allocate_static_memory(struct domain *d,
struct kernel_info *kinfo,
const struct dt_device_node *node)
{
-const struct dt_property *prop;
u32 addr_cells, size_cells, reg_cells;
unsigned int nr_banks, gbank, bank = 0;
const uint64_t rambase[] = GUEST_RAM_BANK_BASES;
@@ -523,24 +580,10 @@ static void __init allocate_static_memory(struct domain
*d,
u64 tot_size = 0;
paddr_t pbase, psize, gsize;
mfn_t smfn;
-int res;
-
-prop = dt_find_property(node, "xen,static-mem", NULL);
-if ( !dt_property_read_u32(node, "#xen,static-mem-address-cells",
- &addr_cells) )
-{
-printk(XENLOG_ERR
- "%pd: failed to read \"#xen,static-mem-address-cells\".\n", d);
-goto fail;
-}
+int length;
-if ( !dt_property_read_u32(node, "#xen,static-mem-size-cells",
- &size_cells) )
-{
-printk(XENLOG_ERR
- "%pd: failed to read \"#xen,static-mem-size-cells\".\n", d);
+if ( parse_static_mem_prop(node, &addr_cells, &size_cells, &length, &cell)
)
goto fail;
-}
reg_cells = addr_cells + size_cells;
/*
@@ -551,29 +594,14 @@ static void __init allocate_static_memory(struct domain
*d,
gbank = 0;
gsize = ramsize[gbank];
kinfo->mem.bank[gbank].start = rambase[gbank];
-
-cell = (const __be32 *)prop->value;
-nr_banks = (prop->length) / (reg_cells * sizeof (u32));
+nr_banks = length / (reg_cells * sizeof (u32));
for ( ;
[PATCH v6 05/11] xen/arm: introduce direct-map for domUs
Cases where domU needs direct-map memory map:
* IOMMU not present in the system.
* IOMMU disabled if it doesn't cover a specific device and all the guests
are trusted. Thinking a mixed scenario, where a few devices with IOMMU and
a few without, then guest DMA security still could not be totally guaranteed.
So users may want to disable the IOMMU, to at least gain some performance
improvement from IOMMU disabled.
* IOMMU disabled as a workaround when it doesn't have enough bandwidth.
To be specific, in a few extreme situation, when multiple devices do DMA
concurrently, these requests may exceed IOMMU's transmission capacity.
* IOMMU disabled when it adds too much latency on DMA. For example,
TLB may be missing in some IOMMU hardware, which may bring latency in DMA
progress, so users may want to disable it in some realtime scenario.
* Guest OS relies on the host memory layout
This commit introduces a new helper assign_static_memory_11 to allocate
static memory as guest RAM for direct-map domain.
Signed-off-by: Penny Zheng
Reviewed-by: Stefano Stabellini
Tested-by: Stefano Stabellini
---
v2 changes:
- split the common codes into two helpers: parse_static_mem_prop and
acquire_static_memory_bank to deduce complexity.
- introduce a new helper allocate_static_memory_11 for allocating static
memory for direct-map guests
- remove redundant use "bool direct_map", to be replaced by
d_cfg.flags & XEN_DOMCTL_CDF_directmap
- remove panic action since it is fine to assign a non-DMA capable device when
IOMMU and direct-map both off
---
v3 changes:
- doc refinement
- drop the pointless gbank
- add check of the size of nr_banks shall not exceed NR_MEM_BANKS
- add ASSERT_UNREACHABLE to catch any misuse
- add another check of validating flag XEN_DOMCTL_CDF_INTERNAL_directmap only
when CONFIG_STATIC_MEMORY is set.
---
v4 changes:
- comment refinement
- rename function allocate_static_memory_11() to assign_static_memory_11()
to make clear there is actually no allocation done. Instead we are only
mapping pre-defined host regions to pre-defined guest regions.
- remove tot_size to directly substract psize from kinfo->unassigned_mem
- check kinfo->unassigned_mem doesn't underflow or overflow
- remove nested if/else
- refine "panic" info
---
v5 changes:
- fix coding style
---
v6 changes:
- no changes
---
xen/arch/arm/domain_build.c | 97 +++--
1 file changed, 94 insertions(+), 3 deletions(-)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index e61d2d53ba..ec29bd302c 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -494,8 +494,17 @@ static bool __init append_static_memory_to_bank(struct
domain *d,
{
int res;
unsigned int nr_pages = PFN_DOWN(size);
-/* Infer next GFN. */
-gfn_t sgfn = gaddr_to_gfn(bank->start + bank->size);
+gfn_t sgfn;
+
+/*
+ * For direct-mapped domain, the GFN match the MFN.
+ * Otherwise, this is inferred on what has already been allocated
+ * in the bank.
+ */
+if ( !is_domain_direct_mapped(d) )
+sgfn = gaddr_to_gfn(bank->start + bank->size);
+else
+sgfn = gaddr_to_gfn(mfn_to_maddr(smfn));
res = guest_physmap_add_pages(d, sgfn, smfn, nr_pages);
if ( res )
@@ -668,12 +677,92 @@ static void __init allocate_static_memory(struct domain
*d,
fail:
panic("Failed to allocate requested static memory for domain %pd.", d);
}
+
+/*
+ * Allocate static memory as RAM for one specific domain d.
+ * The static memory will be directly mapped in the guest(Guest Physical
+ * Address == Physical Address).
+ */
+static void __init assign_static_memory_11(struct domain *d,
+ struct kernel_info *kinfo,
+ const struct dt_device_node *node)
+{
+u32 addr_cells, size_cells, reg_cells;
+unsigned int nr_banks, bank = 0;
+const __be32 *cell;
+paddr_t pbase, psize;
+mfn_t smfn;
+int length;
+
+if ( parse_static_mem_prop(node, &addr_cells, &size_cells, &length, &cell)
)
+{
+printk(XENLOG_ERR
+ "%pd: failed to parse \"xen,static-mem\" property.\n", d);
+goto fail;
+}
+reg_cells = addr_cells + size_cells;
+nr_banks = length / (reg_cells * sizeof(u32));
+
+if ( nr_banks > NR_MEM_BANKS )
+{
+printk(XENLOG_ERR
+ "%pd: exceed max number of supported guest memory banks.\n", d);
+goto fail;
+}
+
+for ( ; bank < nr_banks; bank++ )
+{
+smfn = acquire_static_memory_bank(d, &cell, addr_cells, size_cells,
+ &pbase, &psize);
+if ( mfn_eq(smfn, INVALID_MFN) )
+goto fail;
+
+printk(XENLOG_INFO "%pd: STATIC BANK[%u] %#"PRIpaddr"-%#"P
[PATCH v6 06/11] xen/arm: add ASSERT_UNREACHABLE in allocate_static_memory
Helper allocate_static_memory is not meant to be reachable when built with
!CONFIG_STATIC_MEMORY, so this commit adds ASSERT_UNREACHABLE in it to catch
potential misuse.
Signed-off-by: Penny Zheng
Acked-by: Julien Grall
Tested-by: Stefano Stabellini
---
v3 changes:
- new commit
---
v4 changes:
- nothing changed
---
v5 changes:
- nothing changed
---
v6 changes:
- nothing changed
---
xen/arch/arm/domain_build.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index ec29bd302c..52f256de9c 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -755,6 +755,7 @@ static void __init allocate_static_memory(struct domain *d,
struct kernel_info *kinfo,
const struct dt_device_node *node)
{
+ASSERT_UNREACHABLE();
}
static void __init assign_static_memory_11(struct domain *d,
--
2.25.1
