apply_policy_zone() raises dynamic_policy_zone to ZONE_MOVABLE when none of a policy's nodes have non-movable memory, so a bind to movable-only node does not force kernel-zone allocations onto a zone that cannot satisfy them - resulting retries forever (livelock).
The current code uses N_HIGH_MEMORY to denote eligibility, but private nodes are deliberately not in N_HIGH_MEMORY nor N_NORMAL_MEMORY. This means a private node onlined into ZONE_NORMAL is judged as movable-only. This means a policy set with set_mempolicy() binding a private node with ZONE_NORMAL memory will shunt otherwise eligible allocations (like the task's page tables) back to N_MEMORY nodes. This also becomes relevant for in-kernel users that want non-movable memory (such as KVM placing pinned VM memory solely on a private node). Add policy_private_has_kernel_zone() to consult the actual zones of the policy's private nodes and keep the policy applied at lower zones when a kernel-zoned private node can satisfy the allocation. Only a mempolicy binding a private node pays the zone scan, all other mempolicies preserve fast-path performance. Additionally: update the comments in apply_policy_zone for clarity, because this particular policy mechanism is very confusing. Signed-off-by: Gregory Price <[email protected]> --- mm/mempolicy.c | 48 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 42 insertions(+), 6 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 90110e9761122..e83c2c7a94c1d 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2091,6 +2091,34 @@ bool vma_policy_mof(struct vm_area_struct *vma) return mof; } +/* + * true if any policy *private* node can satisfy a !__GFP_MOVABLE allocation. + * Policies without private nodes (~MPOL_F_PRIVATE) always return false. + * + * This provides a way for otherwise-bound kernel allocations to make their + * way onto private nodes with ZONE_NORMAL memory without having to audit + * every caller. The cost is a zone scan for private nodes in the mask. + */ +static bool policy_private_has_kernel_zone(const struct mempolicy *pol) +{ + int nid; + + if (!(pol->flags & MPOL_F_PRIVATE)) + return false; + + for_each_node_mask(nid, pol->nodes) { + pg_data_t *pgdat = NODE_DATA(nid); + enum zone_type zt; + + if (!node_is_private(nid)) + continue; + for (zt = ZONE_NORMAL; zt < ZONE_MOVABLE; zt++) + if (managed_zone(&pgdat->node_zones[zt])) + return true; + } + return false; +} + bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) { enum zone_type dynamic_policy_zone = policy_zone; @@ -2098,14 +2126,22 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) BUG_ON(dynamic_policy_zone == ZONE_MOVABLE); /* - * if policy->nodes has movable memory only, - * we apply policy when gfp_zone(gfp) = ZONE_MOVABLE only. + * dynamic_policy_zone is the lowest zone this policy is enforced for, + * allocations below it ignore the nodemask and fall back freely. + * + * If all policy-nodes are movable-only (all ZONE_MOVABLE), raise the + * dynamic_policy_zone to ZONE_MOVABLE so that only movable allocations + * stay bound - otherwise this would force kernel-zone allocations + * (e.g. page tables) onto a zone that cannot satisfy them. This + * results in allocation failure and retry loop (livelock). * - * policy->nodes is intersect with node_states[N_MEMORY]. - * so if the following test fails, it implies - * policy->nodes has movable memory only. + * N_HIGH_MEMORY tells us "has a non-movable zone" for ordinary nodes. + * Private nodes are deliberately excluded from N_HIGH_MEMORY, so we + * need to check their actual zones - a kernel-zoned private node can + * hold the allocation and must keep the policy applied. */ - if (!nodes_intersects(policy->nodes, node_states[N_HIGH_MEMORY])) + if (!nodes_intersects(policy->nodes, node_states[N_HIGH_MEMORY]) && + !policy_private_has_kernel_zone(policy)) dynamic_policy_zone = ZONE_MOVABLE; return zone >= dynamic_policy_zone; -- 2.53.0-Meta
