On Mon, Aug 24, 2020 at 08:45:12AM -0300, Daniel Henrique Barboza wrote: > > > On 8/24/20 3:08 AM, David Gibson wrote: > > On Fri, Aug 21, 2020 at 09:47:47AM -0300, Daniel Henrique Barboza wrote: > > > > > > > > > On 8/21/20 5:55 AM, Igor Mammedov wrote: > > > > On Thu, 20 Aug 2020 12:51:03 -0400 > > > > Eduardo Habkost <ehabk...@redhat.com> wrote: > > > > > > > > > On Thu, Aug 20, 2020 at 02:15:04PM +1000, David Gibson wrote: > > > > > > On Wed, Aug 19, 2020 at 10:11:28PM -0400, Eduardo Habkost wrote: > > > > > > > On Thu, Aug 20, 2020 at 11:17:26AM +1000, David Gibson wrote: > > > > > > > > On Fri, Aug 14, 2020 at 05:54:16PM -0300, Daniel Henrique > > > > > > > > Barboza wrote: > > > > > > > > > The pSeries machine does not support asymmetrical NUMA > > > > > > > > > configurations. > > > > > > > > > > > > > > > > This seems a bit oddly specific to have as a global machine > > > > > > > > class > > > > > > > > property. > > > > > > > > > > > > > > > > Would it make more sense for machines with specific NUMA > > > > > > > > constraints > > > > > > > > to just verify those during their initialization? > > > > > > > > > > > > > > This would be much simpler. However, I like the idea of > > > > > > > representing machine-specific configuration validation rules as > > > > > > > data that can eventually be exported to management software. > > > > > > > > > > > > Ah, ok, so basically the usual tradeoff between flexibility and > > > > > > advertisability. > > > > > > > > > > > > So, in that case, I guess the question is whether we envisage "no > > > > > > assymmetry" as a constraint common enough that it's worth creating > > > > > > an > > > > > > advertisable rule or not. If we only ever have one user, then we > > > > > > haven't really done any better than hard coding the constraint in > > > > > > the > > > > > > manageent software. > > > > > > > > > > > > Of course to complicate matters, in the longer term we're looking at > > > > > > removing that constraint from pseries - but doing so will be > > > > > > dependent > > > > > > on the guest kernel understanding a new format for the NUMA > > > > > > information in the device tree. So qemu alone won't have enough > > > > > > information to tell if such a configuration is possible or not. > > > > > > > > > > Requiring both QEMU (and possibly management software) to be > > > > > patched again after the guest kernel is fixed sounds undesirable. > > > > If we drop this restriction, then we don't need to touch QEMU when > > > > guest kernel is ready. > > > > > > > > Btw, what spapr spec says about the matter? > > > > > > LOPAPR support a somewhat asymmetrical NUMA setup in its current > > > form, > > > > Huh, I didn't even realize that. What's the mechanism? > > LOPAPR mentions that a single resource/node can have multiple associativity > arrays. The idea is to contemplate the situations where the node has > more than one connection with the board. > > I say "somewhat" because, right after mentioning that, the spec also says that > the OS should consider that the distance between two nodes must always be > the shortest one of all available arrays. I'll copy/paste the except here > (end of section 15.2, "Numa Resource Associativity":
Ah. I didn't think that's what "asymmetric NUMA" meant... but come to
think of it, I'm not very sure about that.
> -----
>
> The reason that the “ibm,associativity” property may contain multiple
> associativity
> lists is that a resource may be multiply connected into the platform. This
> resource
> then has a different associativity characteristics relative to its multiple
> connections.
> To determine the associativity between any two resources, the OS scans down
> the two
> resources associativity lists in all pair wise combinations counting how many
> domains
> are the same until the first domain where the two list do not agree. The
> highest such
> count is the associativity between the two resources.
>
> ----
>
>
> DHB
>
>
> >
> > > but
> > > the Linux kernel doesn't support it. The effort to implement it in the
> > > current
> > > spapr machine code, given that Linux wouldn't mind it, is not worth it.
> > > This
> > > is why I chose to invalidate it for pseries.
> >
> > Igor,
> >
> > It's kind of difficult to answer that question - PAPR doesn't
> > specifically describe limitations, it's just that the representation
> > it uses is inherently limited. Instead of the obvious, simple and
> > pretty much universal method (used in the generic kernel and qemu) of
> > having a matrix of distance between all the nodes, it instead
> > describes the hierarchy of components that give rise to the different
> > distances.
> >
> > So, for each NUMA relevant object (cpu, memory block, host bridge,
> > etc.) there is a vector of IDs. Each number in the vector gives one
> > level of the objects location in the heirarchy.
> >
> > So, for example the first number might be the physical chip/socket.
> > the second one which group of cores & memory interfaces sharing an Ln
> > cache, the third one the specific core number. So to work out how far
> > objects are from each other you essentially look at how long a prefix
> > of their vector they share, which tells you how far above in the
> > hierarchy you have to go to reach it.
> >
> > There's a bunch of complicating details, but that's the gist of it.
> >
> > > > > Perhaps a warning would be better in this case?
> > > > >
> > > > > In either case, it sounds like this won't be a common constraint
> > > > > and I now agree with your original suggestion of doing this in
> > > > > machine initialization code.
> > > > Agreed, if it goes to spapr specific machine code I will not object
> > > > much.
> > > > (it will burden just spapr maintainers, so it's about convincing
> > > > David in the end)
> > >
> > > I believe he's ok with it given that he suggested it in his first reply.
> > >
> > > I'll move this verification to spapr machine_init in the next version.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > DHB
> > >
> > >
> > > >
> > >
> >
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
signature.asc
Description: PGP signature
