Find below a summary of the technical details, implications and options
What can be done for 4.14?
We basically have two options: Revert at the driver level or ship as
is.
Even if we come up with a quick and dirty hack then it will be too late
for proper testing before sunday.
What can be done with some time to work on?
The managed mechanism consists of 3 pieces:
1) Vector spreading
2) Managed vector allocation, which becomes a guaranteed reservation in
4.15 due of the big rework of the vector management code.
Non managed interrupts get a best effort reservation to handle theCPU
unplug vector pressure problem in a sane way.
3) CPU hotplug management
If the last CPU in the affinity set goes offline, then the interrupt is
shutdown and restarted when the first CPU in the affinity set comes
online again. The driver code needs to ensure that the queue associated
to that interrupt is drained before shutdown and nothing is queued
there after this point.
So we have options:
1) Initial vector spreading
Let the driver use the initial vector spreading. That does only the
initial affinity setup, but otherwise the interrupts are handled like any
other non managed interrupt, i.e. best effort reservation, affinity
settings enabled and CPU unplug breaks affinity and moves them to some
random other online CPU.
The simplest solution of all.
2) Allowing a driver supplied mask
Certainly simple to do, but as you said it's not really a solution. I'm
not sure whether we want to go there as this is going to be replaced fast
enough and then create another breakage/frustration level.
3) Affinity override in managed mode
Doable, but there are a couple of things to think about:
* How is this enabled?
- Opt-in by driver
- Extra sysfs/procfs knob
We definitely should not enable it per default because that would
surprise users/drivers which work with the current managed devices and
rely on the affinity files to be non writeable in managed mode.
* Is it allowed to set the affinity to offline, but present CPUs?
In principle yes, because the core management code can do that as well
at setup time.
* The affinity setting must fail when it cannot do a guaranteed
reservation on the new target CPU(s).
This is not much of a question. That's a matter of fact because
otherwise the association cannot be guaranteed and things fall apart
all over the place.
* When and how is the driver informed about the change?
When:
#1 Before the core tries to move the interrupt so it can veto the
move if it cannot allocate new resources or whatever is required
to operate after the move.
#2 After the core made the move effective because:
- The interrupt might be moved from an offline set to an online
set and needs to be started up, so the related queue must be
enabled as well.
- The interrupt might be moved from an online set to an offline
set, so the queue needs to be drained and disabled.
- Resources which have been allocated in the first step must be
made effective and old resources freed.
How:
The existing affinity notification mechanism does not work for this
and it's a horrible piece of crap which should go away sooner than
later.
So we need some sensible way to provide callback. Emphasis on
callbacks as one multiplexing callback is not a good idea.
* How can the change made effective?
When the preliminaries (vector reservation on the new set and
evtl. resource allocation in the subsystem have been done, then the
actual move can be made.
But, there is a caveat. x86 is not good in reassociating interrupts on
the fly except when it sits behind an interrupt remapping unit, but we
cannot rely on that.
So the change flow which works for everything would be:
if (reserve_vectors() < 0)
return FAIL;
if (subsys_prep_callback() < 0) {
release_vectors();
return FAIL;
}
shutdown(irq);
if (!online(newset))
return SUCCESS;
startup(irq);
subsys_post_callback();
return SUCCESS;
subsys_prep_callback() must basically work the same way as the CPU
offline mechanism and drain the queue and prevent queueing before the
irq is restarted. If the move results in keeping it shutdown because
the new set is offline, then the irq will be restarted via the CPU
hotplug code and the subsystem will be informed about that via the
hotplug mechanism as well.
subsys_post_callback() is more or less the same as the hotplug callback
and restarts the queue. The only difference to the hotplug code as of
today is that it might need to make previously allocated resources
effective and free the old ones.
I named that subsys_*_callback() on purpose because this should be
handled in a generic way for multiqueue devices and not done at the
driver level.
There are some very interesting locking problems to solve, especially
vs. CPU hotplug, but that should be solvable.
4) Break managed mode when affinity is changed by user
I'm not going to describe that because this is going to require at least
as much effort as #2 plus a few extra interesting twists versus vector
management and CPU hotplug.
5) Other options:
Maybe ponies, but I have no clue how to implement them.
Thoughts?
Thanks,
tglx
