> From: Jerin Jacob [mailto:jerin.ja...@caviumnetworks.com] > Sent: Tuesday, March 28, 2017 6:36 PM > To: Van Haaren, Harry <harry.van.haa...@intel.com> > Cc: dev@dpdk.org; Richardson, Bruce <bruce.richard...@intel.com> > Subject: Re: [PATCH v5 06/20] event/sw: add support for event queues >
<snip IQ priority question> > > > A few question on everyone benefit: > > > > > > 1) Does RTE_EVENT_QUEUE_CFG_SINGLE_LINK has any other meaning other than > > > an > > > event queue linked only to single port? Based on the discussions, It was > > > add in the header file so that SW PMD can know upfront only single port > > > will be linked to the given event queue. It is added as an optimization > > > for SW > > > PMD. Does it has any functional expectation? > > > > In the context of the SW PMD, SINGLE_LINK means that a specific queue and > > port have a unique > relationship in that there is only connection. This allows bypassing of > Atomic, Ordering and > Load-Balancing code. The result is a good performance increase, particularly > if the worker port > dequeue depth is large, as then large bursts of packets can be dequeued with > little overhead. > > > > As a result, (ATOMIC | SINGLE_LINK) is not a supported combination for the > > sw pmd queue > types. > > To be more precise, a SINGLE_LINK is its own queue type, and can not be > > OR-ed with any other > type. > > > > > > > 2) Based on following topology given in documentation patch for queue > > > based event pipelining, > > > > > > rx_port w1_port > > > \ / \ > > > qid0 - w2_port - qid1 > > > \ / \ > > > w3_port tx_port > > > > > > a) I understand, rx_port is feeding events to qid0 > > > b) But, Do you see any issue with following model? IMO, It scales well > > > linearly based on number of cores available to work(Since it is ATOMIC to > > > ATOMIC). Nothing wrong with > > > qid1 just connects to tx_port, I am just trying understand the rational > > > behind it? > > > > > > rx_port w1_port w1_port > > > \ / \ / > > > qid0 - w2_port - qid1- w2_port > > > \ / \ > > > w3_port w3_port > > > > > > This is also a valid model from the SW eventdev. > > OK. If understand it correctly, On the above topology, Even though you > make qid2 as ATOMIC. SW PMD will not maintain ingress order when comes out of > qid1 on different workers. If qid0 is ORDERED, and qid1 is Atomic, then the following happens; - after qid 0, the packets are sprayed across cores, - they are returned out of order by worker cores - *at the start* of qid1, packets are re-ordered back into ingress order (maintain 100% of ordering) - on dequeue from qid1, the atomic flow distribution will keep order per flow > A SINGLE_LINK queue with one port attached > scheme is required at end of the pipeline or where ever ordering has to be > maintained. Is my understanding correct? Not quite, the SINGLE_LINK is not required at the end - we just see it as useful for common use cases. If not useful, there is no reason (due to SW PMD) for an application to create this SINGLE_LINK to finish the pipeline. If you have three cores that wish to TX, the above pipeline is 100% valid in the SW PMD case. > > The value of using a SINGLE_LINK at the end of a pipeline is > > A) can TX all traffic on a single core (using a single queue) > > B) re-ordering of traffic from the previous stage is possible > > > > To illustrate (B), a very simple pipeline here > > > > RX port -> QID #1 (Ordered) -> workers(eg 4 ports) -> QID # 2 (SINGLE_LINK > > to tx) -> TX port > > > > Here, QID #1 is allowed to send the packets out of order to the 4 worker > > ports - because they > are later passed back to the eventdev for re-ordering before they get to the > SINGLE_LINK stage, > and then TX in the correct order. > > > > > > > 3) > > > > Does anybody have a need for a queue to be both Atomic *and* > > > > Single-link? I understand > the > > > current API doesn't prohibit it, but I don't see the actual use-case in > > > which that may be > > > useful. Atomic implies load-balancing is occurring, single link implies > > > there is only one > > > consuming core. Those seem like opposites to me? > > > > > > I can think about the following use case: > > > > > > topology: > > > > > > rx_port w1_port > > > \ / \ > > > qid0 - w2_port - qid1 > > > \ / \ > > > w3_port tx_port > > > > > > Use case: > > > > > > Queue based event pipeling: > > > ORERDED(Stage1) to ATOMIC(Stage2) pipeline: > > > - For ingress order maintenance > > > - For executing Stage 1 in parallel for better scaling > > > i.e A fat flow can spray over N cores while maintaining the ingress > > > order when it sends out on the wire(after consuming from tx_port) > > > > > > I am not sure how SW PMD work in the use case of ingress order > > > maintenance. > > > > I think my illustration of (B) above is the same use-case as you have here. > > Instead of using > an ATOMIC stage2, the SW PMD benefits from using the SINGLE_LINK port/queue, > and the > SINGLE_LINK queue ensures ingress order is also egress order to the TX port. > > > > > > > But the HW and header file expects this form: > > > Snippet from header file: > > > -- > > > * The source flow ordering from an event queue is maintained when events > > > are > > > * enqueued to their destination queue within the same ordered flow > > > context. > > > * > > > * Events from the source queue appear in their original order when > > > dequeued > > > * from a destination queue. > > > -- > > > Here qid0 is source queue with ORDERED sched_type and qid1 is destination > > > queue with ATOMIC sched_type. qid1 can be linked to only port(tx_port). > > > > > > Are we on same page? If not, let me know the differences? We will try to > > > accommodate the same in header file. > > > > Yes I think we are saying the same thing, using slightly different words. > > > > To summarize; > > - SW PMD sees SINGLE_LINK as its own queue type, and does not support > > load-balanced (Atomic > Ordered, Parallel) queue functionality. > > - SW PMD would use a SINGLE_LINK queue/port for the final stage of a > > pipeline > > A) to allow re-ordering to happen if required > > B) to merge traffic from multiple ports into a single stream for TX > > > > A possible solution; > > 1) The application creates a SINGLE_LINK for the purpose of ensuring > > re-ordering is taking > place as expected, and linking only one port for TX. > > The only issue is in Low-end cores case it wont scale. TX core will become as > bottleneck and we need to have different pipelines based on the amount of > traffic(40G or 10G) > a core can handle. See above - the SINGLE_LINK isn't required to maintain ordering. Using multiple TX cores is also valid in SW PMD. > > 2) SW PMDs can create a SINGLE_LINK queue type, and benefit from the > > optimization > > Yes. > > > 3) HW PMDs can ignore the "SINGLE_LINK" aspect and uses an ATOMIC instead > > (as per your > example in 3) above) > > But topology will be fixed for both HW and SW. An extra port and > extra core needs to wasted for ordering business in case HW. Right? Nope, no wasting cores, see above :) The SINGLE_LINK is just an easy way to "fan in" traffic from lots of cores to one core (in a performant way in SW) to allow a single core do TX. A typical use-case might be putting RX and TX on the same core - TX is just a dequeue from a port with a SINGLE_LINK queue, and an enqueue to NIC. Summary from the SW PMD point-of-view; - SINGLE_LINK is its own queue type - SINGLE_LINK queue can NOT schedule according to (Atomic, Ordered or Parallel) rules Is that acceptable from an API and HW point of view? If so, I will send a new patch for the API to specify more clearly what SINGLE_LINK is. If not, I'm open to using a capability flag to solve the problem but my understanding right now is that there is no need. > I think, we can roll out something based on capability. Yes, if required that would be a good solution. > > The application doesn't have to change anything, and just configures its > > pipeline. The PMD is > able to optimize if it makes sense (SW) or just use another queue type to > provide the same > functionality to the application (HW). > > > > Thoughts? -Harry
