On 07/30/15 20:33, Zhang, Helin wrote: > >> -----Original Message----- >> From: Vlad Zolotarov [mailto:vladz at cloudius-systems.com] >> Sent: Thursday, July 30, 2015 9:44 AM >> To: Zhang, Helin; Ananyev, Konstantin >> Cc: dev at dpdk.org >> Subject: Re: i40e xmit path HW limitation >> >> >> >> On 07/30/15 19:10, Zhang, Helin wrote: >>>> -----Original Message----- >>>> From: Vlad Zolotarov [mailto:vladz at cloudius-systems.com] >>>> Sent: Thursday, July 30, 2015 7:58 AM >>>> To: dev at dpdk.org; Ananyev, Konstantin; Zhang, Helin >>>> Subject: RFC: i40e xmit path HW limitation >>>> >>>> Hi, Konstantin, Helin, >>>> there is a documented limitation of xl710 controllers (i40e driver) >>>> which is not handled in any way by a DPDK driver. >>>> From the datasheet chapter 8.4.1: >>>> >>>> "? A single transmit packet may span up to 8 buffers (up to 8 data >>>> descriptors per packet including both the header and payload buffers). >>>> ? The total number of data descriptors for the whole TSO (explained >>>> later on in this chapter) is unlimited as long as each segment within >>>> the TSO obeys the previous rule (up to 8 data descriptors per segment >>>> for both the TSO header and the segment payload buffers)." >>> Yes, I remember the RX side just supports 5 segments per packet receiving. >>> But what's the possible issue you thought about? >> Note that it's a Tx size we are talking about. >> >> See 30520831f058cd9d75c0f6b360bc5c5ae49b5f27 commit in linux net-next repo. >> If such a cluster arrives and you post it on the HW ring - HW will shut this >> HW ring >> down permanently. The application will see that it's ring is stuck. > That issue was because of using more than 8 descriptors for a packet for TSO.
There is no problem in transmitting the TSO packet with more than 8 fragments. On the opposite - one can't transmit a non-TSO packet with more than 8 fragments. One also can't transmit the TSO packet that would contain more than 8 fragments in a single TSO segment including the TSO headers. Pls., read the HW spec as I quoted above for more details. > >>>> This means that, for instance, long cluster with small fragments has to be >>>> linearized before it may be placed on the HW ring. >>> What type of size of the small fragments? Basically 2KB is the default size >>> of >> mbuf of most >>> example applications. 2KB x 8 is bigger than 1.5KB. So it is enough for the >> maximum >>> packet size we supported. >>> If 1KB mbuf is used, don't expect it can transmit more than 8KB size of >>> packet. >> I kinda lost u here. Again, we talk about the Tx side here and buffers >> are not obligatory completely filled. Namely there may be a cluster with >> 15 fragments 100 bytes each. > The root cause is using more than 8 descriptors for a packet. That would be if u would like to SUPER simplify the HW limitation above. In that case u would significantly limit the different packets that may be sent without the linearization. > Linux driver can help > on reducing number of descriptors to be used by merging small size of payload > together, right? > It is not for TSO, it is just for packet transmitting. 2 options in my mind: > 1. Use should ensure it will not use more than 8 descriptors per packet for > transmitting. This requirement is too restricting. Pls., see above. > 2. DPDK driver should try to merge small packet together for such case, like > Linux kernel driver. > I prefer to use option 1, users should ensure that in the application or up > layer software, > and keep the PMD driver as simple as possible. The above statement is super confusing: on the one hand u suggest the DPDK driver to merge the small packet (fragments?) together (how?) and then u immediately propose the user application to do that. Could u, pls., clarify what exactly u suggest here? If that's to leave it to the application - note that it would demand patching all existing DPDK applications that send TCP packets. > > But I have a thought that the maximum number of RX/TX descriptor should be > able to be > queried somewhere. There is no such thing as maximum number of Tx fragments in a TSO case. It's only limited by the Tx ring size. > > Regards, > Helin >>>> In more standard environments like Linux or FreeBSD drivers the solution is >>>> straight forward - call skb_linearize()/m_collapse() corresponding. >>>> In the non-conformist environment like DPDK life is not that easy - there >>>> is no >>>> easy way to collapse the cluster into a linear buffer from inside the >>>> device >> driver >>>> since device driver doesn't allocate memory in a fast path and utilizes >>>> the user >>>> allocated pools only. >>>> Here are two proposals for a solution: >>>> >>>> 1. We may provide a callback that would return a user TRUE if a give >>>> cluster has to be linearized and it should always be called before >>>> rte_eth_tx_burst(). Alternatively it may be called from inside the >>>> rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return some >>>> error code for a case when one of the clusters it's given has to be >>>> linearized. >>>> 2. Another option is to allocate a mempool in the driver with the >>>> elements consuming a single page each (standard 2KB buffers would >>>> do). Number of elements in the pool should be as Tx ring length >>>> multiplied by "64KB/(linear data length of the buffer in the pool >>>> above)". Here I use 64KB as a maximum packet length and not taking >>>> into an account esoteric things like "Giant" TSO mentioned in the >>>> spec above. Then we may actually go and linearize the cluster if >>>> needed on top of the buffers from the pool above, post the buffer >>>> from the mempool above on the HW ring, link the original cluster to >>>> that new cluster (using the private data) and release it when the >>>> send is done. >>>> >>>> >>>> The first is a change in the API and would require from the application >>>> some >>>> additional handling (linearization). The second would require some >>>> additional >>>> memory but would keep all dirty details inside the driver and would leave >>>> the >>>> rest of the code intact. >>>> >>>> Pls., comment. >>>> >>>> thanks, >>>> vlad >>>>
