Hi Bill, We did consider the option of spreading the connects over all the workers but, because that type of performance was never an issue[0], I kept it as low priority due to the changes that would be needed. As you’ve found out, there are a number of assumptions the code currently makes with respect to the half-open pool. In particular, the workers can only read it and assume it won’t change while they do that, hence the lockless operation from their perspective and the need to handle updates via the main thread.
Moving to per-worker half-open pools would speed up the sending of syns but would also require cross-worker synchronization. If you plan to further look into this, I can try to provide some guidance but otherwise probably this won’t change short-term. Regards, Florin [0] If memory serves it should be good for O(100k) connects/s. Note however that I haven’t measure that in years .. > On Feb 5, 2021, at 11:27 AM, Bill Vaughan <billvaug...@gmail.com> wrote: > > Thanks for the feedback Florin. I'll make sure the sessions are preallocated > and the reason for doing the connect() on the main thread makes perfect > sense. I was just trying to keep the data path completely off the main > thread so it could be reserved for background tasks. I am not seeing any > performance issues at present but I wanted the design to be able to scale up > without any bottlenecks. Looks like the main thread could eventually be a > bottleneck but is ok for now. > > I'll take a look at session_mq_connect_handler(). > > Much appreciated, > -Bill > > > > > > > > > > > On Fri, Feb 5, 2021 at 11:23 AM Florin Coras <fcoras.li...@gmail.com > <mailto:fcoras.li...@gmail.com>> wrote: > Hi Bill, > > First of all, thanks for sharing the details of the project. The scale you're > looking, at least in terms of total number of active sessions should be > relatively easily achievable with the right hardware. In case you haven’t > done it yet, I’ll only recommend that you preallocate the number of sessions > and connections to the maximum you’re expecting at startup (see startup.conf > session and tcp stanzas), to avoid pool reallocation delays. > > Regarding the active opens, i.e, the connects, those are done from main > thread because for half-opens we do not know in advance the thread they will > end up on and migrating sessions and tcp connections between threads would > require locks and rpcs. So, instead, we wait for peer’s syn-ack to discover > the thread selected by RSS for the flow and only allocate the established > connection and session at that point. In contrast, for passive opens, i.e., > accepts, the syn is received on the right thread so there’s no need for extra > work. > > If you do something similar to what session_mq_connect_handler does, you can > rpc from your workers the main thread and in theory even get some coalescing > benefits as the connect requests accumulate in a queue. Did you try that and > hit performance limitations? > > Regards, > Florin > > > On Feb 5, 2021, at 6:14 AM, Bill Vaughan <billvaug...@gmail.com > > <mailto:billvaug...@gmail.com>> wrote: > > > > Hi all, > > > > I have a need to call vnet_connect() from worker threads but the stack > > requires it to be called from the main thread. For some context, the > > project I am working on includes a TCP PEP (Performance Enhancing Proxy). A > > PEP can speed up TCP connections over high latency paths, such as > > satellites. The PEP terminates the connection near the client, transports > > the data over the high latency path using a proprietary protocol, then > > reestablishes the TCP connection to the server on the other end. This > > greatly increases TCP ramp-up over high RTTs. My VPP application runs on > > both sides of the high latency path (ie: client and server model). > > > > The PEP is designed to process tens of thousands of simultaneous TCP > > connections at a rate of many 100 per second. So to achieve this I wanted > > to distribute the TCP flows across several workers but keep each individual > > flow on the same worker. Currently I have to context switch to the main > > thread to perform the connect, but all other operations can be performed > > within the worker context (accept(), svm fifo read/write). > > > > I went down the path to try to enable vnet_connect() on workers but there > > are asserts() at several places in this path. > > > > Any insight or work-around for this would be appreciated. I am using VPP > > 2009. > > > > Thanks, > > -Bill > > > > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18685): https://lists.fd.io/g/vpp-dev/message/18685 Mute This Topic: https://lists.fd.io/mt/80406012/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
