Thanks for the feedback Florin. I'll make sure the sessions are preallocated and the reason for doing the connect() on the main thread makes perfect sense. I was just trying to keep the data path completely off the main thread so it could be reserved for background tasks. I am not seeing any performance issues at present but I wanted the design to be able to scale up without any bottlenecks. Looks like the main thread could eventually be a bottleneck but is ok for now.
I'll take a look at session_mq_connect_handler(). Much appreciated, -Bill On Fri, Feb 5, 2021 at 11:23 AM Florin Coras <fcoras.li...@gmail.com> wrote: > Hi Bill, > > First of all, thanks for sharing the details of the project. The scale > you're looking, at least in terms of total number of active sessions should > be relatively easily achievable with the right hardware. In case you > haven’t done it yet, I’ll only recommend that you preallocate the number of > sessions and connections to the maximum you’re expecting at startup (see > startup.conf session and tcp stanzas), to avoid pool reallocation delays. > > Regarding the active opens, i.e, the connects, those are done from main > thread because for half-opens we do not know in advance the thread they > will end up on and migrating sessions and tcp connections between threads > would require locks and rpcs. So, instead, we wait for peer’s syn-ack to > discover the thread selected by RSS for the flow and only allocate the > established connection and session at that point. In contrast, for passive > opens, i.e., accepts, the syn is received on the right thread so there’s no > need for extra work. > > If you do something similar to what session_mq_connect_handler does, you > can rpc from your workers the main thread and in theory even get some > coalescing benefits as the connect requests accumulate in a queue. Did you > try that and hit performance limitations? > > Regards, > Florin > > > On Feb 5, 2021, at 6:14 AM, Bill Vaughan <billvaug...@gmail.com> wrote: > > > > Hi all, > > > > I have a need to call vnet_connect() from worker threads but the stack > requires it to be called from the main thread. For some context, the > project I am working on includes a TCP PEP (Performance Enhancing Proxy). A > PEP can speed up TCP connections over high latency paths, such as > satellites. The PEP terminates the connection near the client, transports > the data over the high latency path using a proprietary protocol, then > reestablishes the TCP connection to the server on the other end. This > greatly increases TCP ramp-up over high RTTs. My VPP application runs on > both sides of the high latency path (ie: client and server model). > > > > The PEP is designed to process tens of thousands of simultaneous TCP > connections at a rate of many 100 per second. So to achieve this I wanted > to distribute the TCP flows across several workers but keep each individual > flow on the same worker. Currently I have to context switch to the main > thread to perform the connect, but all other operations can be performed > within the worker context (accept(), svm fifo read/write). > > > > I went down the path to try to enable vnet_connect() on workers but > there are asserts() at several places in this path. > > > > Any insight or work-around for this would be appreciated. I am using > VPP 2009. > > > > Thanks, > > -Bill > > > > > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18683): https://lists.fd.io/g/vpp-dev/message/18683 Mute This Topic: https://lists.fd.io/mt/80406012/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
