Hi Damjan, I would say there is nothing we can do but to drop the packets. Developers have to take care of how to use handoff mechanism. In our case or NAT case, it has to be ‘no wait’, while it could be ‘wait forever’ in a stream line mode – worker A -> B -> C.
BTW, my colleague Lollita thought there might be some improvement we can do to the handoff queue. Please Lollita share what you found. Regards, Kingwel From: Damjan Marion (damarion) [mailto:damar...@cisco.com] Sent: Monday, April 23, 2018 8:32 PM To: Kingwel Xie <kingwel....@ericsson.com> Cc: Ole Troan <otr...@employees.org>; vpp-dev <vpp-dev@lists.fd.io>; Lollita Liu <lollita....@ericsson.com> Subject: Re: [vpp-dev] Question of worker thread handoff Dear Kingwel, What would you expect from us to do if A waits for B to take stuff from the queue and on the same time B waits for A for the same reason beside what we already do in NAT code, and that is to drop instead of wait. -- Damjan On 23 Apr 2018, at 14:14, Kingwel Xie <kingwel....@ericsson.com<mailto:kingwel....@ericsson.com>> wrote: Hi Ole, Damjan, Thanks, for the comments. But I’m afraid this is the typical case that workers handoff to each other if we don’t want to create an I/O thread which might become the bottleneck in the end. Regards, Kingwel From: Damjan Marion (damarion) [mailto:damar...@cisco.com] Sent: Monday, April 23, 2018 6:25 PM To: Ole Troan <otr...@employees.org<mailto:otr...@employees.org>>; Kingwel Xie <kingwel....@ericsson.com<mailto:kingwel....@ericsson.com>> Cc: vpp-dev <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>>; Lollita Liu <lollita....@ericsson.com<mailto:lollita....@ericsson.com>> Subject: Re: [vpp-dev] Question of worker thread handoff Yes, there are 2 options when handoff queue is full, drop or wait. Wait gives you nice back-pressure mechanism as it will slow down input worker, but it will not work in case when A handoffs to B and B handoffs to A. -- Damjan On 23 Apr 2018, at 10:50, Ole Troan <otr...@employees.org<mailto:otr...@employees.org>> wrote: Kingwei, Yes, it's possible to dead lock in this case. We had a similar issue with the NAT implementation. While testing I think we ended up dropping when the queue was full. Best regards, Ole On 23 Apr 2018, at 10:33, Kingwel Xie <kingwel....@ericsson.com<mailto:kingwel....@ericsson.com>> wrote: Hi Damjan and all, We are currently thinking of how to utilize the handoff mechanism to serve our application logic – to run a packet re-ordering and re-transmit queue in the same worker context to avoid any lock between threads. We come across with a question when looking into the implementation of handoff. Hope you can figure it out for us. In vlib_get_worker_handoff_queue_elt -> vlib_get_frame_queue_elt : /* Wait until a ring slot is available */ while (new_tail >= fq->head_hint + fq->nelts) vlib_worker_thread_barrier_check (); We understand that the worker has wait for the any available slot from the other worker before putting buffer into it. Then the question is: is it a potential dead lock that two workers waiting for each other? F.g., Worker A and B, A is going to handoff to B but unfortunately at the same time B has the same thing to do to A, then they are both waiting forever. If it is true, is it better to drop the packet when the ring is full? I copied my colleague Lollita into this discussion, she is working on it and knows better about this hypothesis. Regards, Kingwel
