Hi Nate correct, dummy interfaces are not always a viable solution based on the traffic rate. For better performance you should use native zc queues, however as you noticed if the Bro process crashes, it leaves the queue in an inconsistent state and the cluster does not allow you to use it again.
Alfredo > On 31 Jan 2019, at 22:36, Alexander, Nathan > <nathan.alexan...@morganstanley.com> wrote: > > > So what is the recommended way to run zbalance_ipc with docker containers? > https://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/ > > <https://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/> > It sounds like the dummy interfaces are not very scalable. > > I’ve got everything working native with pf_ring directly, but when a > container crashes/stops, the zbalance queue disappears (e.g. if I start > zbalance –c 4 –n 1,11, and do a pf_count on zc:4@0, things work fine, but if > I start a consumer on zc:4@0 and it crashes/stops, I can’t do a pf_count on > zc:4@0 any longer). > > Cheers, > > Nate > > > > From: ntop-boun...@listgateway.unipi.it > [mailto:ntop-boun...@listgateway.unipi.it] On Behalf Of Alfredo Cardigliano > Sent: Monday, January 28, 2019 1:29 PM > To: n...@unipi.it > Cc: ntop@listgateway.unipi.it > Subject: Re: [Ntop] Zbalance Issue > > Hi Nate > please note that a dummy interface is a kernel interface with limited > performance. This regardless of the consumers actually running (when > you run a consumer, you just add load increasing the packet loss), as > zbalance_ipc is sending traffic to the kernel anyway. > > Best Regards > Alfredo > > > On 28 Jan 2019, at 17:46, Alexander, Nathan > <nathan.alexan...@morganstanley.com > <mailto:nathan.alexan...@morganstanley.com>> wrote: > > > I am having an issue with zbalance_ipc where we are dropping 10% of packets > at 1.5Gbps+/interface even before any consuming applications begin to > consume the data. The setup is a 2x18 core server where one application is > using a single queue and the other application is consuming 18 queues, and we > have a zbalance process per NUMA Node. When we start z_balance with only one > application (-n 1), we see no loss but with -n 18,1, we see ~10% packet drops > before anything is hooked to the dummies: > > 23/Jan/2019 21:58:59 [zbalance_ipc.c:266] Absolute Stats: Recv 1'547'957'308 > pkts (138'616'411 drops) - Forwarded 3'095'914'616 pkts (0 drops) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:305] zc:p1p2 RX > 1547957313 pkts Dropped 138616411 pkts (8.2 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 0 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 1 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 2 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 3 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 4 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 5 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 6 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 7 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 8 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 9 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 10 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 11 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 12 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 13 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 14 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 15 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 16 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 17 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:319] Q 18 RX 0 pkts > Dropped 0 pkts (0.0 %) > 23/Jan/2019 21:58:59 [zbalance_ipc.c:338] Actual Stats: Recv 256'628.26 pps > (0.00 drops) - Forwarded 513'257.53 pps (0.00 drops) > > The interfaces are intel NICs, 2x10GE but we're only consuming data of of one > interface on each card. > > 0% packet loss (running as background so that I can get logs, but normally > run with -d): > sudo /usr/bin/zbalance_ipc -i zc:p1p2 -a -q 131072 -c 99 -n 1 -m 1 -S 4 -g 2 > -r 0:dummy57 -l /var/log/zbalance_p1p2.log -v -p -u /dev/hugepages & > sudo /usr/bin/zbalance_ipc -i zc:p4p2 -a -q 131072 -c 97 -n 1 -m 1 -S 5 -g 3 > -r 0:dummy59 -l /var/log/zbalance_p4p2.log -v -p -u /dev/hugepages & > > 10% Packet drops to start, greater with actual consumers on the dummies: > > sudo /usr/bin/zbalance_ipc -i zc:p1p2 -a -q 131072 -c 98 -n 18,1 -m 4 -S 8 -g > 6 -r 0:dummy20 -r 1:dummy21 -r 2:dummy22 -r 3:dummy23 -r 4:dummy24 -r > 5:dummy25 -r 6:dummy26 -r 7:dummy27 -r 8:dummy28 -r 9:dummy30 -r 10:dummy31 > -r 11:dummy32 -r 12:dummy33 -r 13:dummy34 -r 14:dummy35 -r 15:dummy36 -r > 16:dummy37 -r 17:dummy38 -r 18:dummy57 -l /var/log/zbalance_p1p2.log -v -p -u > /dev/hugepages & > > sudo /usr/bin/zbalance_ipc -i zc:p4p2 -a -q 131072 -c 96 -n 18,1 -m 4 -S 9 -g > 7 -r 0:dummy0 -r 1:dummy1 -r 2:dummy2 -r 3:dummy3 -r 4:dummy4 -r 5:dummy5 -r > 6:dummy6 -r 7:dummy7 -r 8:dummy8 -r 9:dummy10 -r 10:dummy11 -r 11:dummy12 -r > 12:dummy13 -r 13:dummy14 -r 14:dummy15 -r 15:dummy16 -r 16:dummy17 -r > 17:dummy18 -r 18:dummy59 -l /var/log/zbalance_p4p2.log -v -p -u > /dev/hugepages & > > p4p2 is hooked to NUMA 1 > P1p2 is hooked to NUMA 0 > Other relevant data from the server: > NUMA node0 CPU(s): > 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70 > NUMA node1 CPU(s): > 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71 > > ixgbe.conf: > RSS=1,1,1,1,1,1 numa_cpu_affinity=2,2,2,2,3,3 > > PF Ring version is: > zbalance_ipc -h > zbalance_ipc - (C) 2014-2018 ntop.org <http://ntop.org/> > Using PFRING_ZC v.7.3.0.180817 > > Let me know if I'm missing something obvious or any ideas on how to improve > performance. Thanks so much! > > Cheers, > > Nate > > > > > > > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions > or views contained herein are not intended to be, and do not constitute, > advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform > and Consumer Protection Act. If you have received this communication in > error, please destroy all electronic and paper copies and notify the sender > immediately. Mistransmission is not intended to waive confidentiality or > privilege. Morgan Stanley reserves the right, to the extent required and/or > permitted under applicable law, to monitor electronic communications, > including telephone calls with Morgan Stanley personnel. This message is > subject to the Morgan Stanley General Disclaimers available at the following > link: http://www.morganstanley.com/disclaimers > <http://www.morganstanley.com/disclaimers>. If you cannot access the links, > please notify us by reply message and we will send the contents to you. By > communicating with Morgan Stanley you acknowledge that you have read, > understand and consent, (where applicable), to the foregoing and the Morgan > Stanley General Disclaimers. > > > > > > > > _______________________________________________ > Ntop mailing list > Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it> > http://listgateway.unipi.it/mailman/listinfo/ntop > <http://listgateway.unipi.it/mailman/listinfo/ntop> > > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions > or views contained herein are not intended to be, and do not constitute, > advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform > and Consumer Protection Act. If you have received this communication in > error, please destroy all electronic and paper copies and notify the sender > immediately. Mistransmission is not intended to waive confidentiality or > privilege. Morgan Stanley reserves the right, to the extent required and/or > permitted under applicable law, to monitor electronic communications, > including telephone calls with Morgan Stanley personnel. This message is > subject to the Morgan Stanley General Disclaimers available at the following > link: http://www.morganstanley.com/disclaimers > <http://www.morganstanley.com/disclaimers>. If you cannot access the links, > please notify us by reply message and we will send the contents to you. By > communicating with Morgan Stanley you acknowledge that you have read, > understand and consent, (where applicable), to the foregoing and the Morgan > Stanley General Disclaimers. > > > > > > > > _______________________________________________ > Ntop mailing list > Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it> > http://listgateway.unipi.it/mailman/listinfo/ntop > <http://listgateway.unipi.it/mailman/listinfo/ntop>
_______________________________________________ Ntop mailing list Ntop@listgateway.unipi.it http://listgateway.unipi.it/mailman/listinfo/ntop
