Hi David it seems that you have packet loss both on zbalance and nprobe, I recommend you to: 1. set the core affinity for both zbalance_ipc and the nprobe instances, trying to use a different core for each (at least do not share the zbalance_ipc physical core with nprobe instances) 2. did you try using zc drivers for capturing traffic from the interfaces? (zc:p2p1,zc:p2p2) Please also provide the top output (press 1 to see all cored) with the current configuration, I guess kernel is using some of the available cpu with this configuration.
Alfredo > On 26 Jun 2018, at 16:31, David Notivol <[email protected]> wrote: > > Hi Alfredo, > Thanks for replying. > This is an excerpt of the zbalance and nprobe statistics: > > 26/Jun/2018 17:29:58 [zbalance_ipc.c:265] ========================= > 26/Jun/2018 17:29:58 [zbalance_ipc.c:266] Absolute Stats: Recv 1'285'430'239 > pkts (1'116'181'903 drops) - Forwarded 1'266'272'285 pkts (19'157'949 drops) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:305] p2p1,p2p2 RX > 1285430267 pkts Dropped 1116181981 pkts (46.5 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 0 RX 77050882 > pkts Dropped 1127883 pkts (1.4 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 1 RX 70722562 > pkts Dropped 756409 pkts (1.1 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 2 RX 76092418 > pkts Dropped 1017335 pkts (1.3 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 3 RX 75088386 > pkts Dropped 896678 pkts (1.2 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 4 RX 91991042 > pkts Dropped 2114739 pkts (2.2 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 5 RX 81384450 > pkts Dropped 1269385 pkts (1.5 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 6 RX 84310018 > pkts Dropped 1801848 pkts (2.1 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 7 RX 84554242 > pkts Dropped 1487329 pkts (1.7 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 8 RX 84090370 > pkts Dropped 1482864 pkts (1.7 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 9 RX 73642498 > pkts Dropped 732237 pkts (1.0 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 10 RX 76481026 > pkts Dropped 1000496 pkts (1.3 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 11 RX 72496642 > pkts Dropped 929049 pkts (1.3 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 12 RX 79386626 > pkts Dropped 1122169 pkts (1.4 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 13 RX 79418370 > pkts Dropped 1187172 pkts (1.5 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 14 RX 80284162 > pkts Dropped 1195559 pkts (1.5 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 15 RX 79143426 > pkts Dropped 1036797 pkts (1.3 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:338] Actual Stats: Recv 369'127.51 pps > (555'069.74 drops) - Forwarded 369'129.51 pps (0.00 drops) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:348] ========================= > > > # cat /proc/net/pf_ring/stats/* > ClusterId: 1 > TotQueues: 16 > Applications: 1 > App0Queues: 16 > Duration: 0:00:41:18:386 > Packets: 1191477340 > Forwarded: 1174033613 > Processed: 1173893301 > IFPackets: 1191477364 > IFDropped: 1036448041 > > Duration: 0:00:41:15:587 > Bytes: 42626434538 > Packets: 71510530 > Dropped: 845465 > > Duration: 0:00:41:15:557 > Bytes: 40686677370 > Packets: 65656322 > Dropped: 533675 > > Duration: 0:00:41:15:534 > Bytes: 41463519299 > Packets: 70565378 > Dropped: 804282 > > Duration: 0:00:41:15:523 > Bytes: 42321923225 > Packets: 69566978 > Dropped: 650333 > > Duration: 0:00:41:14:659 > Bytes: 45415334638 > Packets: 85479938 > Dropped: 1728521 > > Duration: 0:00:41:14:597 > Bytes: 42615821825 > Packets: 75445250 > Dropped: 951386 > > Duration: 0:00:41:14:598 > Bytes: 44722410915 > Packets: 78252409 > Dropped: 1479387 > > Duration: 0:00:41:14:613 > Bytes: 44788855334 > Packets: 78318926 > Dropped: 1202905 > > Duration: 0:00:41:14:741 > Bytes: 43950263720 > Packets: 77821954 > Dropped: 1135693 > > Duration: 0:00:41:14:608 > Bytes: 41211162757 > Packets: 68241354 > Dropped: 496494 > > Duration: 0:00:41:14:629 > Bytes: 43064091353 > Packets: 70834104 > Dropped: 712427 > > Duration: 0:00:41:14:551 > Bytes: 42072869897 > Packets: 67360770 > Dropped: 696460 > > Duration: 0:00:41:14:625 > Bytes: 44323715294 > Packets: 73420290 > Dropped: 851818 > > Duration: 0:00:41:14:625 > Bytes: 43018671083 > Packets: 73651110 > Dropped: 917985 > > Duration: 0:00:41:14:600 > Bytes: 42730057210 > Packets: 74312500 > Dropped: 799922 > > Duration: 0:00:41:14:611 > Bytes: 42519248547 > Packets: 73394690 > Dropped: 771941 > > > > El mar., 26 jun. 2018 a las 16:25, Alfredo Cardigliano (<[email protected] > <mailto:[email protected]>>) escribió: > Hi David > please also provide statistics from zbalance_ipc (output or log file) > and nprobe (you can get live stats from /proc/net/pf_ring/stats/) > > Thank you > Alfredo > >> On 26 Jun 2018, at 15:32, David Notivol <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hello list, >> >> We're using nProbe to export flows information to kafka. We're listening >> from two 10Gb interfaces that we merge with zbalance_ipc, and we split them >> into 16 queues to have 16 nprobe instances. >> >> The problem is we are seeing about 40% packet drops reported by >> zbalance_ipc, so it looks like nprobe is not capable of reading and >> processing all the traffic. The CPU usage is really high, and the load >> average is over 25-30. >> >> Merging both interfaces we're getting up to 5.5 Gbps, and 1.2 million >> packets / second; and we're using i40e_zc driver. >> >> Do you have any advice to try to improve this performance? >> Does it make sense we're having packet drops with this amount of traffic, >> and we're reaching the server limits? Or is any configuration we could tune >> up to improve it? >> >> Thanks in advance. >> >> >> >> -- System: >> >> nProbe: nProbe v.8.5.180625 (r6185) >> System RAM: 64GB >> System CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 12 cores (6 cores, >> 2 threads per core) >> System OS: CentOS Linux release 7.4.1708 (Core) >> Linux Kernel: 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC >> 2018 x86_64 x86_64 x86_64 GNU/Linux >> >> -- zbalance configuration: >> >> zbalance_ipc -i p2p1,p2p2 -c 1 -n 16 -m 4 -a -p -l /var/tmp/zbalance.log -v >> -w >> >> -- nProbe configuration: >> >> --interface=zc:1@0 >> --pid-file=/var/run/nprobe-zc1-00.pid >> --dump-stats=/var/log/nprobe/zc1-00_flows_stats.txt >> --kafka "192.168.0.1:9092 <http://192.168.0.1:9092/>,192.168.0.2:9092 >> <http://192.168.0.2:9092/>,192.168.0.3:9092;topic" >> --collector=none >> --idle-timeout=60 >> --snaplen=128 >> --aggregation=0/1/1/1/0/0/0 >> --all-collectors=0 >> --verbose=1 >> --dump-format=t >> --vlanid-as-iface-idx=none >> --hash-size=1024000 >> --flow-delay=1 >> --count-delay=10 >> --min-flow-size=0 >> --netflow-engine=0:0 >> --sample-rate=1:1 >> --as-list=/usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat >> --city-list=/usr/share/ntopng/httpdocs/geoip/GeoLiteCity.dat >> --flow-templ="%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %IN_BYTES %OUT_PKTS >> %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT >> %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS %L7_PROTO %L7_PROTO_NAME >> %SRC_IP_COUNTRY %SRC_IP_CITY %SRC_IP_LONG %SRC_IP_LAT %DST_IP_COUNTRY >> %DST_IP_CITY %DST_IP_LONG %DST_IP_LAT %SRC_VLAN %DST_VLAN %DOT1Q_SRC_VLAN >> %DOT1Q_DST_VLAN %DIRECTION %SSL_SERVER_NAME %SRC_AS_MAP %DST_AS_MAP >> %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST >> %HTTP_SITE %UPSTREAM_TUNNEL_ID %UPSTREAM_SESSION_ID %DOWNSTREAM_TUNNEL_ID >> %DOWNSTREAM_SESSION_ID %UNTUNNELED_PROTOCOL %UNTUNNELED_IPV4_SRC_ADDR >> %UNTUNNELED_L4_SRC_PORT %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_L4_DST_PORT >> %GTPV2_REQ_MSG_TYPE %GTPV2_RSP_MSG_TYPE %GTPV2_C2S_S1U_GTPU_TEID >> %GTPV2_C2S_S1U_GTPU_IP %GTPV2_S2C_S1U_GTPU_TEID %GTPV2_S5_S8_GTPC_TEID >> %GTPV2_S2C_S1U_GTPU_IP %GTPV2_C2S_S5_S8_GTPU_TEID %GTPV2_S2C_S5_S8_GTPU_TEID >> %GTPV2_C2S_S5_S8_GTPU_IP %GTPV2_S2C_S5_S8_GTPU_IP %GTPV2_END_USER_IMSI >> %GTPV2_END_USER_MSISDN %GTPV2_APN_NAME %GTPV2_ULI_MCC %GTPV2_ULI_MNC >> %GTPV2_ULI_CELL_TAC %GTPV2_ULI_CELL_ID %GTPV2_RESPONSE_CAUSE %GTPV2_RAT_TYPE >> %GTPV2_PDN_IP %GTPV2_END_USER_IMEI %GTPV2_C2S_S5_S8_GTPC_IP >> %GTPV2_S2C_S5_S8_GTPC_IP %GTPV2_C2S_S5_S8_SGW_GTPU_TEID >> %GTPV2_S2C_S5_S8_SGW_GTPU_TEID %GTPV2_C2S_S5_S8_SGW_GTPU_IP >> %GTPV2_S2C_S5_S8_SGW_GTPU_IP %GTPV1_REQ_MSG_TYPE %GTPV1_RSP_MSG_TYPE >> %GTPV1_C2S_TEID_DATA %GTPV1_C2S_TEID_CTRL %GTPV1_S2C_TEID_DATA >> %GTPV1_S2C_TEID_CTRL %GTPV1_END_USER_IP %GTPV1_END_USER_IMSI >> %GTPV1_END_USER_MSISDN %GTPV1_END_USER_IMEI %GTPV1_APN_NAME %GTPV1_RAT_TYPE >> %GTPV1_RAI_MCC %GTPV1_RAI_MNC %GTPV1_RAI_LAC %GTPV1_RAI_RAC %GTPV1_ULI_MCC >> %GTPV1_ULI_MNC %GTPV1_ULI_CELL_LAC %GTPV1_ULI_CELL_CI %GTPV1_ULI_SAC >> %GTPV1_RESPONSE_CAUSE %SRC_FRAGMENTS %DST_FRAGMENTS %CLIENT_NW_LATENCY_MS >> %SERVER_NW_LATENCY_MS %APPL_LATENCY_MS %RETRANSMITTED_IN_BYTES >> %RETRANSMITTED_IN_PKTS %RETRANSMITTED_OUT_BYTES %RETRANSMITTED_OUT_PKTS >> %OOORDER_IN_PKTS %OOORDER_OUT_PKTS %FLOW_ACTIVE_TIMEOUT >> %FLOW_INACTIVE_TIMEOUT %MIN_TTL %MAX_TTL %IN_SRC_MAC %OUT_DST_MAC >> %PACKET_SECTION_OFFSET %FRAME_LENGTH %SRC_TO_DST_MAX_THROUGHPUT >> %SRC_TO_DST_MIN_THROUGHPUT %SRC_TO_DST_AVG_THROUGHPUT >> %DST_TO_SRC_MAX_THROUGHPUT %DST_TO_SRC_MIN_THROUGHPUT >> %DST_TO_SRC_AVG_THROUGHPUT %NUM_PKTS_UP_TO_128_BYTES >> %NUM_PKTS_128_TO_256_BYTES %NUM_PKTS_256_TO_512_BYTES >> %NUM_PKTS_512_TO_1024_BYTES %NUM_PKTS_1024_TO_1514_BYTES >> %NUM_PKTS_OVER_1514_BYTES %LONGEST_FLOW_PKT %SHORTEST_FLOW_PKT >> %NUM_PKTS_TTL_EQ_1 %NUM_PKTS_TTL_2_5 %NUM_PKTS_TTL_5_32 %NUM_PKTS_TTL_32_64 >> %NUM_PKTS_TTL_64_96 %NUM_PKTS_TTL_96_128 %NUM_PKTS_TTL_128_160 >> %NUM_PKTS_TTL_160_192 %NUM_PKTS_TTL_192_224 %NUM_PKTS_TTL_224_255 >> %DURATION_IN %DURATION_OUT %TCP_WIN_MIN_IN %TCP_WIN_MAX_IN %TCP_WIN_MSS_IN >> %TCP_WIN_SCALE_IN %TCP_WIN_MIN_OUT %TCP_WIN_MAX_OUT %TCP_WIN_MSS_OUT >> %TCP_WIN_SCALE_OUT" >> --flow-version=9 >> --tunnel >> --smart-udp-frags >> >> >> >> >> -- >> Regards, >> David Notivol >> [email protected] <mailto:[email protected]> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > _______________________________________________ > Ntop-misc mailing list > [email protected] <mailto:[email protected]> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > -- > Saludos, > David Notivol > [email protected] <mailto:[email protected]> > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
