Hi Alfredo, Thanks for your recommendations. I tested using core affinity as you suggested, and the in drops disappeared in zbalance. The output drops persist, but the absolute drops are less than before. Actually I had tested the core affinity, but I didn't have in mind the physical cores. Now I put zbalance in one physical core, and 10 nprobe instances not sharing the physical core with zbalance.
About your point 2, by using zc drivers, how could I run several nprobe instances to share the load? I'm testing with one instance: -i zc:p2p1,zc:p2p2 Attached you can find: - 0.log = top output for the scenario in my previous email. - 1.log = scenario in your point 1, including top, zbalance output, and nprobe stats. El mié., 27 jun. 2018 a las 12:13, Alfredo Cardigliano (< [email protected]>) escribió: > Hi David > it seems that you have packet loss both on zbalance and nprobe, > I recommend you to: > 1. set the core affinity for both zbalance_ipc and the nprobe instances, > trying to > use a different core for each (at least do not share the zbalance_ipc > physical core > with nprobe instances) > 2. did you try using zc drivers for capturing traffic from the interfaces? > (zc:p2p1,zc:p2p2) > Please also provide the top output (press 1 to see all cored) with the > current configuration, > I guess kernel is using some of the available cpu with this configuration. > > Alfredo > > On 26 Jun 2018, at 16:31, David Notivol <[email protected]> wrote: > > Hi Alfredo, > Thanks for replying. > This is an excerpt of the zbalance and nprobe statistics: > > 26/Jun/2018 17:29:58 [zbalance_ipc.c:265] ========================= > 26/Jun/2018 17:29:58 [zbalance_ipc.c:266] Absolute Stats: Recv > 1'285'430'239 pkts (1'116'181'903 drops) - Forwarded 1'266'272'285 pkts > (19'157'949 drops) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:305] p2p1,p2p2 RX > 1285430267 pkts Dropped 1116181981 pkts (46.5 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 0 RX 77050882 > pkts Dropped 1127883 pkts (1.4 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 1 RX 70722562 > pkts Dropped 756409 pkts (1.1 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 2 RX 76092418 > pkts Dropped 1017335 pkts (1.3 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 3 RX 75088386 > pkts Dropped 896678 pkts (1.2 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 4 RX 91991042 > pkts Dropped 2114739 pkts (2.2 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 5 RX 81384450 > pkts Dropped 1269385 pkts (1.5 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 6 RX 84310018 > pkts Dropped 1801848 pkts (2.1 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 7 RX 84554242 > pkts Dropped 1487329 pkts (1.7 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 8 RX 84090370 > pkts Dropped 1482864 pkts (1.7 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 9 RX 73642498 > pkts Dropped 732237 pkts (1.0 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 10 RX 76481026 > pkts Dropped 1000496 pkts (1.3 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 11 RX 72496642 > pkts Dropped 929049 pkts (1.3 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 12 RX 79386626 > pkts Dropped 1122169 pkts (1.4 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 13 RX 79418370 > pkts Dropped 1187172 pkts (1.5 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 14 RX 80284162 > pkts Dropped 1195559 pkts (1.5 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 15 RX 79143426 > pkts Dropped 1036797 pkts (1.3 %) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:338] Actual Stats: Recv 369'127.51 > pps (555'069.74 drops) - Forwarded 369'129.51 pps (0.00 drops) > 26/Jun/2018 17:29:58 [zbalance_ipc.c:348] ========================= > > > # cat /proc/net/pf_ring/stats/* > ClusterId: 1 > TotQueues: 16 > Applications: 1 > App0Queues: 16 > Duration: 0:00:41:18:386 > Packets: 1191477340 > Forwarded: 1174033613 > Processed: 1173893301 > IFPackets: 1191477364 > IFDropped: 1036448041 > > Duration: 0:00:41:15:587 > Bytes: 42626434538 > Packets: 71510530 > Dropped: 845465 > > Duration: 0:00:41:15:557 > Bytes: 40686677370 > Packets: 65656322 > Dropped: 533675 > > Duration: 0:00:41:15:534 > Bytes: 41463519299 > Packets: 70565378 > Dropped: 804282 > > Duration: 0:00:41:15:523 > Bytes: 42321923225 > Packets: 69566978 > Dropped: 650333 > > Duration: 0:00:41:14:659 > Bytes: 45415334638 > Packets: 85479938 > Dropped: 1728521 > > Duration: 0:00:41:14:597 > Bytes: 42615821825 > Packets: 75445250 > Dropped: 951386 > > Duration: 0:00:41:14:598 > Bytes: 44722410915 > Packets: 78252409 > Dropped: 1479387 > > Duration: 0:00:41:14:613 > Bytes: 44788855334 > Packets: 78318926 > Dropped: 1202905 > > Duration: 0:00:41:14:741 > Bytes: 43950263720 > Packets: 77821954 > Dropped: 1135693 > > Duration: 0:00:41:14:608 > Bytes: 41211162757 > Packets: 68241354 > Dropped: 496494 > > Duration: 0:00:41:14:629 > Bytes: 43064091353 > Packets: 70834104 > Dropped: 712427 > > Duration: 0:00:41:14:551 > Bytes: 42072869897 > Packets: 67360770 > Dropped: 696460 > > Duration: 0:00:41:14:625 > Bytes: 44323715294 > Packets: 73420290 > Dropped: 851818 > > Duration: 0:00:41:14:625 > Bytes: 43018671083 > Packets: 73651110 > Dropped: 917985 > > Duration: 0:00:41:14:600 > Bytes: 42730057210 > Packets: 74312500 > Dropped: 799922 > > Duration: 0:00:41:14:611 > Bytes: 42519248547 > Packets: 73394690 > Dropped: 771941 > > > > El mar., 26 jun. 2018 a las 16:25, Alfredo Cardigliano (< > [email protected]>) escribió: > >> Hi David >> please also provide statistics from zbalance_ipc (output or log file) >> and nprobe (you can get live stats from /proc/net/pf_ring/stats/) >> >> Thank you >> Alfredo >> >> On 26 Jun 2018, at 15:32, David Notivol <[email protected]> wrote: >> >> Hello list, >> >> We're using nProbe to export flows information to kafka. We're listening >> from two 10Gb interfaces that we merge with zbalance_ipc, and we split them >> into 16 queues to have 16 nprobe instances. >> >> The problem is we are seeing about 40% packet drops reported by >> zbalance_ipc, so it looks like nprobe is not capable of reading and >> processing all the traffic. The CPU usage is really high, and the load >> average is over 25-30. >> >> Merging both interfaces we're getting up to 5.5 Gbps, and 1.2 million >> packets / second; and we're using i40e_zc driver. >> >> Do you have any advice to try to improve this performance? >> Does it make sense we're having packet drops with this amount of traffic, >> and we're reaching the server limits? Or is any configuration we could tune >> up to improve it? >> >> Thanks in advance. >> >> >> >> -- System: >> >> nProbe: nProbe v.8.5.180625 (r6185) >> System RAM: 64GB >> System CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 12 cores (6 >> cores, 2 threads per core) >> System OS: CentOS Linux release 7.4.1708 (Core) >> Linux Kernel: 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC >> 2018 x86_64 x86_64 x86_64 GNU/Linux >> >> -- zbalance configuration: >> >> zbalance_ipc -i p2p1,p2p2 -c 1 -n 16 -m 4 -a -p -l /var/tmp/zbalance.log >> -v -w >> >> -- nProbe configuration: >> >> --interface=zc:1@0 >> --pid-file=/var/run/nprobe-zc1-00.pid >> --dump-stats=/var/log/nprobe/zc1-00_flows_stats.txt >> --kafka "192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092;topic" >> --collector=none >> --idle-timeout=60 >> --snaplen=128 >> --aggregation=0/1/1/1/0/0/0 >> --all-collectors=0 >> --verbose=1 >> --dump-format=t >> --vlanid-as-iface-idx=none >> --hash-size=1024000 >> --flow-delay=1 >> --count-delay=10 >> --min-flow-size=0 >> --netflow-engine=0:0 >> --sample-rate=1:1 >> --as-list=/usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat >> --city-list=/usr/share/ntopng/httpdocs/geoip/GeoLiteCity.dat >> --flow-templ="%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %IN_BYTES %OUT_PKTS >> %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT >> %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS %L7_PROTO %L7_PROTO_NAME >> %SRC_IP_COUNTRY %SRC_IP_CITY %SRC_IP_LONG %SRC_IP_LAT %DST_IP_COUNTRY >> %DST_IP_CITY %DST_IP_LONG %DST_IP_LAT %SRC_VLAN %DST_VLAN %DOT1Q_SRC_VLAN >> %DOT1Q_DST_VLAN %DIRECTION %SSL_SERVER_NAME %SRC_AS_MAP %DST_AS_MAP >> %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST >> %HTTP_SITE %UPSTREAM_TUNNEL_ID %UPSTREAM_SESSION_ID %DOWNSTREAM_TUNNEL_ID >> %DOWNSTREAM_SESSION_ID %UNTUNNELED_PROTOCOL %UNTUNNELED_IPV4_SRC_ADDR >> %UNTUNNELED_L4_SRC_PORT %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_L4_DST_PORT >> %GTPV2_REQ_MSG_TYPE %GTPV2_RSP_MSG_TYPE %GTPV2_C2S_S1U_GTPU_TEID >> %GTPV2_C2S_S1U_GTPU_IP %GTPV2_S2C_S1U_GTPU_TEID %GTPV2_S5_S8_GTPC_TEID >> %GTPV2_S2C_S1U_GTPU_IP %GTPV2_C2S_S5_S8_GTPU_TEID >> %GTPV2_S2C_S5_S8_GTPU_TEID %GTPV2_C2S_S5_S8_GTPU_IP >> %GTPV2_S2C_S5_S8_GTPU_IP %GTPV2_END_USER_IMSI %GTPV2_END_USER_MSISDN >> %GTPV2_APN_NAME %GTPV2_ULI_MCC %GTPV2_ULI_MNC %GTPV2_ULI_CELL_TAC >> %GTPV2_ULI_CELL_ID %GTPV2_RESPONSE_CAUSE %GTPV2_RAT_TYPE %GTPV2_PDN_IP >> %GTPV2_END_USER_IMEI %GTPV2_C2S_S5_S8_GTPC_IP %GTPV2_S2C_S5_S8_GTPC_IP >> %GTPV2_C2S_S5_S8_SGW_GTPU_TEID %GTPV2_S2C_S5_S8_SGW_GTPU_TEID >> %GTPV2_C2S_S5_S8_SGW_GTPU_IP %GTPV2_S2C_S5_S8_SGW_GTPU_IP >> %GTPV1_REQ_MSG_TYPE %GTPV1_RSP_MSG_TYPE %GTPV1_C2S_TEID_DATA >> %GTPV1_C2S_TEID_CTRL %GTPV1_S2C_TEID_DATA %GTPV1_S2C_TEID_CTRL >> %GTPV1_END_USER_IP %GTPV1_END_USER_IMSI %GTPV1_END_USER_MSISDN >> %GTPV1_END_USER_IMEI %GTPV1_APN_NAME %GTPV1_RAT_TYPE %GTPV1_RAI_MCC >> %GTPV1_RAI_MNC %GTPV1_RAI_LAC %GTPV1_RAI_RAC %GTPV1_ULI_MCC %GTPV1_ULI_MNC >> %GTPV1_ULI_CELL_LAC %GTPV1_ULI_CELL_CI %GTPV1_ULI_SAC %GTPV1_RESPONSE_CAUSE >> %SRC_FRAGMENTS %DST_FRAGMENTS %CLIENT_NW_LATENCY_MS %SERVER_NW_LATENCY_MS >> %APPL_LATENCY_MS %RETRANSMITTED_IN_BYTES %RETRANSMITTED_IN_PKTS >> %RETRANSMITTED_OUT_BYTES %RETRANSMITTED_OUT_PKTS %OOORDER_IN_PKTS >> %OOORDER_OUT_PKTS %FLOW_ACTIVE_TIMEOUT %FLOW_INACTIVE_TIMEOUT %MIN_TTL >> %MAX_TTL %IN_SRC_MAC %OUT_DST_MAC %PACKET_SECTION_OFFSET %FRAME_LENGTH >> %SRC_TO_DST_MAX_THROUGHPUT %SRC_TO_DST_MIN_THROUGHPUT >> %SRC_TO_DST_AVG_THROUGHPUT %DST_TO_SRC_MAX_THROUGHPUT >> %DST_TO_SRC_MIN_THROUGHPUT %DST_TO_SRC_AVG_THROUGHPUT >> %NUM_PKTS_UP_TO_128_BYTES %NUM_PKTS_128_TO_256_BYTES >> %NUM_PKTS_256_TO_512_BYTES %NUM_PKTS_512_TO_1024_BYTES >> %NUM_PKTS_1024_TO_1514_BYTES %NUM_PKTS_OVER_1514_BYTES %LONGEST_FLOW_PKT >> %SHORTEST_FLOW_PKT %NUM_PKTS_TTL_EQ_1 %NUM_PKTS_TTL_2_5 %NUM_PKTS_TTL_5_32 >> %NUM_PKTS_TTL_32_64 %NUM_PKTS_TTL_64_96 %NUM_PKTS_TTL_96_128 >> %NUM_PKTS_TTL_128_160 %NUM_PKTS_TTL_160_192 %NUM_PKTS_TTL_192_224 >> %NUM_PKTS_TTL_224_255 %DURATION_IN %DURATION_OUT %TCP_WIN_MIN_IN >> %TCP_WIN_MAX_IN %TCP_WIN_MSS_IN %TCP_WIN_SCALE_IN %TCP_WIN_MIN_OUT >> %TCP_WIN_MAX_OUT %TCP_WIN_MSS_OUT %TCP_WIN_SCALE_OUT" >> --flow-version=9 >> --tunnel >> --smart-udp-frags >> >> >> >> >> -- >> Regards, >> David Notivol >> [email protected] >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc > > > > -- > Saludos, > David Notivol > [email protected] > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc -- Saludos, David Notivol [email protected]
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
