Hi Alfredo, Thanks for replying. This is an excerpt of the zbalance and nprobe statistics:
26/Jun/2018 17:29:58 [zbalance_ipc.c:265] ========================= 26/Jun/2018 17:29:58 [zbalance_ipc.c:266] Absolute Stats: Recv 1'285'430'239 pkts (1'116'181'903 drops) - Forwarded 1'266'272'285 pkts (19'157'949 drops) 26/Jun/2018 17:29:58 [zbalance_ipc.c:305] p2p1,p2p2 RX 1285430267 pkts Dropped 1116181981 pkts (46.5 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 0 RX 77050882 pkts Dropped 1127883 pkts (1.4 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 1 RX 70722562 pkts Dropped 756409 pkts (1.1 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 2 RX 76092418 pkts Dropped 1017335 pkts (1.3 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 3 RX 75088386 pkts Dropped 896678 pkts (1.2 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 4 RX 91991042 pkts Dropped 2114739 pkts (2.2 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 5 RX 81384450 pkts Dropped 1269385 pkts (1.5 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 6 RX 84310018 pkts Dropped 1801848 pkts (2.1 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 7 RX 84554242 pkts Dropped 1487329 pkts (1.7 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 8 RX 84090370 pkts Dropped 1482864 pkts (1.7 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 9 RX 73642498 pkts Dropped 732237 pkts (1.0 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 10 RX 76481026 pkts Dropped 1000496 pkts (1.3 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 11 RX 72496642 pkts Dropped 929049 pkts (1.3 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 12 RX 79386626 pkts Dropped 1122169 pkts (1.4 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 13 RX 79418370 pkts Dropped 1187172 pkts (1.5 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 14 RX 80284162 pkts Dropped 1195559 pkts (1.5 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 15 RX 79143426 pkts Dropped 1036797 pkts (1.3 %) 26/Jun/2018 17:29:58 [zbalance_ipc.c:338] Actual Stats: Recv 369'127.51 pps (555'069.74 drops) - Forwarded 369'129.51 pps (0.00 drops) 26/Jun/2018 17:29:58 [zbalance_ipc.c:348] ========================= # cat /proc/net/pf_ring/stats/* ClusterId: 1 TotQueues: 16 Applications: 1 App0Queues: 16 Duration: 0:00:41:18:386 Packets: 1191477340 Forwarded: 1174033613 Processed: 1173893301 IFPackets: 1191477364 IFDropped: 1036448041 Duration: 0:00:41:15:587 Bytes: 42626434538 Packets: 71510530 Dropped: 845465 Duration: 0:00:41:15:557 Bytes: 40686677370 Packets: 65656322 Dropped: 533675 Duration: 0:00:41:15:534 Bytes: 41463519299 Packets: 70565378 Dropped: 804282 Duration: 0:00:41:15:523 Bytes: 42321923225 Packets: 69566978 Dropped: 650333 Duration: 0:00:41:14:659 Bytes: 45415334638 Packets: 85479938 Dropped: 1728521 Duration: 0:00:41:14:597 Bytes: 42615821825 Packets: 75445250 Dropped: 951386 Duration: 0:00:41:14:598 Bytes: 44722410915 Packets: 78252409 Dropped: 1479387 Duration: 0:00:41:14:613 Bytes: 44788855334 Packets: 78318926 Dropped: 1202905 Duration: 0:00:41:14:741 Bytes: 43950263720 Packets: 77821954 Dropped: 1135693 Duration: 0:00:41:14:608 Bytes: 41211162757 Packets: 68241354 Dropped: 496494 Duration: 0:00:41:14:629 Bytes: 43064091353 Packets: 70834104 Dropped: 712427 Duration: 0:00:41:14:551 Bytes: 42072869897 Packets: 67360770 Dropped: 696460 Duration: 0:00:41:14:625 Bytes: 44323715294 Packets: 73420290 Dropped: 851818 Duration: 0:00:41:14:625 Bytes: 43018671083 Packets: 73651110 Dropped: 917985 Duration: 0:00:41:14:600 Bytes: 42730057210 Packets: 74312500 Dropped: 799922 Duration: 0:00:41:14:611 Bytes: 42519248547 Packets: 73394690 Dropped: 771941 El mar., 26 jun. 2018 a las 16:25, Alfredo Cardigliano (< [email protected]>) escribió: > Hi David > please also provide statistics from zbalance_ipc (output or log file) > and nprobe (you can get live stats from /proc/net/pf_ring/stats/) > > Thank you > Alfredo > > On 26 Jun 2018, at 15:32, David Notivol <[email protected]> wrote: > > Hello list, > > We're using nProbe to export flows information to kafka. We're listening > from two 10Gb interfaces that we merge with zbalance_ipc, and we split them > into 16 queues to have 16 nprobe instances. > > The problem is we are seeing about 40% packet drops reported by > zbalance_ipc, so it looks like nprobe is not capable of reading and > processing all the traffic. The CPU usage is really high, and the load > average is over 25-30. > > Merging both interfaces we're getting up to 5.5 Gbps, and 1.2 million > packets / second; and we're using i40e_zc driver. > > Do you have any advice to try to improve this performance? > Does it make sense we're having packet drops with this amount of traffic, > and we're reaching the server limits? Or is any configuration we could tune > up to improve it? > > Thanks in advance. > > > > -- System: > > nProbe: nProbe v.8.5.180625 (r6185) > System RAM: 64GB > System CPU: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 12 cores (6 > cores, 2 threads per core) > System OS: CentOS Linux release 7.4.1708 (Core) > Linux Kernel: 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC > 2018 x86_64 x86_64 x86_64 GNU/Linux > > -- zbalance configuration: > > zbalance_ipc -i p2p1,p2p2 -c 1 -n 16 -m 4 -a -p -l /var/tmp/zbalance.log > -v -w > > -- nProbe configuration: > > --interface=zc:1@0 > --pid-file=/var/run/nprobe-zc1-00.pid > --dump-stats=/var/log/nprobe/zc1-00_flows_stats.txt > --kafka "192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092;topic" > --collector=none > --idle-timeout=60 > --snaplen=128 > --aggregation=0/1/1/1/0/0/0 > --all-collectors=0 > --verbose=1 > --dump-format=t > --vlanid-as-iface-idx=none > --hash-size=1024000 > --flow-delay=1 > --count-delay=10 > --min-flow-size=0 > --netflow-engine=0:0 > --sample-rate=1:1 > --as-list=/usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat > --city-list=/usr/share/ntopng/httpdocs/geoip/GeoLiteCity.dat > --flow-templ="%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %IN_BYTES %OUT_PKTS > %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT > %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS %L7_PROTO %L7_PROTO_NAME > %SRC_IP_COUNTRY %SRC_IP_CITY %SRC_IP_LONG %SRC_IP_LAT %DST_IP_COUNTRY > %DST_IP_CITY %DST_IP_LONG %DST_IP_LAT %SRC_VLAN %DST_VLAN %DOT1Q_SRC_VLAN > %DOT1Q_DST_VLAN %DIRECTION %SSL_SERVER_NAME %SRC_AS_MAP %DST_AS_MAP > %HTTP_METHOD %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST > %HTTP_SITE %UPSTREAM_TUNNEL_ID %UPSTREAM_SESSION_ID %DOWNSTREAM_TUNNEL_ID > %DOWNSTREAM_SESSION_ID %UNTUNNELED_PROTOCOL %UNTUNNELED_IPV4_SRC_ADDR > %UNTUNNELED_L4_SRC_PORT %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_L4_DST_PORT > %GTPV2_REQ_MSG_TYPE %GTPV2_RSP_MSG_TYPE %GTPV2_C2S_S1U_GTPU_TEID > %GTPV2_C2S_S1U_GTPU_IP %GTPV2_S2C_S1U_GTPU_TEID %GTPV2_S5_S8_GTPC_TEID > %GTPV2_S2C_S1U_GTPU_IP %GTPV2_C2S_S5_S8_GTPU_TEID > %GTPV2_S2C_S5_S8_GTPU_TEID %GTPV2_C2S_S5_S8_GTPU_IP > %GTPV2_S2C_S5_S8_GTPU_IP %GTPV2_END_USER_IMSI %GTPV2_END_USER_MSISDN > %GTPV2_APN_NAME %GTPV2_ULI_MCC %GTPV2_ULI_MNC %GTPV2_ULI_CELL_TAC > %GTPV2_ULI_CELL_ID %GTPV2_RESPONSE_CAUSE %GTPV2_RAT_TYPE %GTPV2_PDN_IP > %GTPV2_END_USER_IMEI %GTPV2_C2S_S5_S8_GTPC_IP %GTPV2_S2C_S5_S8_GTPC_IP > %GTPV2_C2S_S5_S8_SGW_GTPU_TEID %GTPV2_S2C_S5_S8_SGW_GTPU_TEID > %GTPV2_C2S_S5_S8_SGW_GTPU_IP %GTPV2_S2C_S5_S8_SGW_GTPU_IP > %GTPV1_REQ_MSG_TYPE %GTPV1_RSP_MSG_TYPE %GTPV1_C2S_TEID_DATA > %GTPV1_C2S_TEID_CTRL %GTPV1_S2C_TEID_DATA %GTPV1_S2C_TEID_CTRL > %GTPV1_END_USER_IP %GTPV1_END_USER_IMSI %GTPV1_END_USER_MSISDN > %GTPV1_END_USER_IMEI %GTPV1_APN_NAME %GTPV1_RAT_TYPE %GTPV1_RAI_MCC > %GTPV1_RAI_MNC %GTPV1_RAI_LAC %GTPV1_RAI_RAC %GTPV1_ULI_MCC %GTPV1_ULI_MNC > %GTPV1_ULI_CELL_LAC %GTPV1_ULI_CELL_CI %GTPV1_ULI_SAC %GTPV1_RESPONSE_CAUSE > %SRC_FRAGMENTS %DST_FRAGMENTS %CLIENT_NW_LATENCY_MS %SERVER_NW_LATENCY_MS > %APPL_LATENCY_MS %RETRANSMITTED_IN_BYTES %RETRANSMITTED_IN_PKTS > %RETRANSMITTED_OUT_BYTES %RETRANSMITTED_OUT_PKTS %OOORDER_IN_PKTS > %OOORDER_OUT_PKTS %FLOW_ACTIVE_TIMEOUT %FLOW_INACTIVE_TIMEOUT %MIN_TTL > %MAX_TTL %IN_SRC_MAC %OUT_DST_MAC %PACKET_SECTION_OFFSET %FRAME_LENGTH > %SRC_TO_DST_MAX_THROUGHPUT %SRC_TO_DST_MIN_THROUGHPUT > %SRC_TO_DST_AVG_THROUGHPUT %DST_TO_SRC_MAX_THROUGHPUT > %DST_TO_SRC_MIN_THROUGHPUT %DST_TO_SRC_AVG_THROUGHPUT > %NUM_PKTS_UP_TO_128_BYTES %NUM_PKTS_128_TO_256_BYTES > %NUM_PKTS_256_TO_512_BYTES %NUM_PKTS_512_TO_1024_BYTES > %NUM_PKTS_1024_TO_1514_BYTES %NUM_PKTS_OVER_1514_BYTES %LONGEST_FLOW_PKT > %SHORTEST_FLOW_PKT %NUM_PKTS_TTL_EQ_1 %NUM_PKTS_TTL_2_5 %NUM_PKTS_TTL_5_32 > %NUM_PKTS_TTL_32_64 %NUM_PKTS_TTL_64_96 %NUM_PKTS_TTL_96_128 > %NUM_PKTS_TTL_128_160 %NUM_PKTS_TTL_160_192 %NUM_PKTS_TTL_192_224 > %NUM_PKTS_TTL_224_255 %DURATION_IN %DURATION_OUT %TCP_WIN_MIN_IN > %TCP_WIN_MAX_IN %TCP_WIN_MSS_IN %TCP_WIN_SCALE_IN %TCP_WIN_MIN_OUT > %TCP_WIN_MAX_OUT %TCP_WIN_MSS_OUT %TCP_WIN_SCALE_OUT" > --flow-version=9 > --tunnel > --smart-udp-frags > > > > > -- > Regards, > David Notivol > [email protected] > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc -- Saludos, David Notivol [email protected]
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
