Thanks Alfredo, all is working as expected now!
________________________________ From: [email protected] <[email protected]> on behalf of Alfredo Cardigliano <[email protected]> Sent: Thursday, November 17, 2016 5:49 AM To: [email protected] Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 2.6.32-642.4.2.el6.x86_64 Exactly, you need a configuration line per CPU node node=0 hugepagenumber=1024 node=1 hugepagenumber=1024 Alfredo On 16 Nov 2016, at 20:30, Spransy, Derek <[email protected]<mailto:[email protected]>> wrote: Hi Alfredo, Here's the output: Node 0 HugePages_Total: 1024 Node 0 HugePages_Free: 1024 Node 0 HugePages_Surp: 0 Node 1 HugePages_Total: 0 Node 1 HugePages_Free: 0 Node 1 HugePages_Surp: 0 Our hugepages.conf consists of: node=0 hugepagenumber=1024 Do you think this needs to be added for node 1 as well? Thanks, Derek ________________________________ From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> on behalf of Alfredo Cardigliano <[email protected]<mailto:[email protected]>> Sent: Tuesday, November 15, 2016 9:01 AM To: [email protected]<mailto:[email protected]> Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 2.6.32-642.4.2.el6.x86_64 Hi Derek this seems to be related to hugepages, could you provide more info about memory in your system including: cat /sys/devices/system/node/node*/meminfo | grep Huge Alfredo On 14 Nov 2016, at 15:13, Spransy, Derek <[email protected]<mailto:[email protected]>> wrote: Hi Alfredo, Yes the behavior is the same as before: $ sudo /usr/local/bin/n2disk10g /etc/n2disk/n2disk-eth5.conf 14/Nov/2016 09:04:21 [n2disk.c:4819] Welcome to n2disk10g v.2.6.161111 (r4672) [SandyBridge] 14/Nov/2016 09:04:21 [n2disk.c:4846] Running on 2 node(s) system with 24 core(s). NUMA affinity set to node 1. 14/Nov/2016 09:04:21 [n2disk.c:4875] Using PF_RING for packet capture 14/Nov/2016 09:04:21 [n2disk.c:4901] WARNING: If you are using standard drivers (packet capture via kernel) please disable time-pulse thread 14/Nov/2016 09:04:21 [n2disk.c:4904] Multithread support enabled 14/Nov/2016 09:04:21 [n2disk.c:5018] Dump files max size is set to 1024 MB 14/Nov/2016 09:04:21 [n2disk.c:5035] Buffer memory is set to 3 GB (x 3 pcap files) 14/Nov/2016 09:04:21 [n2disk.c:5070] Using directory /data1/captures for dump files 14/Nov/2016 09:04:21 [n2disk.c:5075] No sub-directories will be created 14/Nov/2016 09:04:21 [n2disk.c:5080] Up to 43000 files will be written before overwriting 14/Nov/2016 09:04:21 [n2disk.c:5090] Dump files max duration is set to 600 sec 14/Nov/2016 09:04:21 [n2disk.c:5106] Dumping data in 0.1 MB chunks 14/Nov/2016 09:04:21 [n2disk.c:5149] Index processing memory is set to 847 MB (x 3 index files) 14/Nov/2016 09:04:24 [n2disk.c:5339] Memory allocated successfully 14/Nov/2016 09:04:24 [n2disk.c:3597] Using time pulse timestamps 14/Nov/2016 09:04:24 [n2disk.c:3630] Started PF_RING packet reader thread for device zc:eth5 $ Running zbalance works fine: $ sudo /usr/local/bin/zbalance_ipc -i zc:eth5 -c 99 -n 1 -m 1 14/Nov/2016 09:01:24 [zbalance_ipc.c:686] Starting balancer with 1 consumer queues.. 14/Nov/2016 09:01:24 [zbalance_ipc.c:696] Run your application instances as follows: 14/Nov/2016 09:01:24 [zbalance_ipc.c:701] pfcount -i zc:99@0 14/Nov/2016 09:01:25 [zbalance_ipc.c:151] ========================= 14/Nov/2016 09:01:25 [zbalance_ipc.c:152] Absolute Stats: Recv 354'797 pkts (0 drops) - Forwarded 8'190 pkts (346'607 drops) 14/Nov/2016 09:01:25 [zbalance_ipc.c:234] ========================= 14/Nov/2016 09:01:26 [zbalance_ipc.c:151] ========================= 14/Nov/2016 09:01:26 [zbalance_ipc.c:152] Absolute Stats: Recv 714'213 pkts (0 drops) - Forwarded 8'190 pkts (706'023 drops) 14/Nov/2016 09:01:26 [zbalance_ipc.c:224] Actual Stats: Recv 359'348.08 pps (0.00 drops) - Forwarded 0.00 pps (359'348.08 drops) 14/Nov/2016 09:01:26 [zbalance_ipc.c:234] ========================= 14/Nov/2016 09:01:27 [zbalance_ipc.c:151] ========================= 14/Nov/2016 09:01:27 [zbalance_ipc.c:152] Absolute Stats: Recv 1'053'124 pkts (0 drops) - Forwarded 8'190 pkts (1'044'926 drops) 14/Nov/2016 09:01:27 [zbalance_ipc.c:224] Actual Stats: Recv 338'839.84 pps (0.00 drops) - Forwarded 0.00 pps (338'831.84 drops) 14/Nov/2016 09:01:27 [zbalance_ipc.c:234] ========================= 14/Nov/2016 09:01:28 [zbalance_ipc.c:151] ========================= 14/Nov/2016 09:01:28 [zbalance_ipc.c:152] Absolute Stats: Recv 1'410'053 pkts (0 drops) - Forwarded 8'190 pkts (1'401'863 drops) 14/Nov/2016 09:01:28 [zbalance_ipc.c:224] Actual Stats: Recv 356'860.48 pps (0.00 drops) - Forwarded 0.00 pps (356'868.48 drops) 14/Nov/2016 09:01:28 [zbalance_ipc.c:234] ========================= 14/Nov/2016 09:01:29 [zbalance_ipc.c:151] ========================= 14/Nov/2016 09:01:29 [zbalance_ipc.c:152] Absolute Stats: Recv 1'782'541 pkts (0 drops) - Forwarded 8'190 pkts (1'774'351 drops) 14/Nov/2016 09:01:29 [zbalance_ipc.c:224] Actual Stats: Recv 372'419.84 pps (0.00 drops) - Forwarded 0.00 pps (372'419.84 drops) 14/Nov/2016 09:01:29 [zbalance_ipc.c:234] ========================= Here's strace output of the crash, if that helps any: write(1, "14/Nov/2016 09:06:12 [n2disk.c:3"..., 9314/Nov/2016 09:06:12 [n2disk.c:3630] Started PF_RING packet reader thread for device zc:eth5 ) = 93 stat("/sys/devices/system/node/node0/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 stat("/sys/devices/system/node/node1/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 stat("/sys/devices/system/node/node2/cpumap", 0x7ffc10d5fa60) = -1 ENOENT (No such file or directory) stat("/sys/devices/system/node/node0/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 stat("/sys/devices/system/node/node1/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 stat("/sys/devices/system/node/node2/cpumap", 0x7ffc10d5fa40) = -1 ENOENT (No such file or directory) set_mempolicy(MPOL_BIND, 0x2d3b3b0, 4) = 0 socket(0x1b /* PF_??? */, SOCK_RAW, 768) = 5 setsockopt(5, SOL_IP, 0x87 /* IP_??? */, "\0\0\0\0\0\0\0\0", 8) = 0 setsockopt(5, SOL_IP, 0x6e /* IP_??? */, "pfring-zc-cluster-0", 19) = 0 open("/proc/meminfo", O_RDONLY) = 6 fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faed3cf3000 read(6, "MemTotal: 16291360 kB\nMemF"..., 1024) = 1024 read(6, "e: 1024\nHugePages_Rsvd: "..., 1024) = 174 close(6) = 0 munmap(0x7faed3cf3000, 4096) = 0 open("/proc/mounts", O_RDONLY) = 6 fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faed3cf3000 read(6, "rootfs / rootfs rw 0 0\nproc /pro"..., 1024) = 1024 read(6, "ugepages hugetlbfs rw,relatime 0"..., 1024) = 35 open("/mnt/hugepages/pfring_zc_0", O_RDWR|O_CREAT, 0755) = 7 flock(7, LOCK_SH) = 0 mmap(NULL, 134217728, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_LOCKED, 7, 0) = 0x7fb033200000 --- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7fb033200018} --- +++ killed by SIGBUS (core dumped) +++ Thanks, Derek ________________________________ This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). _______________________________________________ Ntop-misc mailing list [email protected]<mailto:[email protected]> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
