Ok,
after further testing, the problem seems to be with nprobe.
Basically, I'm collecting netflow data from various routers (v5 and v9)
and send these to the collector, which is a licensed nprobe.
The netflow data could be collected from a Cisco router, it could come
from a Linux box running pmacctd or it could come from a Linux box
running nprobe.
nprobe.conf example for one of those boxes that collect netflows and
send them on as for example netflow v9:
-i=bond0
-g=/var/run/nprobe-zmq.pid
-n=XXX.XXX.XXX.XXX:2055
-V=9
-T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP
%OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED
%L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS
%IPV4_SRC_MASK %IPV4_DST_MASK %IPV6_SRC_ADDR %IPV6_DST_ADDR
%IPV6_NEXT_HOP %IPV6_SRC_MASK %IPV6_DST_MASK %IP_PROTOCOL_VERSION
%EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS %FLOW_ID %FLOW_START_SEC
%FLOW_END_SEC %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS
%BIFLOW_DIRECTION"
My collector nprobe is configured like this:
-i none
-n none
-3 2055
--zmq tcp://127.0.0.1:1234
-V 9
-T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP
%OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED
%L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS
%IPV4_SRC_MASK %IPV4_DST_MASK %IPV6_SRC_ADDR %IPV6_DST_ADDR
%IPV6_NEXT_HOP %IPV6_SRC_MASK %IPV6_DST_MASK %IP_PROTOCOL_VERSION
%EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS %FLOW_ID %FLOW_START_SEC
%FLOW_END_SEC %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS
%BIFLOW_DIRECTION"
And this then hands the data off to ntopng, which is configured like this:
-G=/var/run/ntopng.pid
-m=XXXX/X
-i=tcp://127.0.0.1:1234
-x=400000
-X=800000
Flows are present on startup, after the redis-database is cleared but
then stop coming in and flows view empties after a short while and stays
empty.
When I downgrade to v2.4 this setup works perfectly and I have flows.
With the builds of v2.5 from November, this also worked perfectly.
I spend the evening to strip it all down and where it fails is at the
point where the individual netflows are collected by the collector on
port 2055.
If i for example take the host collecting netflows on interface bond0
above and send the netflows using zmq directly to ntopng, then I have
flows .. and it works perfectly.
When I collect flows via the Netflows/sFlow/IPFix port, it's broken.
I hope this helps to pinpoint the problem.
Tested tonight with:
nprobe 7.5.170112-5587
pfring 6.5.0-1094
pfring-dkms 6.5.0
ntopng 2.5.170112-2154
ntopng-data 2.5.170112
Kind regards,
Martin List-Petersen
On 11/01/17 22:36, Martin List-Petersen wrote:
Hi,
upgraded to 2.5.170111 after a ntopng 2.5.170108 failing with
segmentation faults a couple of times.
The issue with flows only being shown after a restart, then disappearing
remains.
When I downgrade to 2.4, flows work perfectly.
The same configuration in both cases.
Kind regards,
Martin List-Petersen
Airwire Ltd.
On 09/01/17 10:07, Martin List-Petersen wrote:
Hi,
ntopng.conf only contains:
-G=/var/run/ntopng.pid
nothing more.
nprobe output:
09/Jan/2017 10:01:03 [nprobe.c:3492] Valid nProbe license found
09/Jan/2017 10:01:03 [nprobe.c:5201] WARNING: The output interfaceId is
set to 0: did you forget to use -Q perhaps ?
09/Jan/2017 10:01:03 [nprobe.c:5204] WARNING: The input interfaceId is
set to 0: did you forget to use -u perhaps ?
09/Jan/2017 10:01:03 [nprobe.c:5304] Welcome to nProbe v.7.5.170108
($Revision: 5578 $) for x86_64-unknown-linux-gnu with native PF_RING
acceleration
09/Jan/2017 10:01:03 [nprobe.c:5314] Running on Debian GNU/Linux 8.2
(jessie)
09/Jan/2017 10:01:03 [nprobe.c:5325] [LICENSE] nProbe SystemId:
1B71ED8609B0B927
09/Jan/2017 10:01:03 [nprobe.c:7680] Welcome to nProbe v.7.5.170108 for
x86_64-unknown-linux-gnu
09/Jan/2017 10:01:03 [nprobe.c:6757] WARNING: You selected v9/IPFIX
without specifying a template (-T).
09/Jan/2017 10:01:03 [nprobe.c:6758] WARNING: The default template will
be used
09/Jan/2017 10:01:03 [nprobe.c:6763] Using NetFlow Packet Payload Len:
1472
09/Jan/2017 10:01:03 [plugin.c:1078] 0 plugin(s) enabled
09/Jan/2017 10:01:03 [nprobe.c:7176] Each flow is 89 bytes long
09/Jan/2017 10:01:03 [nprobe.c:7177] The # packets per flow has been set
to 15
09/Jan/2017 10:01:03 [nprobe.c:7180] IP TOS is accounted
09/Jan/2017 10:01:03 [nprobe.c:7206] Non IPv4/v6 traffic is discarded
according to the template
09/Jan/2017 10:01:03 [util.c:430] GeoIP: loaded AS config file
/usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
09/Jan/2017 10:01:03 [util.c:441] GeoIP: loaded AS IPv6 config file
/usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
09/Jan/2017 10:01:03 [nprobe.c:8005] Not capturing packet from interface
(collector mode)
09/Jan/2017 10:01:03 [util.c:4043] Initializing ZMQ as server
09/Jan/2017 10:01:03 [util.c:4086] Succesfully created ZMQ endpoint
tcp://127.0.0.1:1234
09/Jan/2017 10:01:03 [collect.c:143] Flow collector listening on port
2055 (IPv4/v6)
09/Jan/2017 10:01:03 [nprobe.c:8230] nProbe started successfully
ntopng output:
09/Jan/2017 10:04:22 [Redis.cpp:108] Successfully connected to redis
127.0.0.1:6379@0
09/Jan/2017 10:04:22 [NtopPro.cpp:118] [LICENSE] Read license from Redis
[XXXXX]
09/Jan/2017 10:04:22 [Ntop.cpp:1236] Registered interface
tcp://127.0.0.1:1234 [id: 0]
09/Jan/2017 10:04:22 [main.cpp:248] PID stored in file
/var/run/ntopng.pid
09/Jan/2017 10:04:22 [HTTPserver.cpp:507] Please read
https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to
enable SSL.
09/Jan/2017 10:04:22 [Utils.cpp:367] User changed to nobody
09/Jan/2017 10:04:22 [HTTPserver.cpp:552] Web server dirs
[/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
09/Jan/2017 10:04:22 [HTTPserver.cpp:555] HTTP server listening on port
3000
09/Jan/2017 10:04:22 [main.cpp:309] Working directory: /var/tmp/ntopng
09/Jan/2017 10:04:22 [main.cpp:311] Scripts/HTML pages directory:
/usr/share/ntopng
09/Jan/2017 10:04:22 [Ntop.cpp:268] Welcome to ntopng x86_64
v.2.5.170108 - (C) 1998-2016 ntop.org
09/Jan/2017 10:04:22 [Ntop.cpp:273] Built on Debian GNU/Linux 8.2
(jessie)
09/Jan/2017 10:04:22 [PeriodicActivities.cpp:55] Started periodic
activities loop...
09/Jan/2017 10:04:22 [NtopPro.cpp:262] [LICENSE] ntopng systemId:
1B71ED8609B0B927
09/Jan/2017 10:04:22 [NtopPro.cpp:273] [LICENSE] ntopng license:
F94DBEB4F844679D6B490B2830E3072715076388282F622A26
09/Jan/2017 10:04:22 [NtopPro.cpp:294] [LICENSE] Maintenance is
available until Tue Oct 10 13:33:48 2017 [274 days left]
09/Jan/2017 10:04:22 [Ntop.cpp:559] Local Interface Addresses (System
Host)
09/Jan/2017 10:04:22 [Ntop.cpp:561] Local Networks
09/Jan/2017 10:04:22 [AddressTree.cpp:134] [AddressTree] XXXXXXX
09/Jan/2017 10:04:22 [NetworkInterface.cpp:1797] Started packet polling
on interface tcp://127.0.0.1:1234 [id: 0]...
09/Jan/2017 10:04:23 [CollectorInterface.cpp:115] Collecting flows on
tcp://127.0.0.1:1234
And as I said, my configuration has not changed. I've upgraded from the
November build to the January build. That's the only difference.
This was working perfectly up until then.
Kind regards,
Martin List-Petersen
Airwire Ltd.
On 09/01/17 09:10, Simone Mainardi wrote:
Martin,
On Sun, Jan 8, 2017 at 5:32 PM, Martin List-Petersen <[email protected]>
wrote:
After upgrading to 2.5.170108-2130 I have no flows in the flows view
.. at
all.
I have tried to downgrade to 2.5.170106 as I had a copy of that lying
around on a host, that listens on a different network without the
use of
nprobe and it has flows.
But when used together with nprobe even that version has no flows.
I then downgraded to 2.4 stable and I have flows again, as I previously
had with the 2.5 releases from November 2016.
This what I installed today:
ntopng 2.5.170108-2130
ntopng-data 2.5.170108
nprobe 7.5.170108-5578
pfring 6.5.0-1089
pfring-dkms 6.5.0
nprobe is started like this:
nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 -V 9
ntopng is started like this:
ntopng /etc/ntopng/ntopng.conf --local-networks xxxx -i tcp://
127.0.0.1:1234 -x 400000 -X 800000
What are the contents of /etc/ntopng/ntopng.conf? Please don't mix a
configuration file with command line arguments.
Post both the ntopng and the nprobe output. You may also want to run
ntopng
and nProbe with verbose/debug flags to see the path of the flows. That
is,
if they correctly reach the nProbe and if they are correctly sent to the
ntopng.
nprobe gets SNMP from 6 routers with a total stream of about 1.4-1.7
Gbit/s data flow, so it's not the lack of flows, that's the problem.
And
the configuration nor the startup parameters have been changed from
before
the upgrade, where I had flows.
Kind regards,
Martin List-Petersen
--
Airwire Ltd. - Ag Nascadh Pobail an Iarthair
http://www.airwire.ie
Phone: 091-865 968
Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 -
Registered in
Ireland No. 508961
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
--
Airwire Ltd. - Ag Nascadh Pobail an Iarthair
http://www.airwire.ie
Phone: 091-865 968
Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 - Registered in
Ireland No. 508961
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
