Hey, I just realize it worked. I think I was little behind on the messages
parking on my kafka, now I can see the tag.
Thank you so much for your help.
On Wed, Apr 15, 2020 at 10:33 AM Emanuel dos Reis Rodrigues <
emanueldosr...@gmail.com> wrote:
> I am using:
>
> NetFlow Accounting Daemon, nfacctd 1.7.2-git (20181018-00+c3)
>
> Arguments:
> '--enable-kafka' '--enable-jansson'
> 'JANSSON_CFLAGS=-I/usr/local/include/' 'JANSSON_LIBS=-L/usr/local/lib
> -ljansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
> '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> '--enable-st-bins'
>
> Libs:
> libpcap version 1.5.3
> rdkafka 0.11.4
> jansson 2.12
>
> I can upgrade it to a newer version and try again.
>
>
> On Wed, Apr 15, 2020 at 8:59 AM Paolo Lucente <pa...@pmacct.net> wrote:
>
>>
>> Hey Emanuel,
>>
>> The config is correct and I did try your same config and that does work
>> for me, ie.:
>>
>> $ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
>> --topic pmacct.flows
>> {"event_type": "purge", "tag": 1, [ .. ]}
>>
>> What version of the software are you using? Is it 1.7.4p1 (latest
>> stable) or master code from GitHub? If so, is it possible an old running
>> nfacctd process is reading the data instead of the newly configured one?
>>
>> Paolo
>>
>> On Wed, Apr 15, 2020 at 12:17:43AM -0400, Emanuel dos Reis Rodrigues
>> wrote:
>> > I tried, follow my config:
>> >
>> > kafka_topic: netflow
>> > kafka_broker_host: 192.168100.105
>> > kafka_broker_port: 9092
>> > kafka_refresh_time: 1
>> > #daemonize: true
>> > plugins: kafka
>> > nfacctd_port: 9995
>> > post_tag: 1
>> > aggregate: tag, peer_src_ip, src_host, dst_host, timestamp_start,
>> > timestamp_end, src_port, dst_port, proto
>> >
>> >
>> > I kept the peer_src_ip, but the tag one is not being posted to Kafka.
>> >
>> > {'event_type': 'purge', 'peer_ip_src': '172.18.0.2', 'ip_src':
>> > '192.168.1.100', 'ip_dst': 'x.46.x.245', 'port_src': 51184, 'port_dst':
>> > 443, 'ip_proto': 'tcp', 'timestamp_start': '2020-04-14 14:15:39.000000',
>> > 'timestamp_end': '2020-04-14 14:15:54.000000', 'packets': 5, 'bytes':
>> 260,
>> > 'writer_id': 'default_kafka/75091'}
>> >
>> > Did I miss anything ?
>> >
>> >
>> > Thanks !
>> >
>> >
>> >
>> > On Tue, Apr 14, 2020 at 10:26 AM Paolo Lucente <pa...@pmacct.net>
>> wrote:
>> >
>> > >
>> > > I may have skipped the important detail you need to add the 'tag' key
>> to
>> > > your 'aggregate' line in the config, my bad. This is in addition to,
>> say,
>> > > 'post_tag: 1' to identify collector 1. Let me know how it goes.
>> > >
>> > > Paolo
>> > >
>> > > On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues
>> wrote:
>> > > > Thank you man, I did this test but I did not see the id being pushed
>> > > along
>> > > > with the Netflow info to Kafka topic. Is there the place the
>> information
>> > > > would show up ?
>> > > >
>> > > >
>> > > > On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente <pa...@pmacct.net>
>> wrote:
>> > > >
>> > > > >
>> > > > > Hi Emanuel,
>> > > > >
>> > > > > Apologies i did not get you wanted and ID for the collector. The
>> > > > > simplest way of achieving that is 'post_tag' as you just have to
>> supply
>> > > > > a number as ID; pre_tag_map expects a map and may be better to be
>> > > > > reserved for more complex use-cases.
>> > > > >
>> > > > > Paolo
>> > > > >
>> > > > > On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis
>> Rodrigues
>> > > wrote:
>> > > > > > Thank you for your help. Appreciate it !
>> > > > > >
>> > > > > > See, I did use it for testing after I sent this email. However,
>> the
>> > > ip
>> > > > > > showed there was the IP from my nfacctd machine, the collector
>> > > itself.
>> > > > > Not
>> > > > > > the exporter.
>> > > > > >
>> > > > > > peer_src_ip : IP address or identificator
>> of
>> > > > > telemetry
>> > > > > > exporting device
>> > > > > >
>> > > > > > In fact, it may have todo with the fact I currently have an SSH
>> > > tunnel
>> > > > > with
>> > > > > > socat with the remote machine in order to collect the data.
>> This may
>> > > be
>> > > > > the
>> > > > > > reason why which is definitively not a ordinary condition. :)
>> > > > > >
>> > > > > > I am wondering if I could use this one to include a different
>> tag on
>> > > it
>> > > > > > process/collector, but have not yet figured out how. Any
>> thoughts ?
>> > > > > >
>> > > > > > label : String label, ie. as result
>> of
>> > > > > > pre_tag_map evaluation
>> > > > > >
>> > > > > >
>> > > > > > Thank you again.
>> > > > > >
>> > > > > > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente <pa...@pmacct.net
>> >
>> > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > Hi Emanuel,
>> > > > > > >
>> > > > > > > I think you are looking for (i admit, non-intuitive)
>> 'peer_src_ip'
>> > > > > > > primitive:
>> > > > > > >
>> > > > > > > $ nfacctd -a | grep peer_src_ip
>> > > > > > > peer_src_ip : IP address or
>> identificator of
>> > > > > > > telemetry exporting device
>> > > > > > >
>> > > > > > > Without the grep you can see all supported primitives by the
>> > > nfacctd
>> > > > > > > release you are using along with a text explanation.
>> > > > > > >
>> > > > > > > Paolo
>> > > > > > >
>> > > > > > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis
>> > > Rodrigues
>> > > > > wrote:
>> > > > > > > > Hello guys,
>> > > > > > > >
>> > > > > > > > I implemented nfacctd acting as a Netflow collector using
>> > > pmacct. It
>> > > > > is
>> > > > > > > > working perfectly and writing the flows to a Kafka topic
>> which I
>> > > > > have an
>> > > > > > > > application processing it.
>> > > > > > > >
>> > > > > > > > Following is my configuration:
>> > > > > > > >
>> > > > > > > > kafka_topic: netflow
>> > > > > > > > kafka_broker_host: Kafka-host
>> > > > > > > > kafka_broker_port: 9092
>> > > > > > > > kafka_refresh_time: 1
>> > > > > > > > daemonize: true
>> > > > > > > > plugins: kafka
>> > > > > > > > pcap_interface: enp0s8
>> > > > > > > > nfacctd_ip: 192.168.1.100
>> > > > > > > > nfacctd_port: 9995
>> > > > > > > > aggregate: src_host, dst_host, timestamp_start,
>> timestamp_end,
>> > > > > src_port,
>> > > > > > > > dst_port, proto
>> > > > > > > >
>> > > > > > > > Currently, there is only one Netflow exporter sending data
>> to
>> > > this
>> > > > > > > > demon and I would like to add another exporter. The problem
>> is
>> > > that
>> > > > > I am
>> > > > > > > > not finding a way to differentiate the flows coming from
>> > > different
>> > > > > > > > exporters.
>> > > > > > > >
>> > > > > > > > Let's say I have the exporter A currently sending data to
>> nfacctd
>> > > > > running
>> > > > > > > > at port 9995 and the data is being written to Kafka topic
>> > > Netflow.
>> > > > > > > >
>> > > > > > > > Now I want a new exporter B to start sending data to
>> nfacctd port
>> > > > > 9996
>> > > > > > > which
>> > > > > > > > will be running as a separate demon ( just because I though
>> so,
>> > > not
>> > > > > sure
>> > > > > > > > yet if it is a necessary approach) and writing the data to
>> the
>> > > > > > > > same Netflow topic in Kafka.
>> > > > > > > >
>> > > > > > > > When the data comes from Kafka to my application, I cannot
>> tell
>> > > from
>> > > > > > > > which exporter the data came from. I would need some sort of
>> > > > > > > identification
>> > > > > > > > in order to make this differentiation. It is important for
>> me,
>> > > > > because my
>> > > > > > > > application may treat differently Netflow traffic coming
>> from
>> > > these
>> > > > > > > > two Netflow exporters.
>> > > > > > > >
>> > > > > > > > Thanks in advance.
>> > > > > > > >
>> > > > > > > > Emanuel
>> > > > > > >
>> > > > > > > > _______________________________________________
>> > > > > > > > pmacct-discussion mailing list
>> > > > > > > > http://www.pmacct.net/#mailinglists
>> > > > > > >
>> > > > > > >
>> > > > > > > _______________________________________________
>> > > > > > > pmacct-discussion mailing list
>> > > > > > > http://www.pmacct.net/#mailinglists
>> > > > > > >
>> > > > >
>> > >
>>
>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
