I see, I actually tried it before and the realized the write_id was
changing based on the PID of nfacctd. Do you know what is the parameter to
customize the writer_id ?
Thanks !
Best Regards,
Emanuel
On Sun, Apr 19, 2020 at 11:42 AM Brian Solar <brian.so...@pm.me> wrote:
> You already seem to have a solution, but to me the writer_id is what you
> want. Change the name of the process in your configuration file.
>
>
>
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Wednesday, April 15, 2020 7:33 PM, Emanuel dos Reis Rodrigues <
> emanueldosr...@gmail.com> wrote:
>
> Hey, I just realize it worked. I think I was little behind on the messages
> parking on my kafka, now I can see the tag.
>
> Thank you so much for your help.
>
> On Wed, Apr 15, 2020 at 10:33 AM Emanuel dos Reis Rodrigues <
> emanueldosr...@gmail.com> wrote:
>
>> I am using:
>>
>> NetFlow Accounting Daemon, nfacctd 1.7.2-git (20181018-00+c3)
>>
>> Arguments:
>> '--enable-kafka' '--enable-jansson'
>> 'JANSSON_CFLAGS=-I/usr/local/include/' 'JANSSON_LIBS=-L/usr/local/lib
>> -ljansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
>> '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
>> '--enable-st-bins'
>>
>> Libs:
>> libpcap version 1.5.3
>> rdkafka 0.11.4
>> jansson 2.12
>>
>> I can upgrade it to a newer version and try again.
>>
>>
>> On Wed, Apr 15, 2020 at 8:59 AM Paolo Lucente <pa...@pmacct.net> wrote:
>>
>>>
>>> Hey Emanuel,
>>>
>>> The config is correct and I did try your same config and that does work
>>> for me, ie.:
>>>
>>> $ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
>>> --topic pmacct.flows
>>> {"event_type": "purge", "tag": 1, [ .. ]}
>>>
>>> What version of the software are you using? Is it 1.7.4p1 (latest
>>> stable) or master code from GitHub? If so, is it possible an old running
>>> nfacctd process is reading the data instead of the newly configured one?
>>>
>>> Paolo
>>>
>>> On Wed, Apr 15, 2020 at 12:17:43AM -0400, Emanuel dos Reis Rodrigues
>>> wrote:
>>> > I tried, follow my config:
>>> >
>>> > kafka_topic: netflow
>>> > kafka_broker_host: 192.168100.105
>>> > kafka_broker_port: 9092
>>> > kafka_refresh_time: 1
>>> > #daemonize: true
>>> > plugins: kafka
>>> > nfacctd_port: 9995
>>> > post_tag: 1
>>> > aggregate: tag, peer_src_ip, src_host, dst_host, timestamp_start,
>>> > timestamp_end, src_port, dst_port, proto
>>> >
>>> >
>>> > I kept the peer_src_ip, but the tag one is not being posted to Kafka.
>>> >
>>> > {'event_type': 'purge', 'peer_ip_src': '172.18.0.2', 'ip_src':
>>> > '192.168.1.100', 'ip_dst': 'x.46.x.245', 'port_src': 51184, 'port_dst':
>>> > 443, 'ip_proto': 'tcp', 'timestamp_start': '2020-04-14
>>> 14:15:39.000000',
>>> > 'timestamp_end': '2020-04-14 14:15:54.000000', 'packets': 5, 'bytes':
>>> 260,
>>> > 'writer_id': 'default_kafka/75091'}
>>> >
>>> > Did I miss anything ?
>>> >
>>> >
>>> > Thanks !
>>> >
>>> >
>>> >
>>> > On Tue, Apr 14, 2020 at 10:26 AM Paolo Lucente <pa...@pmacct.net>
>>> wrote:
>>> >
>>> > >
>>> > > I may have skipped the important detail you need to add the 'tag'
>>> key to
>>> > > your 'aggregate' line in the config, my bad. This is in addition to,
>>> say,
>>> > > 'post_tag: 1' to identify collector 1. Let me know how it goes.
>>> > >
>>> > > Paolo
>>> > >
>>> > > On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues
>>> wrote:
>>> > > > Thank you man, I did this test but I did not see the id being
>>> pushed
>>> > > along
>>> > > > with the Netflow info to Kafka topic. Is there the place the
>>> information
>>> > > > would show up ?
>>> > > >
>>> > > >
>>> > > > On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente <pa...@pmacct.net>
>>> wrote:
>>> > > >
>>> > > > >
>>> > > > > Hi Emanuel,
>>> > > > >
>>> > > > > Apologies i did not get you wanted and ID for the collector. The
>>> > > > > simplest way of achieving that is 'post_tag' as you just have to
>>> supply
>>> > > > > a number as ID; pre_tag_map expects a map and may be better to be
>>> > > > > reserved for more complex use-cases.
>>> > > > >
>>> > > > > Paolo
>>> > > > >
>>> > > > > On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis
>>> Rodrigues
>>> > > wrote:
>>> > > > > > Thank you for your help. Appreciate it !
>>> > > > > >
>>> > > > > > See, I did use it for testing after I sent this email.
>>> However, the
>>> > > ip
>>> > > > > > showed there was the IP from my nfacctd machine, the collector
>>> > > itself.
>>> > > > > Not
>>> > > > > > the exporter.
>>> > > > > >
>>> > > > > > peer_src_ip : IP address or identificator
>>> of
>>> > > > > telemetry
>>> > > > > > exporting device
>>> > > > > >
>>> > > > > > In fact, it may have todo with the fact I currently have an SSH
>>> > > tunnel
>>> > > > > with
>>> > > > > > socat with the remote machine in order to collect the data.
>>> This may
>>> > > be
>>> > > > > the
>>> > > > > > reason why which is definitively not a ordinary condition. :)
>>> > > > > >
>>> > > > > > I am wondering if I could use this one to include a different
>>> tag on
>>> > > it
>>> > > > > > process/collector, but have not yet figured out how. Any
>>> thoughts ?
>>> > > > > >
>>> > > > > > label : String label, ie. as result
>>> of
>>> > > > > > pre_tag_map evaluation
>>> > > > > >
>>> > > > > >
>>> > > > > > Thank you again.
>>> > > > > >
>>> > > > > > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente <
>>> pa...@pmacct.net>
>>> > > wrote:
>>> > > > > >
>>> > > > > > >
>>> > > > > > > Hi Emanuel,
>>> > > > > > >
>>> > > > > > > I think you are looking for (i admit, non-intuitive)
>>> 'peer_src_ip'
>>> > > > > > > primitive:
>>> > > > > > >
>>> > > > > > > $ nfacctd -a | grep peer_src_ip
>>> > > > > > > peer_src_ip : IP address or
>>> identificator of
>>> > > > > > > telemetry exporting device
>>> > > > > > >
>>> > > > > > > Without the grep you can see all supported primitives by the
>>> > > nfacctd
>>> > > > > > > release you are using along with a text explanation.
>>> > > > > > >
>>> > > > > > > Paolo
>>> > > > > > >
>>> > > > > > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis
>>> > > Rodrigues
>>> > > > > wrote:
>>> > > > > > > > Hello guys,
>>> > > > > > > >
>>> > > > > > > > I implemented nfacctd acting as a Netflow collector using
>>> > > pmacct. It
>>> > > > > is
>>> > > > > > > > working perfectly and writing the flows to a Kafka topic
>>> which I
>>> > > > > have an
>>> > > > > > > > application processing it.
>>> > > > > > > >
>>> > > > > > > > Following is my configuration:
>>> > > > > > > >
>>> > > > > > > > kafka_topic: netflow
>>> > > > > > > > kafka_broker_host: Kafka-host
>>> > > > > > > > kafka_broker_port: 9092
>>> > > > > > > > kafka_refresh_time: 1
>>> > > > > > > > daemonize: true
>>> > > > > > > > plugins: kafka
>>> > > > > > > > pcap_interface: enp0s8
>>> > > > > > > > nfacctd_ip: 192.168.1.100
>>> > > > > > > > nfacctd_port: 9995
>>> > > > > > > > aggregate: src_host, dst_host, timestamp_start,
>>> timestamp_end,
>>> > > > > src_port,
>>> > > > > > > > dst_port, proto
>>> > > > > > > >
>>> > > > > > > > Currently, there is only one Netflow exporter sending data
>>> to
>>> > > this
>>> > > > > > > > demon and I would like to add another exporter. The
>>> problem is
>>> > > that
>>> > > > > I am
>>> > > > > > > > not finding a way to differentiate the flows coming from
>>> > > different
>>> > > > > > > > exporters.
>>> > > > > > > >
>>> > > > > > > > Let's say I have the exporter A currently sending data to
>>> nfacctd
>>> > > > > running
>>> > > > > > > > at port 9995 and the data is being written to Kafka topic
>>> > > Netflow.
>>> > > > > > > >
>>> > > > > > > > Now I want a new exporter B to start sending data to
>>> nfacctd port
>>> > > > > 9996
>>> > > > > > > which
>>> > > > > > > > will be running as a separate demon ( just because I
>>> though so,
>>> > > not
>>> > > > > sure
>>> > > > > > > > yet if it is a necessary approach) and writing the data
>>> to the
>>> > > > > > > > same Netflow topic in Kafka.
>>> > > > > > > >
>>> > > > > > > > When the data comes from Kafka to my application, I cannot
>>> tell
>>> > > from
>>> > > > > > > > which exporter the data came from. I would need some sort
>>> of
>>> > > > > > > identification
>>> > > > > > > > in order to make this differentiation. It is important for
>>> me,
>>> > > > > because my
>>> > > > > > > > application may treat differently Netflow traffic coming
>>> from
>>> > > these
>>> > > > > > > > two Netflow exporters.
>>> > > > > > > >
>>> > > > > > > > Thanks in advance.
>>> > > > > > > >
>>> > > > > > > > Emanuel
>>> > > > > > >
>>> > > > > > > > _______________________________________________
>>> > > > > > > > pmacct-discussion mailing list
>>> > > > > > > > http://www.pmacct.net/#mailinglists
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > _______________________________________________
>>> > > > > > > pmacct-discussion mailing list
>>> > > > > > > http://www.pmacct.net/#mailinglists
>>> > > > > > >
>>> > > > >
>>> > >
>>>
>>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
