Hi Aaron, Wrt kafka_refresh_time set to 1 sec, please give it a try. I'd be curious myself about your feedback. In theory it's a valid configuration and it should work. As extra tests, if 60 secs is not live enough, i may recommend towards 10, 15 or 20 secs.
Paolo On Fri, Feb 17, 2017 at 08:28:11AM -0800, Aaron Finney wrote: > Hi Paolo, > > That's what I get for becoming our internal package maintainer for pmacct > by default - updating it now! :D > > One quick sanity check regarding output to Kafka - would setting > kafka_refresh_time to 1 add significant overhead to nfacctd, e.g. related > to buffer flushing process/etc? > > My goal is to produce something closer to a live stream, but with > micro-flows aggregated to 1s granularity, then use Spark to do additional > transformations/aggregations against local data along with basic anomaly > detection. > > Thanks! > > Aaron > > > On Fri, Feb 17, 2017 at 5:58 AM, Paolo Lucente <[email protected]> wrote: > > > > > Hi Aaron, > > > > The feature is post 1.6.1. Can you please switch to master code on GitHub? > > > > Thanks, > > Paolo > > > > On Thu, Feb 16, 2017 at 10:44:23AM -0800, Aaron Finney wrote: > > > Hi Paolo/all, > > > > > > I've been unable to get nfacctd to send the Avro schema to a Kafka topic > > - > > > I receive the following message when starting nfacctd: > > > > > > WARN: [/etc/nfacctd.conf:14] Unknown key: kafka_avro_schema_topic. > > Ignored. > > > WARN: [/etc/nfacctd.conf:15] Unknown key: kafka_avro_schema_refresh_ > > time. > > > Ignored. > > > > > > Kafka/Avro are otherwise working; the process successfully writes the > > > schema to a local file specified in avro_schema_output_file, but I'd like > > > to have it send to a schema topic in the cluster itself if possible. > > > Version info is below. > > > > > > Thanks! > > > > > > Aaron > > > > > > > > > INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.6.1 > > > (20161001-00+c5) > > > INFO ( default/core ): '--enable-kafka' '--enable-jansson' > > '--enable-avro' > > > 'KAFKA_LIBS=-L/usr/lib64/ -lrdkafka' 'AVRO_CFLAGS=-I/usr/include/avro' > > > 'AVRO_LIBS=-L/usr/lib -lavro' > > > > > _______________________________________________ > > > pmacct-discussion mailing list > > > http://www.pmacct.net/#mailinglists > > > > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
