On Tue, Aug 21, 2018 at 8:54 AM Dominik Charousset <dominik.charous...@haw-hamburg.de> wrote:
> This raises a couple of questions. Primarily: where can Broker users learn > the layouts to interpret received data? broker/bro.hh is basically all there is right now. e.g. if you construct a broker::bro::Event from a received broker::data, you get access to event name + "interpretable" arguments. > There’s essentially no hope in deferring a layout, since broker::data doesn’t > include any meta information such as field names. Is meta information stored > somewhere? No, nothing like that is in implicit in message content. > Is there a convention how to place and retrieve such meta information in > Broker’s data stores? No, and any stores created by Bro don't even have such meta info. The types stored in them are just documented like "the keys are type Foo and values are type Bar". > How does Bro itself make such information available? Nothing beyond documentation or the Bro -> Broker type mapping that's implicit in events themselves (or as given in docs for data stores). > Is there a document that lists all topics used by Bro with their respective > broker::data layout? I don't think there's a plan to keep such an up-to-date document. A basic usage premise that I'm wondering about is that none of the current Broker usage in Bro actually seems suitable for generic/public consumption as it is. It's maybe more implementation details of doing cluster-enabled network traffic analysis, so also not a primary goal to make interpretation of those communications easy for external/direct Broker users. (You can ingest it if you want and do the work of "manually" interpreting it all, but maybe won't be a stable/transparent source of data going forward). However, one can still use Bro + Broker to create their own events/stores in a way that does contain the meta information required for easier/programmatic interpretation on the receiving side. e.g. I think, at the moment, if one is interested in ingesting data produced by Bro, they are best served by explicitly defining topic names, event/data types, and explicitly producing those messages at suitable places within Bro scripts themselves. Then, one can be in control of defining a common/expected data format and include whatever meta information is necessary to help receivers interpret the data. Maybe there's a more standardized approach that could be worked towards, but likely we just need more experience in understanding and defining common use-cases for external Bro data consumption. Or if we were just talking about Broker-only usage independent of Bro, then I think it's still the same ideas/answers: currently up to user to decide how to encode broker::data in a way that defines common/expected layouts + any required meta info. Does that help at all? - Jon _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
