Definitely something that I think would really help the community. It might make sense to frame/structure these APIs such that an internal option could be available to reduce dependencies and get up and running but that also just as easily a remote implementation where the engine lives and is managed externally could also be supported.
Thanks On Tue, Jul 30, 2019 at 1:40 PM Andy LoPresto <[email protected]> wrote: > Yolanda, > > I think this sounds like a great idea and will be very useful to > admins/users, as well as enabling some interesting next-level functionality > and insight generation. Thanks for putting this out there. > > Andy LoPresto > [email protected] > [email protected] > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jul 30, 2019, at 5:55 AM, Yolanda Davis <[email protected]> > wrote: > > > > Hello Everyone, > > > > I wanted to reach out to the community to discuss potentially enhancing > > NiFi to include predictive analytics that can help users assess and > predict > > NiFi behavior and performance. Currently NiFi has lots of metrics > available > > for areas including jvm and flow component usage (via component status) > as > > well as provenance data which NiFi makes available either through the UI > or > > reporting tasks (for consumption by other systems). Past discussions in > the > > community cite users shipping this data to applications such as > Prometheus, > > ELK stacks, or Ambari metrics for further analysis in order to > > capture/review performance issues, detect anomalies, and send alerts or > > notifications. These systems are efficient in capturing and helping to > > analyze these metrics however it requires customization work and > knowledge > > of NiFi operations to provide meaningful analytics within a flow context. > > > > In speaking with Matt Burgess and Andy Christianson on this topic we feel > > that there is an opportunity to introduce an analytics framework that > could > > provide users reasonable predictions on key performance indicators for > > flows, such as back pressure and flow rate, to help administrators > improve > > operational management of NiFi clusters. This framework could offer > > several key features: > > > > - Provide a flexible internal analytics engine and model api which > > supports the addition of or enhancement to onboard models > > - Support integration of remote or cloud based ML models > > - Support both traditional and online (incremental) learning methods > > - Provide support for model caching (perhaps later inclusion into a > > model repository or registry) > > - UI enhancements to display prediction information either in existing > > summary data, new data visualizations, or directly within the > flow/canvas > > (where applicable) > > > > For an initial target we thought that back pressure prediction would be a > > good starting point for this initiative, given that back pressure > detection > > is a key indicator of flow performance and many of the metrics currently > > available would provide enough data points to create a reasonable > > performing model. We have some ideas on how this could be achieved > however > > we wanted to discuss this more with the community to get thoughts about > > tackling this work, especially if there are specific use cases or other > > factors that should be considered. > > > > Looking forward to everyone's thoughts and input. > > > > Thanks, > > > > -yolanda > > > > -- > > [email protected] > > @YolandaMDavis > >
