Yolanda, I think this sounds like a great idea and will be very useful to admins/users, as well as enabling some interesting next-level functionality and insight generation. Thanks for putting this out there.
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 30, 2019, at 5:55 AM, Yolanda Davis <[email protected]> wrote: > > Hello Everyone, > > I wanted to reach out to the community to discuss potentially enhancing > NiFi to include predictive analytics that can help users assess and predict > NiFi behavior and performance. Currently NiFi has lots of metrics available > for areas including jvm and flow component usage (via component status) as > well as provenance data which NiFi makes available either through the UI or > reporting tasks (for consumption by other systems). Past discussions in the > community cite users shipping this data to applications such as Prometheus, > ELK stacks, or Ambari metrics for further analysis in order to > capture/review performance issues, detect anomalies, and send alerts or > notifications. These systems are efficient in capturing and helping to > analyze these metrics however it requires customization work and knowledge > of NiFi operations to provide meaningful analytics within a flow context. > > In speaking with Matt Burgess and Andy Christianson on this topic we feel > that there is an opportunity to introduce an analytics framework that could > provide users reasonable predictions on key performance indicators for > flows, such as back pressure and flow rate, to help administrators improve > operational management of NiFi clusters. This framework could offer > several key features: > > - Provide a flexible internal analytics engine and model api which > supports the addition of or enhancement to onboard models > - Support integration of remote or cloud based ML models > - Support both traditional and online (incremental) learning methods > - Provide support for model caching (perhaps later inclusion into a > model repository or registry) > - UI enhancements to display prediction information either in existing > summary data, new data visualizations, or directly within the flow/canvas > (where applicable) > > For an initial target we thought that back pressure prediction would be a > good starting point for this initiative, given that back pressure detection > is a key indicator of flow performance and many of the metrics currently > available would provide enough data points to create a reasonable > performing model. We have some ideas on how this could be achieved however > we wanted to discuss this more with the community to get thoughts about > tackling this work, especially if there are specific use cases or other > factors that should be considered. > > Looking forward to everyone's thoughts and input. > > Thanks, > > -yolanda > > -- > [email protected] > @YolandaMDavis
