Mark and Yolanda, I submitted the PR I mentioned yesterday for the UI changes that surface the exposed prediction data. Let me know what you think.
https://github.com/apache/nifi/pull/3660 Thanks, Rob On Mon, Aug 19, 2019 at 4:17 PM Yolanda Davis <[email protected]> wrote: > Hi Mark and Rob > > Mark thanks so much for the info on your work and Rob thanks for jumping in > on the UI! I just wanted to add, Mark, that looking at your branch I think > we also may have some opportunities to exchange notes or collaborate on the > backend as well. The work in the feature branch is still in progress (with > some decoupling to ensure we can allow flexible configuration of models). > Please feel free to review and leave comments under the parent JIRA. At > the same time I'll take a deeper dive on your branch and perhaps we can > exchange notes on potential areas for improvement/collaboration if it makes > sense? > > Thanks Again, > > -yolanda > > > On Mon, Aug 19, 2019 at 3:34 PM Robert Fellows <[email protected]> > wrote: > > > Hey Mark, > > I've started working on some UI based on the initial commit for this > > proposal. What you have done and what I am working on have a bit of > > overlap, but not much. > > I'm working on getting the predicted count and bytes into the existing > > connection metric display that is already on the canvas. The only overlap > > looks like it might be in the > > Summary table. I plan on adding a PR for my additions hopefully tomorrow. > > Maybe once it is up we can discuss how we bring the them together where > it > > makes sense? > > > > This is the main JIRA case: > > https://issues.apache.org/jira/browse/NIFI-6510 > > And this is the subtask that I am working toward: > > https://issues.apache.org/jira/browse/NIFI-6568 > > > > > > -- Rob Fellows > > > > On Mon, Aug 19, 2019 at 2:26 PM Owens, Mark <[email protected]> > wrote: > > > > > The images from the preview email do not appear to be displaying. They > > can > > > be viewed at: > > > https://github.com/jmark99/nifi-images > > > > > > From: Owens, Mark <[email protected]> > > > Sent: Monday, August 19, 2019 2:25 PM > > > To: [email protected] > > > Subject: RE: Re:[EXT] [DISCUSS] Predictive Analytics for NiFi Metrics > > > > > > > > > Hi Yolanda, > > > > > > > > > > > > I've been working on a feature that appears to possibly overlap with > the > > > work you are pursuing. Perhaps we should see if/should we try to > > coordinate > > > our efforts. I've been updating NiFi to predict the time to queue > > overflow > > > for both flowfiles and bytes and displaying that information in the > GUI. > > > For the initial attempt, I’ve been using a simple model of straight > line > > > prediction over a sliding window of 15 minutes to predict when flows > will > > > fail. This estimate is then displayed on both the NiFi Summary page > under > > > the connections tab and in the status history graphs. Below are > examples > > > of what would be displayed to the user. > > > > > > > > > > > > [cid:[email protected]] > > > > > > > > > > > > The Connection tab contains a new column on the right that displays the > > > prediction for both flow files and data size. The user can select a > > maximum > > > time at which specific times are no longer displayed. In this example, > if > > > the prediction lies beyond 12 hours then the display simply indicates > > that > > > the flow is greater than 12 hours away from failure at the moment. > > > > > > > > > > > > [cid:[email protected]] > > > > > > > > > > > > This display graphs the prediction for byte overflow over time. Note > that > > > if the estimate is greater than the user provided maximum value of > > interest > > > the graph maxes out at that time, effectively indicating no overflow > > > concerns. > > > > > > > > > > > > [cid:[email protected]] > > > > > > > > > > > > A similar display for flowfile count is displayed as well. > > > > > > > > > > > > The current state of work can be found at > > > https://github.com/jmark99/nifi/tree/time-to-overflow > > > > > > > > > > > > I welcome your (or any others) feedback on this effort. > > > > > > > > > > > > Thanks, > > > Mark > > > > > > > > > > > > P.S. If the images are not displaying, they can be viewed at > > > https://github.com/jmark99/nifi-images > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > From: Yolanda Davis <[email protected]<mailto: > > > [email protected]>> > > > Sent: Monday, August 19, 2019 11:29 AM > > > To: [email protected]<mailto:[email protected]> > > > Subject: Re:[EXT] [DISCUSS] Predictive Analytics for NiFi Metrics > > > > > > > > > > > > Hello All, > > > > > > > > > > > > I just wanted to follow up on the discussion we started a couple of > weeks > > > ago concerning an analytics framework for NiFi metrics. Working with > > Andy > > > Christianson and Matt Burgess we shaped our ideas and drafted a > proposal > > > for this feature on the Apache NiFi Wiki [1] . We've also begun > > > implementing some of these ideas in a feature branch (which is work in > > > > > > progress) [2]. We’d appreciate any questions or feedback you may have. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > -yolanda > > > > > > > > > > > > [1] - > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Operational+Analytics+Framework+for+NiFi > > > > > > [2] - https://github.com/apache/nifi/commits/analytics-framework > > > > > > > > > > > > On Wed, Jul 31, 2019 at 9:58 AM Andy Christianson < > > [email protected] > > > .invalid<mailto:[email protected]>> wrote: > > > > > > > > > > > > > As someone who operated a 24/7 mission-critical NiFi flow, this > > > > > > > feature would have been a life saver. If I'm heading home on a > Friday, > > > > > > > it would be great to have some blinking red lights to let me know > that > > > > > > > the system predicts that it is going to experience backpressure > > > > > > > sometime over the weekend, so that corrective action could be taken > > > before leaving. > > > > > > > > > > > > > > Since there is support in the community for this, I created a JIRA to > > > > > > > track the effort: > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/NIFI-6510 > > > > > > > > > > > > > > I also created a JIRA to track the remote protocol: > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/NIFI-6511 > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Andy > > > > > > > > > > > > > > > > > > > > > Sent from ProtonMail, Swiss-based encrypted email. > > > > > > > > > > > > > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > > > > > On Wednesday, July 31, 2019 6:57 AM, Arpad Boda <[email protected] > > > <mailto:[email protected]>> wrote: > > > > > > > > > > > > > > > If you could share a bit more details about your OPC and Modbus > > > > > > > > usage, > > > > > > > that > > > > > > > > would be highly appreciated! > > > > > > > > > > > > > > > > On Wed, Jul 31, 2019 at 12:01 PM Craig Knell [email protected] > > > <mailto:[email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Sounds. Great > > > > > > > > > Let me know if you need some help > > > > > > > > > Best regards > > > > > > > > > Craig > > > > > > > > > > > > > > > > > > > On 31 Jul 2019, at 17:31, Arpad Boda [email protected] > > > <mailto:[email protected]> > > > > > > > wrote: > > > > > > > > > > Craig, > > > > > > > > > > OPC ( https://issues.apache.org/jira/browse/MINIFICPP-819 ) > and > > > > > > > Modbus ( > > > > > > > > > > https://issues.apache.org/jira/browse/MINIFICPP-897 ) are on > the > > > > > > > way for > > > > > > > > > > MiNiFi c++, hopefully both will be part of next release > (0.7.0). > > > > > > > > > > It's gonna be legen... wait for it! :) Regards, Arpad > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 31, 2019 at 2:30 AM Craig Knell > > > > > > > > > > > [email protected]<mailto:[email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Folks > > > > > > > > > > > That's our use case now. All our Models are run in python. > > > > > > > > > > > Currently we send events to the ML via http, although this is > > > > > > > > > > > not optimal > > > > > > > > > > > > > > > > > > > > > Our use case is edge ML where we want a light weight wrapper > > > > > > > > > > > for Python code base. > > > > > > > > > > > Jython however does not work with the code base I'm think of > > > > > > > > > > > changing the interface to some thing like REDIS for > > > > > > > pub/sub > > > > > > > > > > > Id also like this to be a push deployment via minifi Also > > > > > > > > > > > support for sensors via protocols via Modbus and OPC would be > > > > > > > great > > > > > > > > > > > Craig > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 31, 2019 at 1:43 AM Joe Witt > [email protected] > > > <mailto:[email protected]> > > > > > > > wrote: > > > > > > > > > > > > Definitely something that I think would really help the > > > > > > > community. It > > > > > > > > > > > > might make sense to frame/structure these APIs such that an > > > > > > > internal > > > > > > > > > > > > option > > > > > > > > > > > > could be available to reduce dependencies and get up and > > > > > > > > > > > > running > > > > > > > but > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > > also just as easily a remote implementation where the > engine > > > > > > > lives and > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > managed externally could also be supported. > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Tue, Jul 30, 2019 at 1:40 PM Andy LoPresto > > > > > > > [email protected]<mailto:[email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Yolanda, > > > > > > > > > > > > > I think this sounds like a great idea and will be very > > > > > > > > > > > > > useful > > > > > > > to > > > > > > > > > > > > > admins/users, as well as enabling some interesting > > > > > > > > > > > > > next-level functionality > > > > > > > > > > > > > > > > > > > > > > > > > and insight generation. Thanks for putting this out > there. > > > > > > > > > > > > > Andy LoPresto > > > > > > > > > > > > > [email protected]<mailto:[email protected]> > > > > > > > > > > > > > [email protected]<mailto: > > [email protected]> > > > PGP Fingerprint: 70EC B3E5 98A6 > > > > > > > > > > > > > 5A3F D3C4 BACE 3C6E F65B 2F7D > > > > > > > EF69 > > > > > > > > > > > > > > > > > > > > > > > > > > > On Jul 30, 2019, at 5:55 AM, Yolanda Davis < > > > > > > > > > > > > > > [email protected]<mailto: > > [email protected] > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > > > I wanted to reach out to the community to discuss > > > > > > > > > > > > > > potentially enhancing > > > > > > > > > > > > > > > > > > > > > > > > > > NiFi to include predictive analytics that can help > users > > > > > > > assess and > > > > > > > > > > > > > > predict > > > > > > > > > > > > > > NiFi behavior and performance. Currently NiFi has lots > > > > > > > > > > > > > > of > > > > > > > metrics > > > > > > > > > > > > > > available > > > > > > > > > > > > > > for areas including jvm and flow component usage (via > > > > > > > component > > > > > > > > > > > > > > status) > > > > > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > > > > well as provenance data which NiFi makes available > > > > > > > > > > > > > > either > > > > > > > through > > > > > > > > > > > > > > the UI > > > > > > > > > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > > > > > > > > reporting tasks (for consumption by other systems). > Past > > > > > > > discussions > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > community cite users shipping this data to applications > > > > > > > > > > > > > > such > > > > > > > as > > > > > > > > > > > > > > Prometheus, > > > > > > > > > > > > > > ELK stacks, or Ambari metrics for further analysis in > > > > > > > > > > > > > > order > > > > > > > to > > > > > > > > > > > > > > capture/review performance issues, detect anomalies, > and > > > > > > > send alerts > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > > > > > > > notifications. These systems are efficient in capturing > > > > > > > > > > > > > > and > > > > > > > helping > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > analyze these metrics however it requires customization > > > > > > > > > > > > > > work > > > > > > > and > > > > > > > > > > > > > > knowledge > > > > > > > > > > > > > > of NiFi operations to provide meaningful analytics > > > > > > > > > > > > > > within a > > > > > > > flow > > > > > > > > > > > > > > context. > > > > > > > > > > > > > > > > > > > > > > > > > > In speaking with Matt Burgess and Andy Christianson on > > > > > > > > > > > > > > this > > > > > > > topic we > > > > > > > > > > > > > > feel > > > > > > > > > > > > > > > > > > > > > > > > > > that there is an opportunity to introduce an analytics > > > > > > > framework that > > > > > > > > > > > > > > could > > > > > > > > > > > > > > provide users reasonable predictions on key performance > > > > > > > indicators > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > flows, such as back pressure and flow rate, to help > > > > > > > administrators > > > > > > > > > > > > > > improve > > > > > > > > > > > > > > operational management of NiFi clusters. This framework > > > > > > > could offer > > > > > > > > > > > > > > several key features: > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Provide a flexible internal analytics engine and > > model > > > > > > > api which > > > > > > > > > > > > > > supports the addition of or enhancement to onboard > > > > > > > > > > > > > > models > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Support integration of remote or cloud based ML > > models > > > > > > > > > > > > > > - Support both traditional and online (incremental) > > > > > > > learning > > > > > > > > > > > > > > methods > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Provide support for model caching (perhaps later > > > > > > > inclusion into > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > model repository or registry) > > > > > > > > > > > > > > > > > > > > > > > > > > > > - UI enhancements to display prediction information > > > either > > > > > > > in > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > summary data, new data visualizations, or directly > > > > > > > > > > > > > > within the flow/canvas (where applicable) For an > initial > > > > > > > > > > > > > > target we thought that back pressure > > > > > > > prediction would > > > > > > > > > > > > > > be a > > > > > > > > > > > > > > > > > > > > > > > > > > good starting point for this initiative, given that > back > > > > > > > pressure > > > > > > > > > > > > > > detection > > > > > > > > > > > > > > is a key indicator of flow performance and many of the > > > > > > > metrics > > > > > > > > > > > > > > currently > > > > > > > > > > > > > > > > > > > > > > > > > > available would provide enough data points to create a > > > > > > > reasonable > > > > > > > > > > > > > > performing model. We have some ideas on how this could > > > > > > > > > > > > > > be > > > > > > > achieved > > > > > > > > > > > > > > however > > > > > > > > > > > > > > we wanted to discuss this more with the community to > get > > > > > > > thoughts > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > > > > > > > > tackling this work, especially if there are specific > use > > > > > > > cases or > > > > > > > > > > > > > > other > > > > > > > > > > > > > > > > > > > > > > > > > > factors that should be considered. > > > > > > > > > > > > > > Looking forward to everyone's thoughts and input. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > -yolanda > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > [email protected]<mailto: > > [email protected]> > > > @YolandaMDavis > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Regards > > > > > > > > > > > Craig Knell > > > > > > > > > > > Mobile: +61 402 128 615 > > > > > > > > > > > Skype: craigknell > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > [email protected]<mailto:[email protected]> > > > > > > @YolandaMDavis > > > > > > > > > -- > > ------------------------------- > > Rob Fellows > > > > > -- > -- > [email protected] > @YolandaMDavis > -- ------------------------------- Rob Fellows
