As someone who operated a 24/7 mission-critical NiFi flow, this feature would have been a life saver. If I'm heading home on a Friday, it would be great to have some blinking red lights to let me know that the system predicts that it is going to experience backpressure sometime over the weekend, so that corrective action could be taken before leaving.
Since there is support in the community for this, I created a JIRA to track the effort: https://issues.apache.org/jira/browse/NIFI-6510 I also created a JIRA to track the remote protocol: https://issues.apache.org/jira/browse/NIFI-6511 Regards, Andy Sent from ProtonMail, Swiss-based encrypted email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, July 31, 2019 6:57 AM, Arpad Boda <[email protected]> wrote: > If you could share a bit more details about your OPC and Modbus usage, that > would be highly appreciated! > > On Wed, Jul 31, 2019 at 12:01 PM Craig Knell [email protected] wrote: > > > Sounds. Great > > Let me know if you need some help > > Best regards > > Craig > > > > > On 31 Jul 2019, at 17:31, Arpad Boda [email protected] wrote: > > > Craig, > > > OPC ( https://issues.apache.org/jira/browse/MINIFICPP-819 ) and Modbus ( > > > https://issues.apache.org/jira/browse/MINIFICPP-897 ) are on the way for > > > MiNiFi c++, hopefully both will be part of next release (0.7.0). > > > It's gonna be legen... wait for it! :) > > > Regards, > > > Arpad > > > > > > > On Wed, Jul 31, 2019 at 2:30 AM Craig Knell [email protected] > > > > wrote: > > > > > > > Hi Folks > > > > That's our use case now. All our Models are run in python. > > > > Currently we send events to the ML via http, although this is not > > > > optimal > > > > > > > Our use case is edge ML where we want a light weight wrapper for > > > > Python code base. > > > > Jython however does not work with the code base > > > > I'm think of changing the interface to some thing like REDIS for pub/sub > > > > Id also like this to be a push deployment via minifi > > > > Also support for sensors via protocols via Modbus and OPC would be great > > > > Craig > > > > > > > > > On Wed, Jul 31, 2019 at 1:43 AM Joe Witt [email protected] wrote: > > > > > Definitely something that I think would really help the community. It > > > > > might make sense to frame/structure these APIs such that an internal > > > > > option > > > > > could be available to reduce dependencies and get up and running but > > > > > that > > > > > > > > also just as easily a remote implementation where the engine lives and > > > > > is > > > > > > > > managed externally could also be supported. > > > > > Thanks > > > > > On Tue, Jul 30, 2019 at 1:40 PM Andy LoPresto [email protected] > > > > > wrote: > > > > > > > > > > > Yolanda, > > > > > > I think this sounds like a great idea and will be very useful to > > > > > > admins/users, as well as enabling some interesting next-level > > > > > > functionality > > > > > > > > > > > and insight generation. Thanks for putting this out there. > > > > > > Andy LoPresto > > > > > > [email protected] > > > > > > [email protected] > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > > > > > > > > > On Jul 30, 2019, at 5:55 AM, Yolanda Davis < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hello Everyone, > > > > > > > I wanted to reach out to the community to discuss potentially > > > > > > > enhancing > > > > > > > > > > > > NiFi to include predictive analytics that can help users assess > > > > > > > and > > > > > > > predict > > > > > > > NiFi behavior and performance. Currently NiFi has lots of metrics > > > > > > > available > > > > > > > for areas including jvm and flow component usage (via component > > > > > > > status) > > > > > > > > > > > as > > > > > > > > > > > > > well as provenance data which NiFi makes available either through > > > > > > > the UI > > > > > > > > > > > or > > > > > > > > > > > > > reporting tasks (for consumption by other systems). Past > > > > > > > discussions > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > community cite users shipping this data to applications such as > > > > > > > Prometheus, > > > > > > > ELK stacks, or Ambari metrics for further analysis in order to > > > > > > > capture/review performance issues, detect anomalies, and send > > > > > > > alerts > > > > > > > or > > > > > > > > > > > > notifications. These systems are efficient in capturing and > > > > > > > helping > > > > > > > to > > > > > > > > > > > > analyze these metrics however it requires customization work and > > > > > > > knowledge > > > > > > > of NiFi operations to provide meaningful analytics within a flow > > > > > > > context. > > > > > > > > > > > > In speaking with Matt Burgess and Andy Christianson on this topic > > > > > > > we > > > > > > > feel > > > > > > > > > > > > that there is an opportunity to introduce an analytics framework > > > > > > > that > > > > > > > could > > > > > > > provide users reasonable predictions on key performance indicators > > > > > > > for > > > > > > > > > > > > flows, such as back pressure and flow rate, to help administrators > > > > > > > improve > > > > > > > operational management of NiFi clusters. This framework could > > > > > > > offer > > > > > > > several key features: > > > > > > > > > > > > > > - Provide a flexible internal analytics engine and model api > > > > > > > which > > > > > > > supports the addition of or enhancement to onboard models > > > > > > > > > > > > > > - Support integration of remote or cloud based ML models > > > > > > > - Support both traditional and online (incremental) learning > > > > > > > methods > > > > > > > > > > > > > > > > > > > - Provide support for model caching (perhaps later inclusion > > > > > > > into > > > > > > > a > > > > > > > > > > > > > > > > > > > model repository or registry) > > > > > > > > > > > > > > - UI enhancements to display prediction information either in > > > > > > > existing > > > > > > > > > > > > > > > > > > > summary data, new data visualizations, or directly within the > > > > > > > flow/canvas > > > > > > > (where applicable) > > > > > > > For an initial target we thought that back pressure prediction > > > > > > > would > > > > > > > be a > > > > > > > > > > > > good starting point for this initiative, given that back pressure > > > > > > > detection > > > > > > > is a key indicator of flow performance and many of the metrics > > > > > > > currently > > > > > > > > > > > > available would provide enough data points to create a reasonable > > > > > > > performing model. We have some ideas on how this could be achieved > > > > > > > however > > > > > > > we wanted to discuss this more with the community to get thoughts > > > > > > > about > > > > > > > > > > > > tackling this work, especially if there are specific use cases or > > > > > > > other > > > > > > > > > > > > factors that should be considered. > > > > > > > Looking forward to everyone's thoughts and input. > > > > > > > Thanks, > > > > > > > -yolanda > > > > > > > -- > > > > > > > [email protected] > > > > > > > @YolandaMDavis > > > > > > > > -- > > > > Regards > > > > Craig Knell > > > > Mobile: +61 402 128 615 > > > > Skype: craigknell
