I saw that now cascading supports Flink..so maybe you could think in programming a cascading abstraction to have also spark and tez compatibility for free!what do you think? On 22 Sep 2015 11:17, "Christian Kreutzfeldt" <mnx...@gmail.com> wrote:
> Hi Slim, > > thanks for sharing the presentation. Like Flavio I see the (UI) > capabilities provided by Apache NiFi as very helpful for (enterprise) > customers. > > At the Otto Group we are currently think about how to track data through a > stream based system like we do in the batch world using staging layers. > Apache NiFi seems to provide a feasible approach which gives lots of > insights into your running system. Especially in those cases where > topologies are still under development or people are curious on how these > mysterious streaming systems work UI supported development & tracking / > monitoring is a big win. > > Having these requirements in mind, I don't see an integration point with > Apache NiFi as the project does not only provide an UI/orchestration layer > but has a runtime etc. available too. An integration like Spark (shown > right at the beginning of the demo) seems to be possible but what I'd like > to see is a more Apache NiFi-like UI layer exclusively provided for Apache > Flink. It would bring Flink very (!!!) much closer to the enterprise market > as it solves some crucial requirements when it comes to development and > monitoring - no need for developers with in-depth programming knowledge > since topologies may be built visually, no need for depth tech > understanding when it comes to monitoring ... > > From my point of view, solving the issue includes the following aspects: > > * providing an appropriate UI layer to visualize data flow / operator > graphs along with metrics (system & user-defined) exported by operators > * message tracking by integrating K/V stores and index management tools > > ...probably both worth more than a year development time ;-) > > But if there were some efforts to implement & integrate such a feature, > text me if dev support is require. > > Best regards, > Christian > > > 2015-09-19 12:42 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>: > > > As a user that would be very helpful! > > On 19 Sep 2015 12:34, "Slim Baltagi" <sbalt...@gmail.com> wrote: > > > > > Hi Flink experts! > > > > > > I came across Apache Nifi https://nifi.apache.org > > > https://www.youtube.com/watch?v=sQCgtCoZyFQ > > > In the Nifi project, there is an open JIRA > > > issue:https://issues.apache.org/jira/browse/NIFI-823 to evaluate / > > provide > > > integration with Apache Flink. > > > > > > Is the integration of Flink with NiFi on the roadmap of the Apache > Flink > > > project? > > > > > > Expected advantages for Flink when integrating with Nifi would be: > > > - GUI: Web-based user interface to design data flows > > > - Dynamic dataflows: Modify dataflow at runtime > > > - Security: Authentication, authorization, encryption > > > - Data Provenance: Track data flow from beginning to end > > > > > > What do you think? > > > > > > Thanks > > > > > > Slim Baltagi > > > > > > > > > > > > > > > > > > > > > -- > > > View this message in context: > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/On-integrating-Flink-with-Apache-NiFi-tp8059.html > > > Sent from the Apache Flink Mailing List archive. mailing list archive > at > > > Nabble.com. > > > > > >
