Flink’s stable API provides the frameworks (DataStream and DataSet). On top of these frameworks Gelly provides additional models for iterative algorithms, but there are algorithms such as Minimum Spanning Tree which do not easily map to these models (in this instance requiring nested iterations; for PageRank it was handling directed graphs; for HITS it was processing both in- and out-edges in the same iteration).
One challenge with caching results is when to release the resources. New algorithms typically require new capabilities, the latter typically requiring much more work, so the algorithms are virtually free. Updating multiple DataSets in an iteration should be another consideration for improving the scheduler. Where has this been a limitation? > On Feb 27, 2017, at 8:03 AM, Xingcan Cui <xingc...@gmail.com> wrote: > > Hi Vasia and Greg, > > thanks for the discussion. I'd like to share my thoughts. > > 1) I don't think it's necessary to extend the algorithm list intentionally. > It's just like a textbook that can not cover all the existing algorithms > (even if we can). Just representative and commonly used ones will be > enough. After all, Gelly is mainly designed for providing a framework > rather than an algorithm library. Besides, it seems that Gelly's API is not > stable now and thus a huge work of refactoring or even rewriting will rise > once the API changes. > > 2) Unlike other "pure" graph computing framework (e.g. giraph), Gelly is > built on top of Flink, which means that it can only use operations that > provided by it. In my own opinion, Flink's batch processing is not so > outstanding as it's stream. As Grey said, one problem lies on intermediate > results caching. Though it's not clear for me (I'm still a ignorant new > comer...) why this feature has not been implemented for such a long time, > there must be some reasons. What I see is that, to some extent, it's > already obstructed Gelly's development. From this point of view, > self-blessing is better than blessing from others and I'm sure some MLers > may be more anxious than us :) So, I guess "within Gelly" just means a > Gelly-driven development? > > In a nutshell, I will encourage more concentrations on Gelly's API (or even > related Flink's API if necessary), rather than high-level things (e.g. > algorithms, performance) on top of it. What if we can change both the > edges' values and vertices' values during an iteration one day? :) > > Best, > Xingcan > > > On Sat, Feb 25, 2017 at 2:43 AM, Vasiliki Kalavri <vasilikikala...@gmail.com >> wrote: > >> Hi Greg, >> >> On 24 February 2017 at 18:09, Greg Hogan <c...@greghogan.com> wrote: >> >>> Thanks, Vasia, for starting the discussion. >>> >>> I was expecting more changes from the recent discussion on restructuring >>> the project, in particular regarding the libraries. Gelly has always >>> collected algorithms and I have personally taken an algorithms-first >>> approach for contributions. Is that manageable and maintainable? I'd >> prefer >>> to see no limit to good contributions, and if necessary split the >> codebase >>> or the project. >>> >> >> I don't think there should be a limit either. I do think though that >> development should be community-driven, i.e. not making contributions just >> for the sake of it, but evaluating their benefit first. >> The library already has a quite long list of algorithms. Shall we keep on >> extending it? And if yes, how do we choose which algorithms to add? Do we >> accept any algorithm even if it hasn't been asked by anyone? So far, we've >> added algorithms that we thought were useful and common. But continuing to >> extend the library like this doesn't seem maintainable to me, because we >> might end up with a lot of code to maintain that nobody uses. On the other >> hand, adding more algorithms might attract more users, so I see a trade-off >> there. >> >> >>> >>> If so, then a secondary goal is to make the algorithms user-accessible >> and >>> easier to review (especially at scale!). FLINK-4949 rewrites >>> flink-gelly-examples with modular inputs and algorithms, allows users to >>> run all existing algorithms, and makes it trivial to create a driver for >>> new algorithms (and when comparing different implementations). >>> >> >> I'm +1 for anything that makes using existing functionality easier. >> FLINK-4949 sounds like a great addition. Could you maybe extend the JIRA >> and/or PR description a bit? I understand the rationale but it would be >> nice to have a high-level description of the changes and the new >> functionality that the PR adds or the interfaces it modifies. Otherwise, it >> will be difficult to review a PR with +5k line changes :) >> >> >> >>> >>> Regarding BipartiteGraphs, without algorithms or ideas for algorithms >> it's >>> not possible to review the structure of the open pull requests. >>> >> >> >> I'm not sure I understand this point. There was a design document and an >> extensive discussion on this issue. Do you think we should revisit? Some >> common algorithms for bipartitite graphs that I am aware of is SALSA for >> recommendations and relevance search for anomaly detection. >> >> >> >>> >>> +1 to evaluating performance and promoting Flink! >>> >>> Gelly has two shepherds whereas CEP and ML share one committer. New >>> algorithms in Gelly require new features in the Batch API (Gelly may also >>> start doing streaming, we're cool kids, too) >> >> >> ^^ >> >> >>> so we need to find a process >>> for snuffing ideas early and for the right balance in dependence on core >>> committers' time. For example, reworking the iteration scheduler to allow >>> for intermediate outputs and nested iterations. Can this feature be >>> developed and reviewed within Gelly? >> >> Does it need the blessing of a Stephan >>> or Fabian? I'd like to see contributors and committers less dependent on >>> the core team and more autonomous. >>> >> >> What do you mean >> developed and reviewed >> "within Gelly"? >> This feature would require changes in the batch iterations code and will >> probably need to be proposed and reviewed as a FLIP, so it would need the >> blessing of the community :) >> >> Having someone who is more familiar with this part of the code help is of >> course favorable, but I don't think it's absolutely necessary. >> >> -V. >> >> >>> Greg >>> >>> On Fri, Feb 24, 2017 at 10:39 AM, Vasiliki Kalavri < >>> vasilikikala...@gmail.com> wrote: >>> >>>> Hello squirrels, >>>> >>>> this is a discussion thread to organize the Gelly component development >>> for >>>> release 1.3 and discuss longer-term plans for the library. >>>> >>>> I am hoping that with time-based releases, we can distribute the load >> for >>>> PR reviewing and make better use of our time, and also point >> contributors >>>> to "useful" tickets when they offer to help. >>>> >>>> I'm expecting the outcome of this discussion to be: >>>> >>>> (1) a set of open PRs to review and try merging for 1.3 >>>> (2) a set of open JIRAs to work-on before feature freeze >>>> (3) a set of JIRAs and PRs to reorganize/close >>>> (4) ideas on possible FLIPs >>>> >>>> Here's my initial take on things, i.e. features *I* see as important in >>> the >>>> short-term. Feel free to add/remove/discuss: >>>> >>>> Release 1.3 >>>> ========== >>>> - Bipartite graph support. Initial support has been added, but there >>>> are unreviewed >>>> PRs >>>> <https://github.com/apache/flink/pulls?utf8=%E2%9C%93&q= >>>> is%3Apr%20is%3Aopen%20bipartite%20> >>>> and there is no Scala API yet. It would be nice to organize this >> feature, >>>> decide what functionality we need and what functionality is already >>> covered >>>> by the Graph type and have proper bipartite support for 1.3. >>>> - Driver improvements, i.e. #3294 >>>> <https://github.com/apache/flink/pull/3294> >>>> - Algorithm improvements, #2733 <https://github.com/apache/fli >>> nk/pull/2733 >>>>> >>>> - Affinity Propagation algorithm. This one has been developed using a >>> bulk >>>> iteration plan and needs a review. The PR is #2885 >>>> <https://github.com/apache/flink/pull/2885>. >>>> - Object reuse issues, FLINK-5890, FLINK-5891 >>>> - Vertex-centric iteration improvement, i.e. FLINK-5127 >>>> >>>> >>>> Roadmap >>>> ======== >>>> Regarding longer-term plans, I see the following issues as still being >>>> relevant from the existing roadmap [1]: >>>> - Extending the iteration functionality to support algorithms, more >>> complex >>>> than value-propagation, e.g. with nested loops >>>> - Partitioning methods >>>> - Partition-centric iterations >>>> - Performance evaluation >>>> >>>> These two lists are by no means complete or final and the goal of this >>>> thread is to see what the community is interested in, whether these >>>> features / additions make sense to be worked on, or what features are >>>> missing. >>>> So, please provide your feedback! >>>> >>>> Cheers, >>>> -V. >>>> >>>> [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Gelly >>>> >>> >>
