Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-18 Thread Kaan Sancak
Thanks that worked! I wonder what will be the performance difference if I implement this with Stateful Functions. Does anyone knows recent works/papers on similar approach? Best Kaan > On 16 Apr 2020, at 10:00, Yun Gao wrote: > >  > Hi Kaan, > >For the first issue, I think the two impl

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-16 Thread Yun Gao
Hi Kaan, For the first issue, I think the two implementation should have difference and the first should be slower, but I think which one to use should be depend on your algorithm if it could compute incrementally only with the changed edges. However, as far as I know I think most graph algo

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-16 Thread Kaan Sancak
If the vertex type is POJO what happens during the union of the graph? Is there a persistent approach, or can we define a function handle such occasions? Would there be a performance difference between two cases: 1) Graph graph = … // From edges list graph = graph.runScatterGath

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-16 Thread Kaan Sancak
Thanks for the reply. Turns out that my serializer was writing one of the fields wrong. I fixed it and everything seems to be working correctly for now. Best Kaan On Apr 16, 2020, at 3:05 AM, Till Rohrmann wrote: Hi Kaan, I'm not entirely sure what's going wrong w/o having a minimal code exa

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-16 Thread Till Rohrmann
Hi Kaan, I'm not entirely sure what's going wrong w/o having a minimal code example which is able to reproduce the problem. So if you could provide us with this, that would allow us to look into it. Cheers, Till On Wed, Apr 15, 2020 at 6:59 PM Kaan Sancak wrote: > Thanks that is working now! >

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-15 Thread Kaan Sancak
Thanks that is working now! I have one last question. Goin one step further, I have changed vertex value type to be a POJO class. The structure is somewhat similar to this, class LocalStorage { Integer id; Long degree; Boolean active; List labels; Map nei

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-15 Thread Till Rohrmann
Hi Kaan, I think what you are proposing is something like this: Graph graph = ... // get first batch Graph graphAfterFirstSG = graph.runScatterGatherIteration(); Graph secondBatch = ... // get second batch // Adjust the result of SG iteration with secondBatch Graph updatedGraph = graphAfterFi

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-14 Thread Kaan Sancak
Thanks for the useful information! It seems like a good and fun idea to experiment. I will definitely give it a try. I have a very close upcoming deadline and I have already implemented the Scatter-Gather iteration algorithm. I have another question on whether we can chain Scatter-Gather or Ve

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-14 Thread Flavio Pompermaier
>From what I see Gelly is not really maintained or used anymore..do you think it could make sense to deprecate it and write a guide (on the documentation) about how to rewrite a Gelly app into a Statefun one? On Tue, Apr 14, 2020 at 5:16 AM Tzu-Li (Gordon) Tai wrote: > Hi, > > As you mentioned,

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-13 Thread Tzu-Li (Gordon) Tai
Hi, As you mentioned, Gelly Graph's are backed by Flink DataSets, and therefore work primarily on static graphs. I don't think it'll be possible to implement incremental algorithms described in your SO question. Have you tried looking at Stateful Functions, a recent new API added to Flink? It sup