a
> separate source and union them before feeding them to the actual job?
>
> You don't need to modify much on the analysis job and each source can be
> independently read. WDYT?
>
> On Fri, Apr 24, 2020 at 8:46 AM Kaan Sancak <mailto:kaans...@gmail.com>> wro
uld also be easily offloaded to S3.
>
> Alternatively, if graph generation is rather cheap, you could also try to
> incorporate it directly into the analysis job.
>
> On Wed, Apr 22, 2020 at 2:58 AM Kaan Sancak <mailto:kaans...@gmail.com>> wrote:
> Hi,
>
>
Hi,
I have been running some experiments on large graph data, smallest graph I
have been using is around ~70 billion edges. I have a graph generator, which
generates the graph in parallel and feeds to the running system. However, it
takes a lot of time to read the edges, because even though th
first batch
>
> Graph graphAfterFirstSG =
> graph.runScatterGatherIteration();
>
> Graph secondBatch = ... // get second batch
>
> // Adjust the result of SG iteration with secondBatch
>
> Graph updatedGraph =
> graphAfterFirstSG.union/difference(secondBatch));
>
> upd
lieve this should work.
>
> Cheers,
> Till
>
> On Wed, Apr 15, 2020 at 1:14 AM Kaan Sancak <mailto:kaans...@gmail.com>> wrote:
> Thanks for the useful information! It seems like a good and fun idea to
> experiment. I will definitely give it a try.
>
> I
nimal code example
which is able to reproduce the problem. So if you could provide us with
this, that would allow us to look into it.
Cheers,
Till
On Wed, Apr 15, 2020 at 6:59 PM Kaan Sancak wrote:
> Thanks that is working now!
>
> I have one last question.
>
> Goin one step furth
> graphAfterFirstSG.union/difference(secondBatch));
>
> updatedGraph.runScatterGatherIteration();
>
> Then I believe this should work.
>
> Cheers,
> Till
>
> On Wed, Apr 15, 2020 at 1:14 AM Kaan Sancak <mailto:kaans...@gmail.com>> wrote:
> Thanks for the useful information! It seems like a goo
Thanks for the useful information! It seems like a good and fun idea to
experiment. I will definitely give it a try.
I have a very close upcoming deadline and I have already implemented the
Scatter-Gather iteration algorithm.
I have another question on whether we can chain Scatter-Gather or
Ve
ynamic algorithm using Flink Gelly. However, I
couldn’t find any examples. Most of the examples are performed on static
graphs. What I need is an incremental use-case/example. I would be grateful, if
you can point out some examples, or any help is appreciated.
Best
Kaan Sancak
PhD Student
Georgia Tech
