If your graphs fit in memory (at least after an initial partitioning),
you could use any external library for graph processing within a single
node in a Flink ProcessFunction.
Flink is a general data processor that allows to have arbitrary logic
where user code is allowed.
Regards,
Timo
On
Good point what is the better option for graph processing with flink.. any
suggestions
On Friday, September 10, 2021, 04:52:30 PM GMT+5:30, Martijn Visser
wrote:
Hi,
Please keep in mind that Gelly is approaching end-of-life [1]
Regards,
Martijn
[1] https://flink.apache.org/roadmap.htm
Hi,
Please keep in mind that Gelly is approaching end-of-life [1]
Regards,
Martijn
[1] https://flink.apache.org/roadmap.html
On Fri, 10 Sept 2021 at 09:09, Dipanjan Mazumder wrote:
> Hi Jing,
>
> Thanks for the input another question i had was can Gelly be used for
> processing the graph
Hi Dipanjan,
Gelly is built on top of the DataSet API which is a batch-only API that
is slowly phasing out.
It is not possible to connect a DataStream API program with a DataSet
API program unless you go through a connector such as CSV in between.
Regards,
Timo
On 10.09.21 09:09, Dipanjan
Hi Jing,
Thanks for the input another question i had was can Gelly be used for
processing the graph that flink receives through kafka and then using Gelly i
decompose the graph into its nodes and edges and then process them individually
through substreams and then write the final output of
Hi Dipanjan,
Base your description, I think Flink could handle this user case.
Don't worry that Flink can't handle this kind of data scale because Flink
is a distributed engine. As long as the problem of data skew is carefully
avoided, the input throughput can be handled through appropriate resourc
thanks Stephan,
yes, you got usecase right
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usecase-for-Flink-tp4076p4092.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
If I understand you correctly, you want to write something like:
--
[cassandra]
^
|
V
(even
