Re: PageRank - 4x slower then Spark?!

Does someone has a current performance test based on PageRank or an idea why Flink lost the comparison? > Am 18.08.2017 um 19:51 schrieb Kaepke, Marc : > > Hi everyone, > > I compared Flink and Spark by using PageRank. I guessed Flink will beat Spark > or have the same leve

PageRank - 4x slower then Spark?!

Hi everyone, I compared Flink and Spark by using PageRank. I guessed Flink will beat Spark or have the same level. But Spark is up to 4x faster then Flink. I hope I did a mistake. So please help me to improve the performance of my cluster and config. The cluster has 4 computers: One JobManager

PageRank iteration

Hi everyone, does PageRank use bulk or delta iteration? I mean the implementation of PageRank of the package: package org.apache.flink.graph.library.link_analysis; Thanks. Best, Marc

Re: Standalone cluster - taskmanager settings ignored

t to the > restarted JobManager. > > Greg > > >> On Aug 11, 2017, at 11:25 AM, Kaepke, Marc >> wrote: >> >> Hi, >> >> I have a cluster of 4 dedicated machines (no VMs). My previous config was: 1 >> master and 3 slaves. Each machine pro

Re: Standalone cluster - taskmanager settings ignored

cluster included additionally slave5. My current cluster has not slave5. But the WebUI shows 4 TM -> master, slave1, slave3 and slave5 Am 11.08.2017 um 17:25 schrieb Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>>: Hi, I have a cluster of 4 dedicated machines (no VMs). My previous c

Standalone cluster - taskmanager settings ignored

Hi, I have a cluster of 4 dedicated machines (no VMs). My previous config was: 1 master and 3 slaves. Each machine provides a task- or jobmanager. Now I want to reduce my cluster and have 1 master and 3 slaves, but one machine provides a jobmanager and one task manager in parallel. I changed al

Re: FileNotFound Exception in Cluster Standalone

it is already on HDFS then you need to provide an HDFS URL to the file. On 5. Aug 2017, at 14:27, Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>> wrote: Hi there, my really small test job reads an external file and print the input to console. Execute it as standalone with a local cluster, e

FileNotFound Exception in Cluster Standalone

Hi there, my really small test job reads an external file and print the input to console. Execute it as standalone with a local cluster, everything is fine. If I execute the same job as standalone with 1 job manager und 1 task manager, I get an FileNotFound Exception. As a real distributed cluste

k8s FileNotFoundException

Hi everyone, I always get an FileNotFoundException by following the kubernetes setup guide [1]. I moved my jar and my input file onto the job manager pod After that I join the job manager pod by using: kubectl exec -it - - /bin/bash With ls I can see both files. WordCount-example worked well.

Re: Gelly PageRank implementations in 1.2 to 1.3

implementation. The last one (without any graph model) works fine. Do you have any idea what I doing wrong? Marc Am 24.07.2017 um 20:56 schrieb Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>>: Thanks for your explanation. The vertex-centric, sg and gsa PageRank need a Double as vertex va

Re: Gelly PageRank implementations in 1.2 to 1.3

at 6:38 AM, Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>> wrote: Hi Greg, I do an evaluation between Gelly and GraphX (Spark). Both frameworks implement PageRank and Gelly provides a lot of variants (*thumbs up*). During a really small initial test I get for the vertex-centric, scatte

Re: Gelly PageRank implementations in 1.2 to 1.3

eshold)); Please let us know of any issues or additional questions! Greg On Jul 22, 2017, at 4:33 PM, Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>> wrote: Hi there, why was the PageRank version (which implements the GraphAlgorithm interface) removed in 1.3? How can I use the n

Gelly PageRank implementations in 1.2 to 1.3

Hi there, why was the PageRank version (which implements the GraphAlgorithm interface) removed in 1.3? How can I use the new PageRank implementation in 1.3.x? Why PageRank doesn’t use the graph processing models (vertex-centric, sg or gsa) anymore? Thanks! Bests, marc

Gelly - bipartite graph runs vertex-centric

Hi, does Gelly provides a vertex-centric iteration on a bipartite graph? A bipartite graph is using BipartiteEdges and vertex-centric supports regular edges only. Thanks! Best, Marc

Re: coGroup exception or something else in Gelly job

Hi Greg if you have an idea, I'm still interested. In case you didn't, please give me a feedback too. Best, Marc Sent from my iPhone On 15. Jun 2017, at 15:19, Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>> wrote: Hi Greg, I wanna ask if there was any news about the i

Re: coGroup exception or something else in Gelly job

Hi Greg, I wanna ask if there was any news about the implementation or opportunities? Thanks and best regards, Marc Am 12.06.2017 um 19:28 schrieb Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>>: I’m working on an implementation of SemiClustering [1]. I used two graph models (Preg

Re: coGroup exception or something else in Gelly job

ription of the algorithm you are working to implement? On Mon, Jun 12, 2017 at 5:47 AM, Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>> wrote: It seems Flink used a different exception graph outside of my IDE (intellij) The job anatomy is: load data from csv and build an initial graph

Re: coGroup exception or something else in Gelly job

Best and thanks Marc Am 10.06.2017 um 00:49 schrieb Greg Hogan mailto:c...@greghogan.com>>: Have you looked at org.apache.flink.gelly.GraphExtension.CustomVertexValue.createInitSemiCluster(CustomVertexValue.java:51)? On Jun 9, 2017, at 4:53 PM, Kaepke, Marc mailto:marc.kae...@haw-ham

coGroup exception or something else in Gelly job

Hi everyone, I don’t have any exceptions if I execute my Gelly job in my IDE (local) directly. The next step is an execution with a real kubernetes cluster (1 JobManager and 3 TaskManager on dedicated machines). The word count example is running without exceptions. My Gelly job throws following

Re: Flink on kubernetes -> shell deployment

ve access to the > Jobmanager in general and can submit jobs from your command line by passing > > flink run --jobmanager ... > > > I've looped in Patrick in case I am missing something kubernetes-specific > here. > > > Nico > > On Wednesday, 7

Flink on kubernetes -> shell deployment

Hi together, I run Flink on kubernetes and I’m able to can access the web dashboard inside k8n. My question is: How do I get access to the shell? I don’t want to deploy my job on the UI. Thanks! Best, Marc

Memory ran out. Compaction failed. - Exception

Hi, I tried to run an vertex-centric and a gather scatter iteration in Gelly and receive an exception like Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$

Re: Gelly - generics with custom vertex value

t work out of the box with Kryo / Flink. Try registering a custom serializer for it... On Wed, May 10, 2017 at 4:16 PM, Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>> wrote: Hi, a part of my bachelor thesis is an implementation of the Semi-Clustering algorithm [1]. I’m using the Sc

Gelly - generics with custom vertex value

Hi, a part of my bachelor thesis is an implementation of the Semi-Clustering algorithm [1]. I’m using the Scatter-Gather-Iteration. Each vertex has to know its neighbors and the edge-value between of that. Because Gelly’s vertex doesn’t provide both information, I wrote an CustomVertexValue cla

Re: RuntimeException: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: null

Hi, did some had an answer or solution? Best Marc Am 05.05.2017 um 20:05 schrieb Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>>: Hi everyone, what does mean that following exception, if I run my gelly program? Exception in thread

RuntimeException: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: null

Hi everyone, what does mean that following exception, if I run my gelly program? Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp

Re: The implementation of the CoGroupFunction is not serializable

Hi Greg, thanks! Should all inner classes be static? Best, Marc Am 01.05.2017 um 00:21 schrieb Greg Hogan mailto:c...@greghogan.com>>: Hi Marc, These errors are usually an inner class which should be a static. Greg On Apr 30, 2017, at 2:13 PM, Kaepke, Marc mailto:marc.kae.

The implementation of the CoGroupFunction is not serializable

Hi guys, I have no errors if I run a Graph.groupReduceOnEdges() in main(String[ ] args). But the same function in its own class throws an exception. DataSet>> vertexDataSet = graph.groupReduceOnEdges(new EdgesFunctionWithVertexValue>>() { @Override public void iterateEdges(Vertex vertex

Re: gelly scatter/gather

Hi Alieh, I can't solve your problem yet. But I work with gelly and Scatter/Gather and later GSA too. Could you publish or show me your algorithm? Best from Hamburg Marc Sent from my iPhone > On 28. Apr 2017, at 13:58, Alieh wrote: > > Hi all > > I have an iterative algorithm implemented us

Re: Graph iteration with triplets or access to edges

to summarize my question: Does Flink or Gelly offers an access to the edges of a single vertex? Or: I need a VertexTriplet and not an EdgeTriplet (graph.getTriplets()) Thanks! Best, Marc > Am 27.04.2017 um 20:20 schrieb Kaepke, Marc : > > Hi everyone, > > in Gelly I use the

Graph iteration with triplets or access to edges

Hi everyone, in Gelly I use the Scatter-Gather Iteration to create clusters depends on edge-values. During a superstep each vertex has to sum all edge values of its neighbors. The scatter and gather functions both works on one vertex. How can I get the out and in-going edges of each vertex?

Re: Beginner question - sum multiple edges

().run(networkSumMultiEdges); How can I reduce and combine (1 2 20) and (2 1 10) to one Tuple? Best regards Marc Am 17.04.2017 um 21:47 schrieb Kaepke, Marc mailto:marc.kae...@haw-hamburg.de>>: Hi Gábor, thanks a lot Best, Marc Am 17.04.2017 um 20:32 schrieb Gábor Gévay mailto:gga...@gmail.c

Re: Beginner question - sum multiple edges

iling list. > > > > > On Mon, Apr 17, 2017 at 5:46 PM, Kaepke, Marc > wrote: >> Hi, >> >> how can I sum and reduce multiple edges in my entire graph? >> >> e.g. my input graph looks like (source-ID, target-ID, value): >> (1, 2, 30) >> (1, 2, 10) >> (2, 1, 55) >> >> And I need: >> (1, 2, 40) >> (2, 1, 55) >> >> >> Thanks! >> Marc

Beginner question - sum multiple edges

Hi, how can I sum and reduce multiple edges in my entire graph? e.g. my input graph looks like (source-ID, target-ID, value): (1, 2, 30) (1, 2, 10) (2, 1, 55) And I need: (1, 2, 40) (2, 1, 55) Thanks! Marc

Re: has insufficient permissions to access it - Error

he file path doesn't look quite right, unless you really have such an > (absolute!) file path. > > Nico > > On Saturday, 8 April 2017 17:41:28 CEST Kaepke, Marc wrote: >> Hi, >> >> if I run my small Gelly application on IntelliJ (macOS and Ubun

has insufficient permissions to access it - Error

Hi, if I run my small Gelly application on IntelliJ (macOS and Ubuntu as well) I have this error: Caused by: java.io.FileNotFoundException: File /org/apache/flink/gelly/gellyTutorials/out.munmun_twitter_social does not exist or the user running Flink ('marc') has insufficient permissions to ac

Gelly Graph Processing - descriptive buzzwords

Hi guys, I need some buzzwords, which describes Gelly. Something like that: * Delta Iteration (native iteration) -> as a base graph programming models * vertex-centric, scatter-gather and GSA -> supported graph programming models * DataSet API -> connection and basement between Flin

Gelly - which partitioning

Hi guys, I can’t found on web which graph partitioning are supported by Gelly. During my search I found this link. But the ticket is still open. https://cwiki.apache.org/confluence/display/FLINK/Flink+Gelly Thanks for help! Best Marc

First Program with WordCount - Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/common/functions/FlatMapFunction

Hi guys, I followed this guide (https://ci.apache.org/projects/flink/flink-docs-release-1.2/quickstart/java_api_quickstart.html), but I get an Exception if I run WordCount /usr/lib/jvm/java-8-oracle/bin/java -Didea.launcher.port=7536 -Didea.launcher.bin.path=/home/marc/Programs/idea-IC-162.203