Re: Creating a representative streaming workload

Hi, Sorry for the ultra-late reply. Another real-life streaming scenario would be the one I am working on: - collecting data from telecom cells in real-time - and filtering out certain information or enriching/correlating (adding additional info based on the parameters received) events - this is

Re: Creating Graphs from DataStream in Flink Gelly

Hi, There is a separate project related to graph streaming. It's called gelly-streaming. And, if you look here: https://github.com/vasia/gelly-streaming/blob/master/src/main/java/org/apache/flink/graph/streaming/GraphStream.java You can find a constructor which creates a graph from a DataStream o

Re: Gelly ran out of memory

Hi Flavio, These kinds of exceptions generally arise from the fact that you ran out of `user` memory. You can try to increase that a bit. In your flink-conf.yaml try adding # The memory fraction allocated system -user taskmanager.memory.fraction: 0.4 This will give 0.6 of the unit of memory to th

Re: extract workers' resources stats

Hi, You can use something like iostat to extract the CPU usage. For instance, I call this script on the JM node: #!/usr/bin/env bash lines=`cat /home/andra.lungu/hostnames.txt | paste -d, -s` echo "${lines}" IFS="," read -ra hostnames <<<"${lines}" for line in "${hostnames[@]}"; do ssh ${line}

Re: Too few memory segments provided exception

I also questioned the vertex-centric approach before. The exact computation does not throw this exception so I guess adapting the approximate version will do the trick [I also suggested improving the algorithm to use less operators offline]. However, the issue still persists. We saw it in Affinity

Re: Too few memory segments provided exception

Hi, I am afraid this is a known issue: http://mail-archives.apache.org/mod_mbox/flink-dev/201503.mbox/%3CCAK5ODX7_-Wxg9pr7CkkkG4CzA+yNCNMvmea5L2i2iZZV=2c...@mail.gmail.com%3E The behavior back then seems to be exactly what Shivani is experiencing at the moment. At that point I remember Fabian sug

Re: Gelly EOFException

elly.. > > On Tue, Jul 14, 2015 at 10:00 PM, Andra Lungu > wrote: > >> Hello, >> >> Sorry for the delay. The bug is not in Gelly, but is, as hinted in the >> exception and as can be seen in the logs, in Flink's Runtime. Mihail may >> actually

Re: Gelly EOFException

s. Thanks! Andra On Tue, Jul 14, 2015 at 2:12 PM, Mihail Vieru wrote: > Hi, > > looks very similar to this bug: > https://issues.apache.org/jira/browse/FLINK-1916 > > Best, > Mihail > > > On 14.07.2015 14:09, Andra Lungu wrote: > > Hi Flavio, > > Could yo

Re: Gelly EOFException

Hi Flavio, Could you also show us a code snippet? On Tue, Jul 14, 2015 at 2:06 PM, Flavio Pompermaier wrote: > Hi to all, > > in my vertex centric iteration I get the following exception, am I doing > something wrong or is it a bug of Gelly? > > starting iteration [1]: CoGroup (Messaging) (6/8

Re: The slot in which the task was scheduled has been killed (probably loss of TaskManager)

me a status update. Thanks a lot! Sorry for the inconvenience! Andra On Tue, Jun 30, 2015 at 10:51 AM, Till Rohrmann wrote: > Do you have the JobManager and TaskManager logs of the corresponding TM, > by any chance? > > On Mon, Jun 29, 2015 at 8:12 PM, Andra Lungu > wrote: >

Re: The slot in which the task was scheduled has been killed (probably loss of TaskManager)

Something similar in flink-0.10-SNAPSHOT: 06/29/2015 10:33:46 CHAIN Join(Join at main(TriangleCount.java:79)) -> Combine (Reduce at main(TriangleCount.java:79))(222/224) switched to FAILED java.lang.Exception: The slot in which the task was executed has been released. Probably loss of TaskMana

Re: Gelly available already?

Hi, >> >> Gelly is not part of any offical flink release. >> You have to use a Snapshot version of Flink if you want to try it out. >> >> >> Sent from my iPhone >> >> On 23.03.2015, at 23:10, Andra Lungu > <mailto:lungu.an...@gmail.com>>

Re: Gelly available already?

Hi Sebastian, For me it works just as described there, with 0.9, but there should be no problem for 0.8.1. Here is an example pom.xml https://github.com/andralungu/gelly-partitioning/blob/first/pom.xml Hope that helps! Andra On Mon, Mar 23, 2015 at 11:02 PM, Sebastian wrote: > Hi, > > Is gelly