Re: passing variable to filter function

Looking at the program on Pastebin, there are some things that look not right. I would be surprised if this program executes at all. In particular, you are referring to outside distributed data sets inside the filter function. You are calling collect() in every filter function, which actually trig

Job Statistics

Hello, Is it possible to view job statistics after it finished to execute directly in the command line? If so, could you please explain how? I could not find any mentions about this in the docs. I also tried to set the logs to debug mode, but no other information was presented. Thank you! Regard

Re: sorting groups

Got it, I solved the problem changing the grouping function: instead of group by r._1 (int the topk case) now I group using a function that return r._1 thus both, group and sort, have a function as parameter is there a reason why it works in this way? probably you should make it very clear in the

Re: Memory in local setting

On 17 Jun 2015, at 10:10, Sebastian wrote: > Hi Ufuk, > > Can I configure this when running locally in the IDE or do I have to install > Flink for that? Yes. org.apache.flink.configuration.Configuration conf = new Configuration(); conf.setDouble(ConfigConstants.TASK_MANAGER_MEMORY_FRACTION_K

Re: Memory in local setting

Hi Ufuk, Can I configure this when running locally in the IDE or do I have to install Flink for that? Best, Sebastian On 17.06.2015 09:34, Ufuk Celebi wrote: Hey Sebastian, with "taskmanager.memory.fraction" you can give more memory to the Flink runtime. Current default is to give 70% to F

Re: Memory in local setting

On 17 Jun 2015, at 09:35, Mihail Vieru wrote: > Hi, > > I had the same problem and setting the solution set to unmanaged helped: > > VertexCentricConfiguration parameters = new VertexCentricConfiguration(); > parameters.setSolutionSetUnmanagedMemory(false); > > runVertexCentricIteration(...,

Re: Flink Streaming State Management

Hi Gyula, thank you. Best Regards, Hilmi Am 17.06.2015 um 09:44 schrieb Gyula Fóra: Hey Hilmi, Flink currently supports user defined state through the Checkpointed interface. Using this interface the user can define what state should the system be aware of when doing snapshots for fault tole

Re: Flink Streaming State Management

Hi Matthias, great! Thank you. Best Regards, Hilmi Am 17.06.2015 um 09:38 schrieb Matthias J. Sax: Hi Hilmi, currently, this is not supported. However, state management is already work in progress and should be available soon. See https://github.com/apache/flink/pull/747 -Matthias On 06/17/2

Re: Flink Streaming State Management

Hey Hilmi, Flink currently supports user defined state through the Checkpointed interface. Using this interface the user can define what state should the system be aware of when doing snapshots for fault tolerance. The state returned in the snapshotState method will be checkpointed (and restored u

Re: Flink Streaming State Management

Hi Hilmi, currently, this is not supported. However, state management is already work in progress and should be available soon. See https://github.com/apache/flink/pull/747 -Matthias On 06/17/2015 09:36 AM, Hilmi Yildirim wrote: > Hi, > does Flink Streaming support state management? For example,

Re: Memory in local setting

Hi, look at slide 35 for more details about memory configuration: http://www.slideshare.net/robertmetzger1/apache-flink-hands-on -Matthias On 06/17/2015 09:29 AM, Chiwan Park wrote: > Hi. > > You can increase the memory given to Flink by increasing JVM Heap memory in > local. > If you are usi

Flink Streaming State Management

Hi, does Flink Streaming support state management? For example, I have a state which will be used inside the streaming operations but the state can be updated. For example: stream.map( use state for operation).updateState(update state). Best Regards, Hilmi -- -- Hilmi Yildirim Software D

Re: Memory in local setting

Hey Sebastian, with "taskmanager.memory.fraction" you can give more memory to the Flink runtime. Current default is to give 70% to Flink and leave 30% for the user code. taskmanager.memory.fraction: 0.9 will increase this to 90%. Does this help? [1] http://ci.apache.org/projects/flink/flin

Re: Memory in local setting

Hi, I had the same problem and setting the solution set to unmanaged helped: VertexCentricConfiguration parameters = new VertexCentricConfiguration(); parameters.setSolutionSetUnmanagedMemory(false); runVertexCentricIteration(..., parameters); Best, Mihail On 17.06.2015 07:01, Sebastian wrote

Re: Memory in local setting

Hi. You can increase the memory given to Flink by increasing JVM Heap memory in local. If you are using Eclipse as IDE, add “-Xmx” option in run configuration. [1]. Although you are using IntelliJ IDEA as IDE, you can increase JVM Heap using the same way. [2] [1] http://help.eclipse.org/luna/