Thread 'SortMerger spilling thread' terminated due to an exception: No space left on device

Hello, I have a problem for which I hope someone will be able to give a hint. I am running the Flink *standalone* cluster with 2 Docker containers (1 TaskManager and 1 JobManager) using 1 TaskManager with 30 GB of RAM. The dataset is a large one: SNAP Friendster, which has around 1800 M edges. ht

Re: Query regarding state backend for Custom Map Function

Thanks a lot Stefan. I got what I was looking for. Is the MapState functionality coming as a part of the 1.2 release? From: Stefan Richter Reply-To: "user@flink.apache.org" Date: Thursday, December 1, 2016 at 2:53 AM To: "user@flink.apache.org

AW: Time To Live-Setting for State/StateDescriptor

Hi Aljoscha, thanks for your reply. It seems like 1.2 will be a very awesome release. I’m looking forward to it :) Best Regards, Julian Von: Aljoscha Krettek [mailto:aljos...@apache.org] Gesendet: Donnerstag, 1. Dezember 2016 12:41 An: user@flink.apache.org Betreff: Re: Time To Live-Setting for

Re: About delta awareness caches

I'm not aware of how windows work in Storm. If you could maybe give some details on your use case we could figure out together how that would map to Flink windows. Cheers, Aljoscha On Tue, 29 Nov 2016 at 15:47 xingcan wrote: > Hi all, > > Recently I tried to transfer some old applications from

Re: Time To Live-Setting for State/StateDescriptor

Hi Julian, we're aware of the issue (https://issues.apache.org/jira/browse/FLINK-3946) but unfortunately it's not implemented yet. I recently added ProcessFunction which is somewhat similar to a flatmap but also allows setting timers. This can be used to set a TTL timer and then call clear() on the

Re: Query regarding state backend for Custom Map Function

Hi, using the ValueState and RocksDB to store a map inside the value state means that you will have a different map for each key, which is automatically swapped on a per record basis, depending on the record’s key. If you are using a map and Checkpointed, there is only one map and your code is

Re: ContinuousFileMonitoringFunction - deleting file after processing

Hi Maciek, The first point seems interesting and we should definitely look into that, also for other filesystems e.g. HDFS. It would be nice if we could together find a more “one-size-fits-all” solution. Because local fs rounds up to a second but other filesystems may have different strategies.

Query regarding state backend for Custom Map Function

Hi Everyone, I am trying to understand the Working With State feature page of the Flink documentation. My question is in case I am using a ValueState in my CustomMap class to store my states with the RocksDb as my state backend then it is clear that every state value is stored in RocksDb. Now i