Re: TestStreamEnvironment: await last flush of processing time-based windows

2017-01-24 Thread Steven Ruppert
ng > seems hard. > > I'm very open for suggestions on this topic, though. :-) > > Cheers, > Aljoscha > > On Mon, 23 Jan 2017 at 23:19 Steven Ruppert wrote: >> >> Hi, >> >> I'm attempting to unit test link with the flink-test-utils support, on

TestStreamEnvironment: await last flush of processing time-based windows

2017-01-23 Thread Steven Ruppert
Hi, I'm attempting to unit test link with the flink-test-utils support, on flink 1.1.4. I've got basic flatMap stuff flowing through just fine, but when running any processing time-based windowing functions, `env.execute()` will return before any values are flushed out of the windows. import org.

Re: Flink streaming with 1+ TB of managed state

2016-11-21 Thread Steven Ruppert
Some responses inline below: On Sat, Nov 19, 2016 at 4:07 PM, Gyula Fóra mailto:gyula.f...@gmail.com>> wrote: Hi Steven, As Robert said some of our jobs have state sizes around a TB or more. We use the RocksDB state backend with some configs tuned to perform well on SSDs (you can get some tips

Flink streaming with 1+ TB of managed state

2016-11-18 Thread Steven Ruppert
Hi, Is anybody currently running flink streaming with north of a terabyte (TB) of managed state? If you are, can you share your experiences wrt hardware, tuning, recovery situations, etc? I'm evaluating flink for a use case I estimate will take around 5TB of state in total, but looking at the act