Re: Minicluster Flink tests, checkpoints and inprogress part files

2021-02-06 Thread Dan Hill
I changed the test to use ExecutionMode.BATCH in v1.11 and it still doesn't work. How did devs write minicluster tests before for similar code? Did they not? On Sat, Feb 6, 2021 at 5:38 PM Dan Hill wrote: > Ah looks like I need to use 1.12 for this. I'm still on 1.11. > > On Fri, Feb 5, 2021,

Re: Minicluster Flink tests, checkpoints and inprogress part files

2021-02-06 Thread Dan Hill
Ah looks like I need to use 1.12 for this. I'm still on 1.11. On Fri, Feb 5, 2021, 08:37 Dan Hill wrote: > Thanks Aljoscha! > > On Fri, Feb 5, 2021 at 1:48 AM Aljoscha Krettek > wrote: > >> Hi Dan, >> >> I'm afraid this is not easily possible using the DataStream API in >> STREAMING execution

Sliding Window Count: Tricky Edge Case / Count Zero Problem

2021-02-06 Thread Jan Brusch
Hi, I was recently working on a problem where we wanted to implement a simple count on a sliding window, e.g. "how many messages of a certain type were emitted by a certain type of sensor in the last n minutes". Which sounds simple enough in theory: messageStream     .keyBy(//EmitterType + Me

Re: Question about Scala Case Class and List in Flink

2021-02-06 Thread Xavier
Thanks for ur suggestions! On Fri, Feb 5, 2021 at 11:16 PM Timo Walther wrote: > Dealing with types is not always easy in Flink. If you have further > issues, it might make sense to just pass them explicitly. We list all > types in: > > org.apache.flink.api.common.typeinfo.Types > > org.apache.f

Re: Dynamic StreamingFileSink

2021-02-06 Thread Sidney Feiner
If anybody is interested, I've implemented a StreamingFileSink with dynamic paths: https://github.com/sidfeiner/DynamicPathFileSink Sidney Feiner / Data Platform Developer M: +972.528197720 / Skype: sidney.feiner.startapp [emailsignature] From: Rafi Aroch Se