Re: Filtering and mapping data after window opertator

Hi Soheil, We can't apply FilterFunction or MapFunction on WindowedStream. It is recommended to do these operations on DataStream, for example, temp.filter().map().keyBy(0).timeWindow(). Best, Hequn On Sat, Jul 14, 2018 at 9:14 PM, Soheil Pourbafrani wrote: > Hi, I'm getting data stream from a

How to customize trigger for Count Time Window

I want to have a time window to trigger data processing in two following condition: 1 - The window has 3 messages 2- Or any number of message (less than 3) is in the window and it reaches a timeout I know someone should extend Trigger class: public static class MyWindowTrigger extends Trigger {

Re: StateMigrationException when switching from TypeInformation.of to createTypeInformation

Apologies for the delay. I've been traveling. On Mon, Jul 9, 2018 at 8:44 AM Till Rohrmann wrote: > could you check whether the `TypeInformation` returned by > `TypeInformation.of(new TypeHint[ConfigState]() {}))` and > `createTypeInformation[ConfigState]` return the same `TypeInformation` > su

Flink Query Optimizer

Hello, I am just wondering, does Flink use Apache Calcite's query optimizer to generate an optimal logical plan for stream queries, or does it have its own independent query optimizer? >From what I observed so far, the Flink's query optimizer only groups operator together without changing the orde

Re: understanding purpose of TextInputFormat

Textinputformat defines the format of the data, it could be also different from text , eg orc, parquet etc > On 14. Jul 2018, at 19:15, chrisr123 wrote: > > I'm building a streaming app that continuously monitors a directory for new > files and I'm confused about why I have to specify a TextInp

understanding purpose of TextInputFormat

I'm building a streaming app that continuously monitors a directory for new files and I'm confused about why I have to specify a TextInputFormat - see source code below. It seems redundant but it is a required parameter. It makes perfect sense to specify the directory I want to monitor, but what

Re: SinkFunction invoke method signature

> That is, whether the context interface is defined as Context or > Context makes no difference, since it is an interface which are always static. I don't think this is the case. Context<> is an inner interface, has a meaning in that scope, does not, so there's a very real difference. When

specifying prefix for print(), printToErr() ?

The documentation states that there is a way to specify a prefix msg to distinguish between different calls to print() (see below), but I have not found a way to do this? Can anyone show me how I would code this? What I'd like to do conceptually, and have the prefix msg show up in the output so

reduce a data stream to other type

Hi, I have a keyed datastream in the type of Tuple2. I want to reduce it and merge all of the byte[] for a key. (the first filed (Long) is the key). So I need reduce function return the type Tuple2>, but reduce function didn't allow that! How can I do such job in Flink?

Filtering and mapping data after window opertator

Hi, I'm getting data stream from a source and after gathering data in a time window I want to do some operation like filtering and mapping on windowed data, but the output of time window operation just allow reduce, aggregate or ... function and after that, I want to apply functions like filter or

Multi-tenancy environment with mutual auth

All, We are running into a blocking production deployment issue. It looks like Flink inter-communications doesnt support SSL mutual auth. Any plans/ways to support it? We are going to have to create DMZ for each tenant without that, not preferable of course. - Ashish

Re: SinkFunction invoke method signature

The variables T and IN aren't related to each other. That is, whether the context interface is defined as Context or Context makes no difference, since it is an interface which are always static. At runtime, the context given to the function should be of type Context, but I don't know why the