Connected Streams - Controlling Order of arrival on the two streams

Hi, I am using connected streams to send rules coded as JavaScript functions on one stream and event data on another stream. They are both keyed by the device id. The rules are cached in the co-map operation until another rule arrives to override existing rule. Is there a way to ensure that the r

Window function - iterator data

When you are using a window function the docs: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/windows.html#windowfunction---the-generic-case state that A WindowFunction gets an Iterable containing all the elements of the window being processed If the input data strea

Re: Flink : CEP processing

In one of the earlier thread Till explained this to me ( http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/CEP-and-Within-Clause-td8159.html ) 1. Within does not use time windows. It sort of uses session windows where the session begins when the first event of the pattern is iden

Re: Flink : CEP processing

Hi Sameer: If we use a within window for event series -  1. Does it interfere with the default time windows ?2. How does it affect snapshotting ?  3. If the window is too large are the events stored in a "processor" for the window to expire ?4. Are there any other know limitations and best pract

Re: Flink : CEP processing

In that case you need to get them into one stream somehow (keyBy a dummy value for example). There is always some logical key to keyBy on when data is arriving from multiple sources (ex some portion of the time stamp). You are looking for patterns within something (events happening around the s

Re: Flink : CEP processing

Thanks Sameer. So does that mean that if the events keys are not same we cannot use the CEP pattern match ?  What if events are coming from different sources and need to be correlated ? Mans On Tuesday, August 9, 2016 9:40 AM, Sameer W wrote: Hi, You will need to use keyBy operation fi

Re: Flink : CEP processing

Hi, You will need to use keyBy operation first to get all the events you need monitored in a pattern on the same node. Only then can you apply Pattern because it depends on the order of the events (first, next, followed by). I even had to make sure that the events were correctly sorted by timestam

Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat

I've started a vote for 1.1.1 containing hopefully fixed artifacts. If you have any spare time, would you mind checking whether it fixes your problem? The artifacts are here: http://home.apache.org/~uce/flink-1.1.1-rc1/ You would have to add the following repository to your Maven project and upda

Re: Flink 1.1.0 : Hadoop 1/2 compatibility mismatch

I've started a vote for 1.1.1 containing hopefully fixed artifacts. If you have any spare time, would you mind checking whether it fixes your problem? The artifacts are here: http://home.apache.org/~uce/flink-1.1.1-rc1/ You would have to add the following repository to your Maven project and upda

Flink : CEP processing

Hey Folks: I have a question about CEP processing in Flink - How does flink processing work when we have multiple partitions in which the events used in the pattern sequence might be scattered across multiple partitions on multiple nodes ? Thanks for your insight. Mans

Re: Release notes 1.1.0?

Oh sorry missed that part, no, Im not explicitly set that. > On 09 Aug 2016, at 15:29, Aljoscha Krettek wrote: > > Hi, > are you setting a StreamTimeCharacteristic, i.e. > env.setStreamTimeCharacteristic? > > Cheers, > Aljoscha > > On Tue, 9 Aug 2016 at 14:52 Andrew Ge Wu

Re: Release notes 1.1.0?

Hi, are you setting a StreamTimeCharacteristic, i.e. env.setStreamTimeCharacteristic? Cheers, Aljoscha On Tue, 9 Aug 2016 at 14:52 Andrew Ge Wu wrote: > Hi Aljoscha > > > Plan attached, there are split streams and union operations around, but > here is how windows are created > > Confidentialit

Re: Release notes 1.1.0?

Hi, could you maybe post how exactly you specify the window? Also, did you set a "stream time characteristic", for example EventTime? That could help us pinpoint the problem. Cheers, Aljoscha On Tue, 9 Aug 2016 at 12:42 Andrew Ge Wu wrote: > I rolled back to 1.0.3 > If I understand this correc

Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat

This is a problem with the Maven artifacts of 1.1.0 :-( I've added a warning to the release note and will start a emergency vote for 1.1.1 which only updates the Maven artifacts. On Tue, Aug 9, 2016 at 1:20 PM, LINZ, Arnaud wrote: > Okay, > > That would also solve my issue. > > Greetings, > > Arn

Re: Flink 1.1.0 : Hadoop 1/2 compatibility mismatch

As noted in the other thread, this is a problem with the Maven artifacts of 1.1.0 :-( I've added a warning to the release note and will start a emergency vote for 1.1.1 which only updates the Maven artifacts. On Tue, Aug 9, 2016 at 9:45 AM, LINZ, Arnaud wrote: > Hello, > > > > I’ve switched to 1.

RE: Classloader issue using AvroParquetInputFormat via HadoopInputFormat

Okay, That would also solve my issue. Greetings, Arnaud De : Stephan Ewen [mailto:se...@apache.org] Envoyé : mardi 9 août 2016 12:41 À : user@flink.apache.org Objet : Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat Hi Shannon! It seams that the something in the maven dep

Re: Release notes 1.1.0?

I rolled back to 1.0.3 If I understand this correctly, the peak when topology starts is because it is trying to fill all the buffers, but I can not see that in 1.1.0. > On 09 Aug 2016, at 12:10, Robert Metzger wrote: > > Which source are you using? > > On Tue, Aug 9, 2016 at 11:50 AM, Andre

Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat

Hi Shannon! It seams that the something in the maven deployment went wrong with this release. There should be: - *flink-java* (the default, with a transitive dependency to hadoop 2.x for hadoop compatibility features) - *flink-java-hadoop1* (with a transitive dependency for hadoop 1.x fir old

Re: Release notes 1.1.0?

We wrote our own source. I noticed our back pressure changed from ok to high after relance().timeWindowAll(), if there is no obvious change on that, the problem can be in our function after this. > On 09 Aug 2016, at 12:10, Robert Metzger wrote: > > Which source are you using? > > On Tue, Au

Re: Release notes 1.1.0?

Which source are you using? On Tue, Aug 9, 2016 at 11:50 AM, Andrew Ge Wu wrote: > Hi Robert > > > Thanks for the quick reply, I guess I’m one of the early birds. > Yes, it is much slower, I’m not sure why, I copied slaves, masters, > log4j.properties and flink-conf.yaml directly from 1.0.3 > I

Re: Release notes 1.1.0?

Hi Robert Thanks for the quick reply, I guess I’m one of the early birds. Yes, it is much slower, I’m not sure why, I copied slaves, masters, log4j.properties and flink-conf.yaml directly from 1.0.3 I have parallelization 1 on my sources, I can increase that to achieve the same speed, but I’m i

Re: Release notes 1.1.0?

Hi Andrew, here is the release announcement, with a list of all changes: http://flink.apache.org/news/2016/08/08/release-1.1.0.html, http://flink.apache.org/blog/release_1.1.0-changelog.html What does the chart say? Are the results different? is Flink faster or slower now? Regards, Robert On T

Release notes 1.1.0?

Hi, We found out there is a new stable version released: 1.1.0 but we can not find any release note. Do anyone know where to find it? We are experience some change of behavior, I’m not sure if it is related. Thanks Andrew -- Confidentiality Notice: This e-mail transmission may contain con

specify user name when connecting to hdfs

Hi. I’m trying to set external hdfs as state backend. my os user name is ec2-user. hdfs user is hadoop. there is a permission denied exception. I wanna specify hdfs user name. I set hadoop.job.ugi in core-site.xml and HADOOP_USER_NAME on command line. but not works. what shall I do? thanks.

Flink 1.1.0 : Hadoop 1/2 compatibility mismatch

Hello, I’ve switched to 1.1.0, but part of my code doesn’t work any longer. Despite the fact that I have no Hadoop 1 jar in my dependencies (2.7.1 clients & flink-hadoop-compatibility_2.10 1.1.0), I have a weird JobContext version mismatch error, that I was unable to understand. Code is a hive

Re: [ANNOUNCE] Flink 1.1.0 Released

Thanks Ufuk and everybody who contributed to the release! Cheers, Fabian 2016-08-08 20:41 GMT+02:00 Henry Saputra : > Great work all. Great Thanks to Ufuk as RE :) > > On Monday, August 8, 2016, Stephan Ewen wrote: > > > Great work indeed, and big thanks, Ufuk! > > > > On Mon, Aug 8, 2016 at 6: