Re: How to maintain variable for each map operator

Hi Kurt, Thanks! Your link helps me a lot. I still have some problems after I glance on the document. As you can see from my first email, I tried to implement a mapfunction class in flink. I actually have 3 arraylists to be maintain at this map operator. I think the Using managed keyed state

Re: How to maintain variable for each map operator

Hi, I think you can use State to achieve your goal: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html Best, Kurt On Thu, Jul 13, 2017 at 1:14 PM, ZalaCheung wrote: > Hi all, > > I am stuck with a problem. I have a stream, I want keyby it and then do a > map func

How to maintain variable for each map operator

Hi all, I am stuck with a problem. I have a stream, I want keyby it and then do a map function on it. But for each map operator, I want to maintain a variable for it. Is that possible? I tried a naive version on local IntelliJ IDE and it works. But I got nullpointerException while trying to ru

Re: global window trigger

Hi We've have custom operators using global windows and are using event time. How are you specifying event time as the time characteristic? Prashant -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/global-window-trigger-tp14206p14239.html

Re: [POLL] Who still uses Java 7 with Flink ?

+1 for dropping Java 7 2017-07-13 9:34 GMT+08:00 ☼ R Nair (रविशंकर नायर) < ravishankar.n...@gmail.com>: > +1 for dropping Java 1.7. > > On Wed, Jul 12, 2017 at 9:10 PM, Kurt Young wrote: > >> +1 for droppint Java 7, we have been using Java 8 for more than one year >> in Alibaba and everything wo

Re: [POLL] Who still uses Java 7 with Flink ?

+1 for dropping Java 1.7. On Wed, Jul 12, 2017 at 9:10 PM, Kurt Young wrote: > +1 for droppint Java 7, we have been using Java 8 for more than one year > in Alibaba and everything work fine. > > Best, > Kurt > > On Thu, Jul 13, 2017 at 2:53 AM, Bowen Li wrote: > >> +1 for dropping Java 7 >> >>

Re: [POLL] Who still uses Java 7 with Flink ?

+1 for droppint Java 7, we have been using Java 8 for more than one year in Alibaba and everything work fine. Best, Kurt On Thu, Jul 13, 2017 at 2:53 AM, Bowen Li wrote: > +1 for dropping Java 7 > > On Wed, Jul 12, 2017 at 9:04 AM, Gyula Fóra wrote: > > > +1 for dropping 1.7 from me as well. >

Reading static data

What is the best way to read a map of lookup data? This lookup data is like a small short lived data that is available in transformation to do things like filtering, additional augmentation of data etc.

Re: sanity check in production

Hi! Assuming you have some spare compute resources on your cluster (which you should have in a production setting to be safe). I think 2) would be the best option, ideally started from a savepoint of the production job to verify your state logic as well. You could also run the test job on a small

FlinkKafkaConsumer subscribes to partitions in restoredState only.

Hello, We're noticing that FlinkKafkaConsumer subscribes to partitions in restored state only. Thus, partitions which aren't in restored state aren't read. We have to restart the job, for FlinkKafkaConsumer to read from all partitions. Here are the details: Environment: Flink-1.3.0, standalone c

Re: Why would a kafka source checkpoint take so long?

Hi Gyula, I have observed similar issue with FlinkConsumer09 and 010 and posted it to the mailing list as well . This issue is not consistent, however whenever it happens it leads to checkpoints getting failed or taking a long time to complete. Regards, Vinay Patil On Wed, Jul 12, 2017 at 7:00 P

Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

+1 to drop Java 7 support On 12.07.2017 16:43, Stephan Ewen wrote: Hi users! Flink currently maintains backwards compatibility for savepoint formats, which means that savepoints taken with Flink version 1.1.x and 1.2.x can be resumed in Flink 1.3.x We are discussing how many versions back

Re: [POLL] Who still uses Java 7 with Flink ?

+1 for dropping Java 7 On Wed, Jul 12, 2017 at 9:04 AM, Gyula Fóra wrote: > +1 for dropping 1.7 from me as well. > > Gyula > > On Wed, Jul 12, 2017, 17:53 Ted Yu wrote: > > > +1 on dropping support for Java 1.7 > > > > Original message > > From: Robert Metzger > > Date: 7/12/

sanity check in production

Hello everyone, Our team ran into an issue that testing new deployment of flink job is difficult as explained below Goal: When we are deploying new version of a flink job in production. we want to be able to have the job process some test messages and verify the output to make sure that the j

Re: System properties when submitting flink job to YARN Session

Hi Aljoscha, I am still using Beam on Flink. I have one yarn session running multiple streaming jobs. The application jar contains some environment specific run time properties( like ip addresses, rest api end points etc). This adds overhead in my usecase as we have to deploy this in multiple

Flink Mesos Outstanding Offers - trouble launching task managers

Hi We’re running Flink 1.3.1 on Mesos. >From time-to-time, the Flink app master seems to have trouble with Mesos offers… At such time, it obviously ends up not launching the requested task managers (mesos.initial-tasks) and we’ve noticed situations where it launches zero tasks. During such times

Re: Getting Errors when using keyby()

Hi Sridhar, Your class is missing default constructor(without arguments) thus it is not a valid POJO in Flink. You can check the requirements for POJO in link here: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/api_concepts.html#pojos > On 12 Jul 2017, at 19:54, Sridhar Chel

Re: Fink: KafkaProducer Data Loss

Hey guys, any update on this? If needed I can attach our code. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14224.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabb

Getting Errors when using keyby()

I have a DataStream on which I am applying a CEP pattern and grouping the results using keyby(). The DataStream Object is a pojo : public class DataStreamObject { private String field1; private String field2; public DataStreamObject(String field1, String field2) { this.field1

Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

Hi users! I can't upgrade from 1.2.x to 1.3.x without code adaptations. Upgrading from 1.(0|1).x to 1.2.x produces configuration mess. Maybe you can discuss changing the release plan, speed it up a little bit and use the major.minor.patch versions as advantage to organize the release process more

Re: [POLL] Who still uses Java 7 with Flink ?

+1 for dropping 1.7 from me as well. Gyula On Wed, Jul 12, 2017, 17:53 Ted Yu wrote: > +1 on dropping support for Java 1.7 > > Original message > From: Robert Metzger > Date: 7/12/17 8:36 AM (GMT-08:00) > To: d...@flink.apache.org > Cc: user > Subject: Re: [POLL] Who still u

Re: [POLL] Who still uses Java 7 with Flink ?

+1 on dropping support for Java 1.7 Original message From: Robert Metzger Date: 7/12/17 8:36 AM (GMT-08:00) To: d...@flink.apache.org Cc: user Subject: Re: [POLL] Who still uses Java 7 with Flink ? +1 to drop Java 7 support I believe that we can move to Java 8 for the argume

Re: [POLL] Who still uses Java 7 with Flink ?

+1 for dropping Java 7. > Am 12.07.2017 um 17:36 schrieb Robert Metzger : > > +1 to drop Java 7 support > > I believe that we can move to Java 8 for the argument you've stated. > ElasticSearch 5, Spark 2.2 require Java 8 already, Hadoop 3.0.0 will require > it as well. > > On Wed, Jul 12, 201

Re: [POLL] Who still uses Java 7 with Flink ?

+1 to drop Java 7 support I believe that we can move to Java 8 for the argument you've stated. ElasticSearch 5, Spark 2.2 require Java 8 already, Hadoop 3.0.0 will require it as well. On Wed, Jul 12, 2017 at 4:02 PM, Driesprong, Fokko wrote: > Hi, > > I would be in favor of dropping Java 7 as

[POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

Hi users! Flink currently maintains backwards compatibility for savepoint formats, which means that savepoints taken with Flink version 1.1.x and 1.2.x can be resumed in Flink 1.3.x We are discussing how many versions back to support. The proposition is the following: * Suggestion: Flink 1.4.0

delta iteration

Hello all, I need iteration number in delta iteration (or any kind of counter). Is there anyway to implement or extract it? Cheers, Alieh

Re: delta iteration

Hi Alieh, From a rich function call getIterationRuntimeContext().getSuperstepNumber() Greg > On Jul 12, 2017, at 9:56 AM, Alieh wrote: > > Hello all, > > I need iteration number in delta iteration (or any kind of counter). Is there > anyway to implement or extract it? > > Cheers, > > Alie

Use processing time and time window flink acts like batch mode.

I using processing time and the data source comes from kafka. My code is like follows: streams.keyBy(XXX) .timeWindow(Time.seconds(30)) .apply(myClassObject) Log in myClassObject is like: 2017-07-12 20:00:00, 2017-07-12 20:00:00, 2017-07-12 20:00:30, 2017

delta iteration

Hello all, I need iteration number in delta iteration (or any kind of counter). Is there anyway to implement or extract it? Cheers, Alieh

delta iteration

Hello all, I need iteration number in delta iteration (or any kind of counter). Is there anyway to implement or extract it? Cheers, Alieh

Re: [POLL] Who still uses Java 7 with Flink ?

Bumping this thread again. There are several strong points for dropping Java 7 support, apart from the fact that it is not maintained - We could really use the Java 8 default methods feature in interfaces to evolve the API without breaking backwards compatibility - Easier build setup for Sca

Re: Why would a kafka source checkpoint take so long?

I have added logging that will help determine this as well, next time this happens I will post the results. (Although there doesnt seem to be high backpressure) Thanks for the tips, Gyula Stephan Ewen ezt írta (időpont: 2017. júl. 12., Sze, 15:27): > Can it be that the checkpoint thread is wait

StreamTableSource

Hi, I have two streams coming from kafka which I want to map into table environment. Because they are not pojo or tuple I will have to map them using, for instance, Types.ROW_NAMED. Can i use StreamTableSource and call registerTableSource or should I use the same code inside getDataStream but call

Re: Why would a kafka source checkpoint take so long?

Can it be that the checkpoint thread is waiting to grab the lock, which is held by the chain under backpressure? On Wed, Jul 12, 2017 at 12:23 PM, Gyula Fóra wrote: > Yes thats definitely what I am about to do next but just thought maybe > someone has seen this before. > > Will post info next ti

Re: System properties when submitting flink job to YARN Session

Can I The specific the jars that I depend on when I submit my project? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/System-properties-when-submitting-flink-job-to-YARN-Session-tp14158p14207.html Sent from the Apache Flink User Mailing List

global window trigger

for a global window with a custom event time trigger that fires every 1 minute and then apply a custom window function to it, the trigger firing seems working but the element collection i get inside of my custom WindowFunction is always the whole inputs from start to end rather than inputs subset

trigger testing

I'm testing with ContinuousEventTimeTrigger with a TumblingWindow. let's say in time frame A, B, C there are 1, 2 and 3 inputs the count result I'd expected was something like ContinuousEventTimeTrigger-> A:1, B:3, C:6 but from the result I get, it seems the inputs haven't been accumulated

Re: Associative operation + windowAll - possible parallelism

Thanks for the suggestions, I will take a look at transform function. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Associative-operation-windowAll-possible-parallelism-tp14187p14204.html Sent from the Apache Flink User Mailing List archiv

Re: Why would a kafka source checkpoint take so long?

Yes thats definitely what I am about to do next but just thought maybe someone has seen this before. Will post info next time it happens. (Not guaranteed to happen soon as it didn't happen for a long time before) Gyula On Wed, Jul 12, 2017, 12:13 Stefan Richter wrote: > Hi, > > could you intro

Re: Why would a kafka source checkpoint take so long?

Hi, could you introduce some logging to figure out from which method call the delay is introduced? Best, Stefan > Am 12.07.2017 um 11:37 schrieb Gyula Fóra : > > Hi, > > We are using the latest 1.3.1 > > Gyula > > Urs Schoenenberger > ezt írta (időpon

Re: Why would a kafka source checkpoint take so long?

Hi, We are using the latest 1.3.1 Gyula Urs Schoenenberger ezt írta (időpont: 2017. júl. 12., Sze, 10:44): > Hi Gyula, > > I don't know the cause unfortunately, but we observed a similiar issue > on Flink 1.1.3. The problem seems to be gone after upgrading to 1.2.1. > Which version are you run

Re: Can AsyncFunction be applied to connected streams

I think this would not necessarily be a problem. If the async operation is directly after the enrichment operation the enriched operations will be directly forwarded to the async operation. (With a copy step, that can be disabled by enabling object reuse at the StreamExecutionEnvironment) Best,

Re: Associative operation + windowAll - possible parallelism

Yes, your observations are correct! Currently, I see two possible solutions that you could implement as a user: 1. Use .window() with a dummy key followed by a .windowAll(): DataStream input = …; input .map( (in) -> new Tuple2(, in)) .keyBy(0) .window(…) .aggregate(...) .windowAll(…)

Re: System properties when submitting flink job to YARN Session

Hi, Yes, setting the property using -D when creating the session should work to make it available on all workers. I think after that it cannot be changed since they JVMs are already running. If I may ask, what’s your use case for this? Are you still using Beam on Flink or are you using vanilla

Re: Should customized Complex Events be Serializable?

What do you mean by ComplexEvents? Do you mean that the output of CEP library is DataStream? If so, then yes, they should be either Serializable or you should provide custom TypeSerializer. > On 12 Jul 2017, at 06:58, Sridhar Chellappa wrote: > > Folks, > > I am using the CEP library to create

Re: Why would a kafka source checkpoint take so long?

Hi Gyula, I don't know the cause unfortunately, but we observed a similiar issue on Flink 1.1.3. The problem seems to be gone after upgrading to 1.2.1. Which version are you running on? Urs On 12.07.2017 09:48, Gyula Fóra wrote: > Hi, > > I have noticed a strange behavior in one of our jobs: ev

Re: FlinkML ALS is taking too long to run

I don't think you need to employ a distributed system for working with this dataset. An SGD implementation on a single machine should easily handle the job. Best, Sebastian 2017-07-12 9:26 GMT+02:00 Andrea Spina : > Dear Ziyad, > > Yep, I had encountered same very long runtimes with ALS as well

Why would a kafka source checkpoint take so long?

Hi, I have noticed a strange behavior in one of our jobs: every once in a while the Kafka source checkpointing time becomes extremely large compared to what it usually is. (To be very specific it is a kafka source chained with a stateless map operator) To be more specific checkpointing the offset

Re: FlinkML ALS is taking too long to run

Dear Ziyad, Yep, I had encountered same very long runtimes with ALS as well at the time and I recorded improvements by increasing the number of blocks / decreasing #TSs/TM like you've stated out. Cheers, Andrea -- View this message in context: http://apache-flink-user-mailing-list-archiv