Re: Event time window questions

Thanks Sendoh. Is there a way to advance watermark even when there are no incoming events. What exactly does setAutoWatermarkInterval do? Also I don't see the watermark displayed in flink dashboard. Will the watermark advance only when there is data from all consuming kafka topic and partitions?

Re: Timer & Window Memory Consumption

Thanks Fabian but for 1.5k messages per second per TM there are several million Internal & TimerWindow objects created within a period of 5 seconds. Is there a way to get debug this issue? Regards, Navneeth On Tue, Jan 23, 2018 at 2:09 AM, Fabian Hueske wrote: > Hi, > > TimeWindows and Timers a

Flip-6 + Dynamic scaling

Hi, There hasn't been any activity on https://issues.apache.org/jira/browse/FLINK-4319 since September 21. Any ETA on this? I required dynamic upscaling and downscaling based on the volume of data. Jayant Ameta

Re: Network memory segments

Thanks Chesnay. On Tue, Jan 23, 2018 at 6:54 AM, Chesnay Schepler wrote: > I could reproduce this locally and opened a JIRA > . > > > On 21.01.2018 04:32, Navneeth Krishnan wrote: > > Hi, > > We recently upgraded from flink 1.3 to 1.4 and in the

Re: Fwd: Replace classes at runtime

Hi Gordon, Is there a way to update the job application without any downtime? Jayant Ameta On Tue, Dec 19, 2017 at 12:09 AM, Tzu-Li (Gordon) Tai wrote: > Hi Jayant, > > Updating your job application / operator code at runtime is currently not > available in Flink. > It is however achievable via

Re: AKA and quarantine

Any suggestions ? I know these are very general issue but these are edge conditions that we want the community to give us general advise on .. On Sun, Jan 21, 2018 at 3:16 PM, Vishal Santoshi wrote: > There have been a couple of instances where one of our TMs was quarantined > ( the cause is ir

Re: Failing to recover once checkpoint fails

If we hit the retry limit, abort the job. In our case we will restart from the last SP ( we as any production pile do it is n time s a day ) and that I would think should be OK for most folks ? On Tue, Jan 23, 2018 at 11:38 AM, Vishal Santoshi wrote: > Thank you for considering this. If I under

Re: Multiple Elasticsearch sinks not working in Flink

As mentioned in the issue, please check if using two different config map objects solves the issue. On Tue, Jan 23, 2018 at 1:32 PM, Teena Kappen // BPRISE < teena.kap...@bprise.com> wrote: > Thanks Fabian. I will go through it and add info if required. > > > > *From:* Fabian Hueske [mailto:fhue.

Re: state.checkpoints.dir

Hi Hao , Thank you for reply . I was more of trying to find how do I manipulate when I run locally from IDE . ~ Biswajit On Mon, Jan 22, 2018 at 12:56 PM, Hao Sun wrote: > We generate flink.conf on the fly, so we can use different values based on > environment. > > On Mon, Jan 22, 2018 at 12:5

Re: Event time window questions

Hi, you can write your own trigger and window, and implement whatever logic there. There are some examples https://github.com/apache/flink/blob/1875cac03042dad4a4c47b0de8364f02fbe457c6/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers/ If you don't see any even

Re: Task Manager detached under load

Hi. Did you find a reason for the detaching ? I sometimes see the same on our system running Flink 1.4 on dc/os. I have enabled taskmanager.Debug.memory.startlogthread for debugging. Med venlig hilsen / Best regards Lasse Nedergaard > Den 20. jan. 2018 kl. 12.57 skrev Kien Truong : > > Hi,

Re: System.exit() vs throwing exception from the pipeline

As I understood if wanted a smooth shutdown, it's recommend to throw exception, and then cancel() is called, where you can even write your own. Don't think it's the same as System.exit() Cheers, Sendoh -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: GetExecutionPlan fails with IllegalArgumentException in Comparator

Great! Thank you :-) 2018-01-23 17:58 GMT+01:00 Bauss, Julian : > Hi Fabian, > > > > sure! > > > > See: https://issues.apache.org/jira/browse/FLINK-8498 > > > > Best Regards, > > > > Julian > > > > *Von:* Fabian Hueske [mailto:fhue...@gmail.com] > *Gesendet:* Montag, 22. Januar 2018 14:42 > *An:*

AW: GetExecutionPlan fails with IllegalArgumentException in Comparator

Hi Fabian, sure! See: https://issues.apache.org/jira/browse/FLINK-8498 Best Regards, Julian Von: Fabian Hueske [mailto:fhue...@gmail.com] Gesendet: Montag, 22. Januar 2018 14:42 An: Bauss, Julian Cc: user@flink.apache.org Betreff: Re: GetExecutionPlan fails with IllegalArgumentException in Co

Re: Flink CEP exception during RocksDB update

Hi Kostas, I was able to reproduce the error with 1.4.0. After upgrading the cluster to 1.5 snapshot and running through the same data I am still experiencing the same exception. CEP patterns that I am running are using followed by patterns e.g AfBfC. From my experience I was never able to get

Re: CEP issue in 1.3.2. Does 1.4 fix this ?

Thanks. Confirmed through tests the above behavior. On Tue, Jan 23, 2018 at 4:09 AM, Kostas Kloudas wrote: > Hi Vishal, > > Thanks for checking and glad to hear that your job works after the fix! > > As for the equals/hashcode question, if your question is if you have to > implement exact equals

Re: Failing to recover once checkpoint fails

Thank you for considering this. If I understand you correctly. * CHK pointer on ZK for a CHK state on hdfs was done successfully. * Some issue restarted the pipeline. * The NN was down unfortunately and flink could not retrieve the CHK state from the CHK pointer on ZK. Before * The CHK pointer

Re: Error with Avro/Kyro after upgrade to Flink 1.4

Thanks for the follow-up Stephan. I have been running this job from a built jar file which was submitted to an existing Flink 1.4 cluster, not from within the IDE. Interestingly, I am now getting the same error when any of the following 3 conditions are true: 1. I run the job on a local cluster fr

Re: NoClassDefFoundError of a Avro class after cancel then resubmit the same job

Hi! We changed a few things between 1.3 and 1.4 concerning Avro. One of the main things is that Avro is no longer part of the core Flink class library, but needs to be packaged into your application jar file. The class loading / caching issues of 1.3 with respect to Avro should disappear in Flink

Re: Error with Avro/Kyro after upgrade to Flink 1.4

Hi! Thanks for posting this. Ugly error. This does indeed look like either a dependency conflict, or a mix of incompatible class files. Some suggestions to diagnose this further: - Is this the setup of your POM to build the JAR file for your application, or for running it from within the IDE?

Re: Network memory segments

I could reproduce this locally and opened a JIRA . On 21.01.2018 04:32, Navneeth Krishnan wrote: Hi, We recently upgraded from flink 1.3 to 1.4 and in the task manager UI it shows there are 0 memory segments whereas in 1.3 I think it was defa

Re: java.lang.ClassCastException: oracle.sql.TIMESTAMP cannot be cast to java.sql.Timestamp

You have to make sure that the property is set on every TaskManager that executes your job. I don't know if there is some easier way to set the system properties on the entire cluster. Am 1/22/18 um 12:25 PM schrieb Puneet Kinra: Interesting System property working while submitting from the

Re: Need Suggestions on the Requirement!!

Hi, Ingesting a table from a retraction stream is not supported yet. You can emulate an upsert ingestion mode with a LAST_VALUE aggregation function and a GROUP BY clause (see mail thread [1]). Best, Fabian [1] https://lists.apache.org/thread.html/b9fc2926f6fa460c70ebee8541b05f 9af5cfe60c61f3563

Need Suggestions on the Requirement!!

Hi Actually we are building a CRM (offer management tool) ,Currently Working on that.. 1) Multiple Streaming Table Sources. 2) We are hitting a query on those Tables for getting aggregated values. 3) Want to store the output of these aggregated values into the another table & then want to run

RE: Multiple Elasticsearch sinks not working in Flink

Thanks Fabian. I will go through it and add info if required. From: Fabian Hueske [mailto:fhue...@gmail.com] Sent: 23 January 2018 15:20 To: Teena Kappen // BPRISE Cc: Timo Walther ; user@flink.apache.org Subject: Re: Multiple Elasticsearch sinks not working in Flink Hi Teena, I created FLINK-84

Re: Failing to recover once checkpoint fails

Hi Vishal, I think you might be right. We fixed the problem that checkpoints where dropped via https://issues.apache.org/jira/browse/FLINK-7783 . However, we still have the problem that if the DFS is not up at all then it will look as if the job

Re: Timer & Window Memory Consumption

Hi, TimeWindows and Timers are created for each window, i.e., every 5 seconds for every distinct key that a task is processing. Event-time windows are completed and cleaned up when a watermark is received that passes the window end timestamp. Therefore, there might be more than one window per key

Re: Kafka Producer timeout causing data loss

Hi Ashish, Originally, Flink always performed full recovery in case of a failure, i.e., it restarted the complete application. There is some ongoing work to improve this and make recovery more fine-grained (FLIP-1 [1]). Some parts have been added for 1.3.0. I'm not familiar with the details, but

Re: Multiple Elasticsearch sinks not working in Flink

Hi Teena, I created FLINK-8489 [1] to track the issue. Please have a look and add information that might be relevant. Best, Fabian [1] https://issues.apache.org/jira/browse/FLINK-8489 2018-01-18 14:16 GMT+01:00 Teena Kappen // BPRISE : > Hi Timo, > > > > It works fine when the second sink is a

Re: Flink standalone scheduler

Flink's standalone mode does not support dynamic resource allocation. Flink integrates with YARN or Mesos and does not try to solve the problem these frameworks are solving. Best, Fabian 2018-01-18 8:31 GMT+01:00 Soheil Pourbafrani : > Does Flink standalone scheduler support dynamic resource all

Re: How to get automatic fail over working in Flink

Hi James, did you configure checkpointing [1] and a recovery strategy [2] for your job? Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/checkpointing.html [2] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/restart_strategies.html 20

Re: Flink streaming (1.3.2) KafkaConsumer08 - Unable to retrieve any partitions

Hi Tarandeep, The error you are seeing only occurs when on startup of the consumer, it couldn’t retrieve any partition information from Kafka. Therefore, according to your description, there should be another error that caused the previous execution of the job to fail. Could you check that? Mayb

Re: Failing to recover once checkpoint fails

Sorry for the late reply. I created FLINK-8487 [1] to track this problem @Vishal, can you have a look and check if if forgot some details? I logged the issue for Flink 1.3.2, is that correct? Please add more information if you think it is relevant. Thanks, Fabian [1] https://issues.apache.org/j

Re: CEP issue in 1.3.2. Does 1.4 fix this ?

Hi Vishal, Thanks for checking and glad to hear that your job works after the fix! As for the equals/hashcode question, if your question is if you have to implement exact equals() method and the corresponding hashcode() then the answer is yes. These methods are used when retrieving and cleaning

Re: Flink Kinesis Consumer re-reading merged shards upon restart

Hi Phil, Thanks a lot for the PR! Let’s continue the discussion there. I think the ICLA signing is only strictly required for becoming a Committer of the project, so we’re good to go for the pull request :) Cheers, Gordon On 23 January 2018 at 9:13:45 AM, Philip Luppens (philip.lupp...@gmail.co

Re: Flink Kinesis Consumer re-reading merged shards upon restart

Hi Gordon, I’ve created a PR [1] with my proposed code changes. Let me know if anything is missing. I think I signed a CLA many years ago, so that should be ok as well. [1] https://github.com/apache/flink/pull/5337 HTH, -Phil On Mon, Jan 22, 2018 at 7:17 PM, Philip Luppens wrote: > Hi Gordo