How do I start a Flink application on my Flink+Mesos cluster?

Hi, I am running Mesos without DC/OS [1] and Flink on it. Whe I start my cluster I receive some messages that I suppose everything was started. However, I see 0 slats available on the Flink web dashboard. But I suppose that Mesos will allocate Slots and Task Managers dynamically. Is that right? $

Re: checkpoint failure suddenly even state size less than 1 mb

Hi Yun, Have captured the heap dump which includes thread stack. There is an lock in thread in elasticsearch sink operator. Screenshot of Jprofiler https://github.com/sushantbprise/flink-dashboard/tree/master/thread-blocked And I see lot many threads in waiting state. I found this link which is k

Re: suggestion of FLINK-10868

Thank you for the reply and look forward to the advice of Till. Anyang Peter Huang 于2019年9月5日周四 下午11:53写道： > Hi Anyang, > > Thanks for raising it up. I think it is reasonable as what you requested > is needed for batch. Let's wait for Till to give some more input. > > > > Best Regards > Peter H

Re: HBase Connectors(sink)

Where did you see an HBase sink? On the current master, and all releases that I can remember, flink-hbase only contains input formats / sources. On 06/09/2019 03:23, Ni Yanchun wrote: Hi all, I have found that flink could use hbase as sink in flink source code, but it does not appear

error in Elasticsearch

*Hallo I try to implement this example to write the results of Popular Places into an Elasticsearch index. But when I run a code .. there was some Error appear * * and w

[ANNOUNCE] Kostas Kloudas joins the Flink PMC

Hi everyone, I'm very happy to announce that Kostas Kloudas is joining the Flink PMC. Kostas is contributing to Flink for many years and puts lots of effort in helping our users and growing the Flink community. Please join me in congratulating Kostas! Cheers, Fabian

Re: Wrong result of MATCH_RECOGNIZE clause

Hi Dongwon, I guess you are right and the example is wrong. The new matching sequence should start from line "18". Regards, Dian > 在 2019年9月5日，下午8:28，Dongwon Kim 写道： > > Oops, I think I explained something wrong in the previous email. > B means not A. > Therefore, after the completed match,

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

On 2019/9/6 8:55 下午, Fabian Hueske wrote: I'm very happy to announce that Kostas Kloudas is joining the Flink PMC. Kostas is contributing to Flink for many years and puts lots of effort in helping our users and growing the Flink community. Please join me in congratulating Kostas! congratulat

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

Congratulations Kostas! Regards, Dian > 在 2019年9月6日，下午8:58，Wesley Peng 写道： > > On 2019/9/6 8:55 下午, Fabian Hueske wrote: >> I'm very happy to announce that Kostas Kloudas is joining the Flink PMC. >> Kostas is contributing to Flink for many years and puts lots of effort in >> helping our users

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

Congrats Klou! Cheers, Till On Fri, Sep 6, 2019 at 3:00 PM Dian Fu wrote: > Congratulations Kostas! > > Regards, > Dian > > > 在 2019年9月6日，下午8:58，Wesley Peng 写道： > > > > On 2019/9/6 8:55 下午, Fabian Hueske wrote: > >> I'm very happy to announce that Kostas Kloudas is joining the Flink PMC. > >>

Re: TABLE API + DataStream outsourcing schema or Pojo?

Hi Steve, The memory catalog does not persist metadata and needs to be repopulated everytime. However, you can implement a catalog that persists the metadata to a file or a database. There is an effort to implement a Catalog interface of Hive's metastore. A preview is available in the latest rele

Re: suggestion of FLINK-10868

Hi Anyang, thanks for your suggestions. 1) I guess one needs to make this interval configurable. A session cluster could theoretically execute batch as well as streaming tasks and, hence, I doubt that there is an optimal value. Maybe the default could be a bit longer than 1 min, though. 2) Which

Re: Implementing CheckpointableInputFormat

Hi, CheckpointableInputFormat is only relevant if you plan to use the InputFormat in a MonitoringFileSource, i.e., in a streaming application. If you plan to use it in a DataSet (batch) program, InputFormat is fine. Btw. the latest release Flink 1.9.0 has major improvements for the recovery of ba

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

Congrats Klou! Best, tison. Till Rohrmann 于2019年9月6日周五 下午9:23写道： > Congrats Klou! > > Cheers, > Till > > On Fri, Sep 6, 2019 at 3:00 PM Dian Fu wrote: > >> Congratulations Kostas! >> >> Regards, >> Dian >> >> > 在 2019年9月6日，下午8:58，Wesley Peng 写道： >> > >> > On 2019/9/6 8:55 下午, Fabian Hueske w

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

It was only natural! I already thought Kostas was PMC member! 🎶 --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com On Fri, Sep 6, 2019 at 9:51 AM Zili Chen wrote: > Congrats Klou! > > Best, > tison. > > > Till Rohrmann 于2019年

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

Congrats Klou! Zili Chen 于2019年9月6日周五 下午9:51写道： > Congrats Klou! > > Best, > tison. > > > Till Rohrmann 于2019年9月6日周五 下午9:23写道： > >> Congrats Klou! >> >> Cheers, >> Till >> >> On Fri, Sep 6, 2019 at 3:00 PM Dian Fu wrote: >> >>> Congratulations Kostas! >>> >>> Regards, >>> Dian >>> >>> > 在 2019

Re: understanding task manager logs

Thanks, I'll check it out. On Thu, Sep 5, 2019 at 5:22 AM Fabian Hueske wrote: > Hi Vishwas, > > This is a log statement from Kafka [1]. > Not sure how when AppInfoParser is created (the log message is written by > the constructor). > > For Kafka versions > 1.0, I'd recommend the universal conne

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

Grats Kostas! ^5^5 -kg From: Fabian Hueske Date: Friday, September 6, 2019 at 7:56 AM To: dev , user Subject: [ANNOUNCE] Kostas Kloudas joins the Flink PMC Hi everyone, I'm very happy to announce that Kostas Kloudas is joining the Flink PMC. Kostas is contributing to Flink for ma

Re: Making broadcast state queryable?

Hi Yu, Excuse my late reply... We simply want Flink to be our centralized stream analysis platform, where we 1) consume data, 2) generate analysis, 3) present analysis. I honestly don't want "stream analysis" to spill out to other components in our ecosystem (e.g., sinking insights into a DB-like

Post-processing batch JobExecutionResult

Due to OptimizerPlanEnvironment.execute() throwing exception on the last line there is not way to post-process batch job execution result, like: JobExecutionResult r = env.execute(); // execute batch job analyzeResult(r); // this will never get executed due to plan optimization https://github.com

Re: Post-processing batch JobExecutionResult

Hi spoganshev, The *OptimizerPlanEnvironment* is for creating optimized plan only, as described in the javadoc "An {@link ExecutionEnvironment} that never executes a job but only creates the optimized plan." It execute() is invoked with some internal handling so that it only generates optimized pl

Re: Post-processing batch JobExecutionResult

Hi spoganshev, If you deploy in per-job mode, OptimizerPlanEnvironment would be used, and thus as you pointed out, there is _no_ way to post processing JobExecutionResult. We the community regard this situation as a shortcoming and work on an enhancement progress to enable you get a JobClient as r

Re: Post-processing batch JobExecutionResult

Besides, if you submit the job by Jar Run REST API, it is also OptimizerPlanEnvironment to be used. So again, _no_ post processing support at the moment. Zili Chen 于2019年9月7日周六 上午12:51写道： > Hi spoganshev, > > If you deploy in per-job mode, OptimizerPlanEnvironment would be used, and > thus > as

Re: Post-processing batch JobExecutionResult

Attach the missing link. [1] https://docs.google.com/document/d/1E-8UjOLz4QPUTxetGWbU23OlsIH9VIdodpTsxwoQTs0/edit#heading=h.s8r4pkyalskt [2] https://lists.apache.org/x/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@ Zili Chen 于2019年9月7日周六 上午12:52写道： > Besides, if you subm

Re: Wrong result of MATCH_RECOGNIZE clause

I have created ticket https://issues.apache.org/jira/browse/FLINK-13999 to track it. > 在 2019年9月6日，下午8:58，Dian Fu 写道： > > Hi Dongwon, > > I guess you are right and the example is wrong. The new matching sequence > should start from line "18"

Flink SQL client support for running in Flink cluster

Hi Guys,     I was going through the Flink sql client configuration YAML from the training example and came across a section in the configuration as below:      # Deployment properties allow for describing the cluster to which table# pr