Re: Flink YARN app terminated before the client receives the result

Hi Yangze and all, I have tried numerous times, and this behavior persists. Below is the tail log of taskmanager.log: 2020-03-13 12:06:14.240 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl - Free slot TaskSlot(index:0, state:ACTIVE, re

Re: Re: dimention table join not work under sql-client fink-1.10.0

You are right. The product on alibaba cloud is based on an internal branch. There are much discrepancy between them. *Best Regards,* *Zhenghua Gao* On Fri, Mar 13, 2020 at 1:09 PM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > Thanks, works now. > > Seems the open source version

Re: Flink YARN app terminated before the client receives the result

Would you mind to share more information about why the task executor is killed? If it is killed by Yarn, you might get such info in Yarn NM/RM logs. Best, Yangze Guo Best, Yangze Guo On Fri, Mar 13, 2020 at 12:31 PM DONG, Weike wrote: > > Hi, > > Recently I have encountered a strange behavior

Re: Re: dimention table join not work under sql-client fink-1.10.0

Thanks, works now. Seems the open source version is different from alibaba cloud: https://www.alibabacloud.com/help/doc-detail/62506.htm wangl...@geekplus.com.cn From: Zhenghua Gao Date: 2020-03-13 12:12 To: wangl...@geekplus.com.cn CC: user Subject: Re: dimention table join not work unde

Flink YARN app terminated before the client receives the result

Hi, Recently I have encountered a strange behavior of Flink on YARN, which is that when I try to cancel a Flink job running in per-job mode on YARN using commands like "cancel -m yarn-cluster -yid application_1559388106022_9412 ed7e2e0ab0a7316c1b65df6047bc6aae" the client happily found and conne

Re: dimention table join not work under sql-client fink-1.10.0

We don't support 'PROCTIME()' in a temporal table join. Please use a left table's proctime field. [1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/streaming/joins.html#usage-1 *Best Regards,* *Zhenghua Gao* On Fri, Mar 13, 2020 at 11:57 AM wangl...@geekplus.com.cn

dimention table join not work under sql-client fink-1.10.0

Kafka source table: CREATE TABLE out_order ( out_order_code VARCHAR, intput_date BIGINT, owner_code VARCHAR ) WITH ( 'connector.type' = 'kafka',MySQL dimention table: CREATE TABLE dim_owner ( owner_code VARCHAR, owner_name VARCHAR ) WITH ( 'connector.t

Re: Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

I think you can modify the operator’s parallelism. It is only if you have set maxParallelism, and while restoring from a checkpoint, you shouldn’t modify the maxParallelism. Otherwise, I believe the state will be lost. - Sivaprasanna On Fri, 13 Mar 2020 at 9:01 AM, LakeShen wrote: > Hi communit

Re: Flink 1.10 StopWithSavepoint only suit for the sources that implement the StoppableFunction interface?

Thanks a lot!, tison tison 于2020年3月12日周四 下午5:56写道： > The StoppableFunction is gone. > > See also https://issues.apache.org/jira/browse/FLINK-11889 > > Best, > tison. > > > LakeShen 于2020年3月12日周四 下午5:44写道： > >> Hi community, >> now I am seeing the FLIP-45 , as I see the stop command onl

Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

Hi community, I have a question is that I cancel the flink task and retain the checkpoint dir, then restore from the checkpoint dir ,can I change the flink operator's parallelism,in my thoughts, I think I can't change the flink operator's parallelism,but I am not sure. Thanks to your rep

Re: Start flink job from the latest checkpoint programmatically

Have you tried to retain checkpoints or use savepoints? Take a look at [1] and see if that can help. Best, Flavio [1] https://eventador.io/blog/apache-flink-checkpoints-and-savepoints/ Il Ven 13 Mar 2020, 00:02 Eleanore Jin ha scritto: > Hi All, > > The setup of my flink application is allow u

Start flink job from the latest checkpoint programmatically

Hi All, The setup of my flink application is allow user to start and stop. The Flink job is running in job cluster (application jar is available to flink upon startup). When stop a running application, it means exit the program. When restart a stopped job, it means to spin up new job cluster wit

Re: Flink gelly dependency in transient EMR cluster

I reply to myself with the solution in case someone else is having the same question It is only needed to add a copy command to copy the jar from flink/opt to flink/lib, in my case: StepConfig addGellyStep = new StepConfig() .withName("add-gelly-step") .withHadoopJarStep(new Hadoo

Communication between two queries

Hello everyone! So essentially, I've two identical queries (q1 and q2) running in parallel (Streams). I'm trying to activate the ingestion of data to q2 based on what is processed in q1. E.g say that we want to start ingesting data to q2 when a tuple with timestamp > 5000 appears in q1. The queri

Expected behaviour when changing operator parallelism but starting from an incremental checkpoint

Hi, What's the expected behaviour of: * changing an operator's parallelism * deploying this change from an incremental (RocksDB) checkpoint instead of a savepoint The flink docs[0][1] are a little unclear on what the expected behaviour is here. I understand that the key-space is being changed be

Re: how to specify yarnqueue when starting a new job programmatically?

Do you mean in the job java code, where you call “env.execute()”? I don't think so. The yarn queue is a Flink cluster level configuration, and I don't think you can change the cluster level configurations in your job code. You job might even be created after the cluster is started, e.g., in a Flin

Re: how to specify yarnqueue when starting a new job programmatically?

Thank you Xintong Song, is there any way to queue programmatically, i.e. via java code? Regards, Vitaliy On Thu, Mar 12, 2020 at 5:56 AM Xintong Song wrote: > Hi Vitaliy, > > You can specify a yarn queue by either setting the configuration option > 'yarn.application.queue' [1], or using the co

Re: Flink 1.10 StopWithSavepoint only suit for the sources that implement the StoppableFunction interface?

The StoppableFunction is gone. See also https://issues.apache.org/jira/browse/FLINK-11889 Best, tison. LakeShen 于2020年3月12日周四 下午5:44写道： > Hi community, > now I am seeing the FLIP-45 , as I see the stop command only suit > for the sources that implement the StoppableFunction interface

Flink 1.10 StopWithSavepoint only suit for the sources that implement the StoppableFunction interface?

Hi community, now I am seeing the FLIP-45 , as I see the stop command only suit for the sources that implement the StoppableFunction interface. So I have a question is that if I use StopWithSavepoint command to stop my flink task , just like './flink stop -p xxx ...', this command

Flink gelly dependency in transient EMR cluster

Hello, I'm trying to run a flink job that works with graphs in a transient cluster in EMR, here is my code: -- HadoopJarStepConfig copyJarStepConf = new HadoopJarStepConfig() .withJar("command-runner.jar") .withArgs("bash", "-c", "aws s3 cp s3://" + BUCKET_NAME

Re: Automatically Clearing Temporary Directories

Hi David, > Would it be safe to automatically clear the temporary storage every time when a TaskManager is started? > (Note: the temporary volumes in use are dedicated to the TaskManager and not shared :-) Yes, it is safe in your case. Best, Gary On Tue, Mar 10, 2020 at 6:39 PM David Maddison w

Re: Flink 1.10 container memory configuration with Mesos.

BTW, the dynamic config will also occur in TM side logs [1]. It would be good to print it in INFO level as well. [1] https://github.com/apache/flink/blob/663af45c7f403eb6724852915bf2078241927258/flink-mesos/src/main/java/org/apache/flink/mesos/entrypoint/MesosTaskExecutorRunner.java#L77 Best, Ya

Re: Flink 1.10 container memory configuration with Mesos.

It seems we already have such logs in [1]. If that is the case, +1 for changing it to INFO level. [1] https://github.com/apache/flink/blob/663af45c7f403eb6724852915bf2078241927258/flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/LaunchableMesosWorker.java#L341 Best, Yangz

Re: Flink 1.10 container memory configuration with Mesos.

Instead of just launching TM as it works right now, I suggest to log launch command first, and then launch TM

Re: Flink 1.10 container memory configuration with Mesos.

Glad to hear that your issue is fixed. I'm not sure what you suggest to add. Could you tell it more specific or create a Jira ticket? Best, Yangze Guo On Thu, Mar 12, 2020 at 3:51 PM Alexander Kasyanenko wrote: > > Hi Yangze, Xintong, > > Thank you for instant response. > > And big thanks for t

Re: Flink 1.10 container memory configuration with Mesos.

Hi Yangze, Xintong, Thank you for instant response. And big thanks for the hint on TM launch command. It indeed was the problem. I've added my own custom mesos-taskmanager.sh to echo the launch command (I've switched to DEBUG lev