Thanks Caizhi, this was very helpful.
// ah
From: Caizhi Weng
Sent: Thursday, August 26, 2021 10:41 PM
To: Hailu, Andreas [Engineering]
Cc: user@flink.apache.org
Subject: Re: 1.9 to 1.11 Managed Memory Migration Questions
Hi!
I've read the first mail again and discover that the direct m
-direct-or-native
// ah
From: Caizhi Weng
Sent: Wednesday, August 25, 2021 10:47 PM
To: Hailu, Andreas [Engineering]
Cc: user@flink.apache.org
Subject: Re: 1.9 to 1.11 Managed Memory Migration Questions
Hi!
Why does this ~30% memory reduction happen?
I don't know how memory is calculated in
Hi folks,
We're about half way complete in migrating our YARN batch processing
applications from Flink 1.9 to 1.11, and are currently tackling the memory
configuration migrations.
Our test application's sink failed with the following exception while writing
to HDFS:
Caused by: java.lang.OutOf
things up.
// ah
From: David Morávek
Sent: Tuesday, August 17, 2021 4:37 AM
To: Hailu, Andreas [Engineering]
Cc: Ravichandran, Soorya Prasanna [Engineering]
; user@flink.apache.org
Subject: Re: Upgrading from Flink on YARN 1.9 to 1.11
Hi Andreas,
the problem here is that the command you're
David Morávek
Sent: Monday, August 16, 2021 6:28 AM
To: Hailu, Andreas [Engineering]
Cc: user@flink.apache.org
Subject: Re: Upgrading from Flink on YARN 1.9 to 1.11
Hi Andreas,
Per-job and session deployment modes should not be affected by this FLIP.
Application mode is just a new deployment mo
Hello folks!
We're looking to upgrade from 1.9 to 1.11. Our Flink applications run on YARN
and each have their own clusters, with each application having multiple jobs
submitted.
Our current submission command looks like this:
$ run -m yarn-cluster --class com.class.name.Here -p 2 -yqu queue-na
Hi Arvid, no. We are leveraging it as part of our application code, but not
Kinesis – after finding and excluding duplicates of this package in our
classpath, we are able to submit a job. Thanks.
// ah
From: Arvid Heise
Sent: Friday, July 30, 2021 1:34 PM
To: Hailu, Andreas [Engineering]
Cc
n application
...
}
These metrics prove immensely valuable as they help us optimize performance,
diagnose issues, as well as predict resource requirements for applications.
// ah
-Original Message-
From: Yangze Guo
Sent: Sunday, August 1, 2021 10:38 PM
To: Hailu, Andreas [Engineeri
07.21 17:37, Hailu, Andreas [Engineering] wrote:
> Hi team, I’m trying to read and write from and to S3 using a custom AWS
> Credential Provider using Flink v1.9.2 on YARN.
>
>
>
> I followed the instructions to create a plugins directory in our Flink
> distribution location and
rom: Yangze Guo
Sent: Thursday, July 29, 2021 11:17 PM
To: Hailu, Andreas [Engineering]
Cc: user@flink.apache.org
Subject: Re: Obtain JobManager Web Interface URL
Hi, Hailu
AFAIK, the ClusterClient#getWebInterfaceURL has been available since 1.10.
Regarding the JobManager web interface, it will
Hi team,
Is there a method available to obtain the JobManager's REST url? We originally
overloaded CliFrontend#executeProgram and nabbed it from the
ClusterClient#getWebInterfaceUrl method, but it seems this method's signature
has been changed and no longer available as of 1.10.0.
Best,
Andrea
Hi team, I'm trying to read and write from and to S3 using a custom AWS
Credential Provider using Flink v1.9.2 on YARN.
I followed the instructions to create a plugins directory in our Flink
distribution location and copy the FS implementation (I'm using s3-fs-hadoop)
package into it. I have al
Monday, 19 April 2021 16:45:25 CEST Hailu, Andreas [Engineering] wrote:
> Hi Flink team,
>
> I'm trying to configure a Flink on YARN with SSL enabled. I've
> followed the documentation's instruction [1] to generate a Keystore
> and Truststore locally, and added a the pr
://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/security-ssl.html#tips-for-yarn--mesos-deployment
// ah
From: Arvid Heise
Sent: Wednesday, April 21, 2021 1:05 PM
To: Hailu, Andreas [Engineering]
Cc: user@flink.apache.org
Subject: Re: [1.9.2] Flink SSL on YARN - NoSuchFileException
Hi Andreas
Hi Flink team,
I'm trying to configure a Flink on YARN with SSL enabled. I've followed the
documentation's instruction [1] to generate a Keystore and Truststore locally,
and added a the properties to my flink-conf.yaml.
security.ssl.rest.keystore: /home/user/ssl/deploy-keys/rest.keystore
securi
Hi folks, I'm trying to get a better understanding of what operations result in
blocked partitions. I've got a batch-processing job that reads from 2 sources,
and then performs a series of Maps/Filters/CoGroups all with the same
parallelism to create a final DataSet to be written to two differen
Hi Robert, I appreciate you having a look. I’ll have a closer look and see what
I can find. Thanks!
// ah
From: Robert Metzger
Sent: Friday, January 22, 2021 2:41 AM
To: Hailu, Andreas [Engineering]
Cc: user@flink.apache.org
Subject: Re: org.apache.flink.runtime.client.JobSubmissionException
this application. Is it possible that it’s busy processing other
jobs?
// ah
From: Robert Metzger
Sent: Thursday, January 21, 2021 10:00 AM
To: Hailu, Andreas [Engineering]
Cc: user@flink.apache.org
Subject: Re: org.apache.flink.runtime.client.JobSubmissionException: Job has
already been
Hello,
We're running 1.9.2 on YARN, and are seeing some interesting behavior when
submitting jobs in a multi-threaded fashion to an application's Flink cluster.
The error we see reported in the client application logs is the following:
org.apache.flink.client.program.ProgramInvocationException:
Thanks Chesnay, Flavio – I believe Flavio’s first recommendation will work well
enough. I agree that the second approach may be a bit finicky to use long-term.
Cheers.
// ah
From: Chesnay Schepler
Sent: Wednesday, December 23, 2020 4:07 AM
To: Flavio Pompermaier ; Hailu, Andreas [Engineering
Hi folks,
I've got a single RichOutputFormat which is comprised of two
HadoopOutputFormats, let's call them A and B, each writing to different HDFS
directories. If a Record matches a certain condition it's written using A,
otherwise it's written with B. Currently, the parallelism that is set at
Hi,
I'm trying to set up the History Server, but none of my applications are
showing up in the Web UI. Looking at the console, I see that all of the calls
to /overview return the following 404 response: {"errors":["File not found."]}.
I've set up my configuration as follows:
JobManager Archive
Okay, I’ll continue to watch the JIRAs. Thanks for the update, Till.
// ah
From: Till Rohrmann
Sent: Wednesday, April 15, 2020 10:51 AM
To: Hailu, Andreas [Engineering]
Cc: Yang Wang ; tison ;
user@flink.apache.org
Subject: Re: Flink Conf "yarn.flink-dist-jar" Question
Hi Andreas,
Yang, Tison,
Do we know when some solution for 13938 and 14964 will arrive? Do you think it
will be in a 1.10.x version?
// ah
From: Hailu, Andreas [Engineering]
Sent: Friday, March 20, 2020 9:19 AM
To: 'Yang Wang'
Cc: tison ; user@flink.apache.org
Subject: RE: Flink Conf "yarn
24 matches
Mail list logo
