RE: Re: flink configuration in flink kubernetes operator question about password

Hi tian tian, I think we can use podTemplate to mount kubernetes secrets as file or environment variables. Then we can access the secrets in our flink program. Please refers to https://github.com/apache/flink-kubernetes-operator/blob/main/examples/pod-template.yaml

Re: flink configuration in flink kubernetes operator question about password

Like s3.secret-key, the plaintext password cannot be directly written in the configuration. Is there a template language like jinja that can be replaced after mounting to the pod? >

flink configuration in flink kubernetes operator question about password

hi all, How to specify the password and other information that needs to be encrypted in the configuration file?

Re: Relation between Flink Configuration and TableEnv

rConfigOptions and > ExecutionConfigOptions. > > By Flink Configuration I guess you mean the configuration in Configuration. > Sadly, as you say, it can be configured only once when creating the execution > environment, and only a few configuration can be modified with the > StreamE

Re: Relation between Flink Configuration and TableEnv

Hi! TableConfig is for configurations related to the Table and SQL API, especially the configurations in OptimizerConfigOptions and ExecutionConfigOptions. By Flink Configuration I guess you mean the configuration in Configuration. Sadly, as you say, it can be configured only once when creating

Re: Relation between Flink Configuration and TableEnv

Sorry, I mean the relation between Flink Configuration and TableConfig, not TableEnv. Best, Paul Lam Paul Lam 于2021年9月24日周五 上午12:24写道： > Hi all, > > Currently, Flink creates a new Configuration in TableConfig of > StreamTableEnvironment, and synchronizes options in it

Relation between Flink Configuration and TableEnv

Hi all, Currently, Flink creates a new Configuration in TableConfig of StreamTableEnvironment, and synchronizes options in it back to the Configuration of the underlying StreamExecutionEnvironment afterward. However, only "relevant" options are set back [1], others are dropped silently. That block

Re: flink configuration: best practice for checkpoint storage secrets

Hello Till Thanks a lot for the reply. But it turns out the IAM is applicable only when the job is running inside AWS, which is not my case (basically we are just using the S3 API provided by other services). By reading again the flink doc, it seems it's suggesting to use the flink-conf.yaml file,

Re: flink configuration: best practice for checkpoint storage secrets

Hi Qinghui, the recommended way would be to use AWS identity and access management (IAM) [1] if possible. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#configure-access-credentials Cheers, Till On Wed, Oct 7, 2020 at 12:31 PM XU Qinghui wrote: > Hello, fol

flink configuration: best practice for checkpoint storage secrets

Hello, folks We are trying to use S3 for the checkpoint storage, and this involves some secrets in the configuration. We tried two approaches to configure those secrets: - in the jvm application argument for jobmanager and taskmanager, such as -Ds3.secret-key - in the flink-conf.yaml file for jobm

Re: Optimal Flink configuration for Standalone cluster.

> > It could really be specific to your workload. Some workload may need more > heap memory while others may need more off-heap. > The main 'process' of my project creates a cross product of datasets and then applies a function to all of them to extract some features. > Alternatively, you can try

Re: Optimal Flink configuration for Standalone cluster.

> > Since changing off-heap removes memory from '.task.heap.size' is there a > ratio that I should follow for better performance? > I don't think so. It could really be specific to your workload. Some workload may need more heap memory while others may need more off-heap. Also, my guess (since I a

Re: Optimal Flink configuration for Standalone cluster.

Hi Xintong, Thank you for the quick response. doing 1), without increasing 'task.off-heap.size' does not change the issue, but increasing the off-heap alone does. What should the off-heap value size be? Since changing off-heap removes memory from '.task.heap.size' is there a ratio that I should f

Re: Optimal Flink configuration for Standalone cluster.

Hi Dimitris, Regarding your questions. a) For standalone clusters, the recommended way is to use `.flink.size` rather than `.process.size`. `.process.size` includes JVM metaspace and overhead in addition to `.flink.size`, which usually do not really matter for standalone clusters. b) In case of di

Optimal Flink configuration for Standalone cluster.

Hello, I'm having a bit of trouble understanding the memory configuration on flink. I'm using flink10.0.0 to read some datasets of edges and extract features. I run this on a cluster consisting of 4 nodes , with 32cores and 252GB Ram each, and hopefully I could expand this as long as I can add ext

Re: Flink configuration on Docker deployment

s > located in the path "/opt/flink/conf". > Docker volume also could be used to override the flink configuration when > you start the jobmanager > and taskmanager containers[1]. > > Best, > Yang > > [1]. https://docs.docker.com/storage/volumes/ > > Soheil

Re: Flink configuration on Docker deployment

Hi Soheil, Since you are not using any container orchestration framework(e.g. docker-compose, Kubernetes, mesos), so you need to manually update the flink-conf.yaml in your docker images. Usually, it is located in the path "/opt/flink/conf". Docker volume also could be used to override

Flink configuration on Docker deployment

Hi, I need to set up a Flink cluster using the docker(and not using the docker-compose). I successfully could strat the jobmanager and taskmanager but the problem is I have no idea how to change the default configuration for them. For example in the case of giving 8 slots to the taskmanager or cha

Re: Flink configuration at runtime

Hi Amran, thanks a lot for your message. I think this is a valid feature request. I've created a JIRA ticket to track it: https://issues.apache.org/jira/browse/FLINK-14856 (this does not mean this gets addressed immediately. However, there are currently quite some improvements to the configuration

Re: Flink configuration at runtime

Hi Amran, Change the config option at runtime? No, Flink does not support this feature currently. However, for Flink on Yarn job cluster mode, you can specify different config options for different jobs via program or flink-conf.yaml(copy a new flink binary package then change config file). Best

Re: Flink configuration at runtime

Hi Amran, Some configs, including "state.checkpoints.num-retained", are cluster configs that always apply to the entire Flink cluster. An alternative is to use per-job mode if you are running Flink jobs on k8s/docker or yarn. Thus to create a Flink cluster for a single job. [1] https://ci.apache.

Flink configuration at runtime

Is it possible to configure certain settings at runtime, on a per-job basis rather than globally within flink-conf.yaml? For example, I have a job where it's desirable to retain a large number of checkpoints via state.checkpoints.num-retained. The checkpoints are cheap, and it's low cost. For oth

Re: Does flink configuration support configed by environment variables?

Thanks, maybe overwrite configuration in 'conf/flink-conf.yaml' by docker-entrypoint.sh is a common way to do it, thanks for your advice. Stephen Connolly 于2019年4月1日周一 下午10:33写道： > I don't think it does. I ended up writing a small CLI tool to enabling > templating the file from environment vari

Re: Does flink configuration support configed by environment variables?

I don't think it does. I ended up writing a small CLI tool to enabling templating the file from environment variables. There are loads of such tools, but mine is https://github.com/stephenc/envsub I have the dockerfile like so: ARG FLINK_VERSION=1.7.2-alpine FROM flink:${FLINK_VERSION} ARG ENVSUB

Does flink configuration support configed by environment variables?

Hi guys, I am using flink 1.7.2 deployed by kubernetes, and I want to change the configurations about flink, for example customize `taskmanager.heap.size`. Does flink support using environment variables to override configurations in `conf/flink-conf.yaml` ?

Re: Access Flink configuration in user functions

-backend), the backend factory does get access to it I believe. On 28.12.2018 05:56, Paul Lam wrote: Hi to all, I would like to use a custom RocksDBStateBackend which uses the default checkpoint dir in Flink configuration, but I failed to find a way to access Flink configuration in the user code

Access Flink configuration in user functions

Hi to all, I would like to use a custom RocksDBStateBackend which uses the default checkpoint dir in Flink configuration, but I failed to find a way to access Flink configuration in the user code. So I wonder is it possible to retrieve Flink configurations (not user-defined global parameters

Re: Flink configuration

t;> [2]https://ci.apache.org/projects/flink/flink-docs- >> release-0.8/yarn_setup.html >> >> >> >> -- >> View this message in context: http://apache-flink-user- >> mailing-list-archive.2336050.n4.nabble.com/Flink- >> configuration-tp11210.html >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive at Nabble.com. >> >

Re: Flink configuration

ease-0.8/config.html > [2] > https://ci.apache.org/projects/flink/flink-docs-release-0.8/yarn_setup.html > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-configuration-tp11210.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Flink configuration

elease-0.8/yarn_setup.html -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-configuration-tp11210.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.