Is it possible to mount node local disk for task managers in a k8s application cluster?

uld I create an issue for this feature. Thanks. Best regards, Chen-Che Huang

Re: Is it possible to mount node local disk for task managers in a k8s application cluster?

ocs-master/docs/deployment/resource-providers/native_kubernetes/#pod-template > > Best, > Yang > > Chen-Che Huang 于2021年3月16日周二 下午1:26写道： > > > Hi, > > > > We use the per-job deployment mode to deploy our Flink services on > > Kubernetes. We're considerin

Question about state processor data outputs

icial solution is usually more efficient and reliable than doing it by myself. Many thanks for any comment. key1.txt key1 value11 key1 value21 key1 value31 key2.txt key2 value21 key2 value22 key2 value23 Best wishes, Chen-Che Huang

Re: Question about state processor data outputs

nsform the dataset to a datastream. I'm not sure it's doable based on https://www.alibabacloud.com/blog/deep-insights-into-flink-sql-flink-advanced-tutorials_596628. If it's doable, then I'll be able to solve our problem with applying streamfilesink to the transformed dataset. B

Re: Question about state processor data outputs

Hi Robert, Due to some concerns, we planned to use state processor to achieve our goal. Now we will consider to reevaluate using datastream to do the job while exploring the possibility of implementing a custom FileOutputFormat. Thanks for your comments! Best wishes, Chen-Che Huang On 2021

Any configuration for accelerating state processor

Hi, I have a job that uses the state processor to load data from checkpoints on google cloud storage to do some processing and then write the result to google cloud storage. The total data size is about 30-50 GB and the job may take more than 2 hours to finish. From the flame graph generated fr

Re: Question about state processor data outputs

(File)OutputFormat. > > > On Fri, Apr 16, 2021 at 5:37 AM Chen-Che Huang wrote: > > > Hi Robert, > > > > Thanks for your code. It's really helpful! > > > > However, with the readKeyedState api of state processor, we get dataset > > for our dat

How to set state.backend.rocksdb.latency-track-enabled

for metrics with a marginal impact. However, I couldn't see. Thanks. Best wishes, Chen-Che Huang

Re: How to set state.backend.rocksdb.latency-track-enabled

ct alternative and has been incorrectly written to the release > note. You can refer to the [1] instead. > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#state-backends-latency-tracking-options > > Best, > Yangze Guo > >

Is there a way to know how long a Flink app takes to finish resuming from Savepoint?

p-2: Restoring job from Savepoint Best wishes, Chen-Che Huang

Questions about checkpoint retention

Hi all, To minimize the recovery time from failure, we employ incremental, retained checkpoint with `state.checkpoints.num-retained as 10` in our Flink apps. With this setting, Flink automatically creates new checkpoints regularly and keeps only the latest 10 checkpoints. Besides, for app upgra

Is it possible to make SideOutput back to input stream?

won't be sent to the input stream. We wonder whether it's doable for our scenario with Flink? If so, how should we modify our code to achieve the goal? Many thanks for any comments. Best regards, Chen-Che Huang

Re: Is it possible to make SideOutput back to input stream?

he/flink/streaming/examples/iteration/IterateExample.java > > Best, > Guowei > > > On Mon, Mar 21, 2022 at 2:27 PM Chen-Che Huang wrote: > >> Hi all, >> >> We have an application where the operations on some keys depend on the >> results of related keys.

Unable to create savepoint for iterative job

on of iterationstream. Without iteration, the savepoints/checkpoints can be created successfully. Does anyone encounter similar issues or know how to fix this? Any comment is appreciated. Best wishes, Chen-Che Huang { "threadName": "OutputFlusher for xxx-operator",

Random incorrect checkpoint existence check

stored at google cloud storage. Does anyone encounter the same problem? Any comment is appreciated. Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 'gs://bucket/flink-app//chk-5812/_metadata' already exists Best wishes, Chen-Che Huang