Re: Apache Flink Serialization Question

2025-05-19 Thread Zhanghao Chen
It would still work 发件人: Richard Cheung 发送时间: 星期二, 五月 20, 2025 4:08:00 上午 收件人: Zhanghao Chen 抄送: Мосин Николай ; Schwalbe Matthias ; user@flink.apache.org 主题: Re: Apache Flink Serialization Question Hi all, Thanks again for the help! I have one more follow

Re: Flink task manager pod auto scaling

2025-05-19 Thread Zhanghao Chen
this end. Best, Zhanghao Chen From: Kamal Mittal via user Sent: Monday, May 19, 2025 13:40 To: User Subject: Flink task manager pod auto scaling Hello, Does flink supports vertical task manager pod auto scaling? Rgds, Kamal

Re: Apache Flink Serialization Question

2025-05-16 Thread Zhanghao Chen
eatment of these nullable cases. Best, Zhanghao Chen From: Мосин Николай Sent: Friday, May 16, 2025 0:02 To: Richard Cheung Cc: Schwalbe Matthias ; Zhanghao Chen ; user@flink.apache.org Subject: Re: Apache Flink Serialization Question For List I just setup TypeI

Re: Keyed watermarks: A fine-grained watermark generation for Apache Flink

2025-05-15 Thread Zhanghao Chen
Thanks for the insightful sharing! Best, Zhanghao Chen From: Lasse Nedergaard Sent: Thursday, May 15, 2025 13:10 To: Zhanghao Chen Cc: mosin...@yandex.ru ; user@flink.apache.org Subject: Re: Keyed watermarks: A fine-grained watermark generation for Apache

Re: Keyed watermarks: A fine-grained watermark generation for Apache Flink

2025-05-14 Thread Zhanghao Chen
P-467%3A+Introduce+Generalized+Watermarks Best, Zhanghao Chen From: Мосин Николай Sent: Thursday, May 15, 2025 3:58 To: user@flink.apache.org Subject: Keyed watermarks: A fine-grained watermark generation for Apache Flink I found paper https://scholar.google.com/sc

Re: Flink task manager PODs autoscaling - K8s installation

2025-05-13 Thread Zhanghao Chen
[3] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#externalized-declarative-resource-management Best, Zhanghao Chen From: Kamal Mittal via user Sent: Monday, May 12, 2025 13:35 To: user@flink.apache.org Subject: Flink task ma

Re: Apache Flink Serialization Question

2025-05-13 Thread Zhanghao Chen
Flink still use PojoSerializer for the class while only using Kryo for the UUID field. Best, Zhanghao Chen From: Richard Cheung Sent: Wednesday, May 14, 2025 3:21 To: user@flink.apache.org Subject: Apache Flink Serialization Question Hi all! I have a question

Re: 请帮忙看下问题,master跑一两天后就退出

2025-04-13 Thread Zhanghao Chen
从日志看是 JM 的堆外内存设置的不足,导致 JVM 自己做 JIT 编译时出现了堆外内存的 OOM。你可以尝试通过增大 jobmanager.memory.jvm-overhead.max 和 jobmanager.memory.jvm-overhead.fraction 参数来扩大 JVM overhead 区内存的大小。 Best, Zhanghao Chen From: 咖啡本色/kf <250071...@qq.com> Sent: Monday, April 14, 2025 7:56 To

Re: Hardware CPU vCore fetch issue when Flink is running on yarn

2025-03-16 Thread Zhanghao Chen
Hi, Vincent. Old versions of JDK8 lack proper container awareness. It is suggested to upgrade your JDK to at least 8u372 or 11.0.16, see [1][2] for more details. [1] https://bugs.openjdk.org/browse/JDK-8146115 [2] https://bugs.openjdk.org/browse/JDK-8230305 Best, Zhanghao Chen

Re: How to register pojo type information for third party pojo classes

2025-02-13 Thread Zhanghao Chen
tring name, TypeInformation typeInfo) API to manually specify the typeinformation on the state side. Best, Zhanghao Chen From: Sachin Mittal Sent: Thursday, February 13, 2025 12:23 To: Zhanghao Chen Cc: user Subject: Re: How to register pojo type information for third

Re: How to register pojo type information for third party pojo classes

2025-02-13 Thread Zhanghao Chen
Hi, You should register a custom type info for java.util.List with a custom list serializer instead of class B itself in this case. Best, Zhanghao Chen From: Sachin Mittal Sent: Thursday, February 13, 2025 16:10 To: Zhanghao Chen Cc: user Subject: Re: How to

Re: How to register pojo type information for third party pojo classes

2025-02-12 Thread Zhanghao Chen
java.util.List is introduced, and you should not need any additional type registrations to disable the generic types in this case. Best, Zhanghao Chen From: Sachin Mittal Sent: Thursday, February 13, 2025 12:23 To: Zhanghao Chen Cc: user Subject: Re: How to

Re: How to register pojo type information for third party pojo classes

2025-02-12 Thread Zhanghao Chen
Hi, you may use the option "pipeline.serialization-config" [1] to register type info for any custom type, which is available since Flink 1.19. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#pipeline-serialization-config Best, Zha

Re: Flink High Availability Data Cleanup

2025-02-06 Thread Zhanghao Chen
Simply put, HA metadata will only be deleted when the job reaches terminal state (either failed or cancelled). The ref doc is https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/#restart-strategies Best, Zhanghao Chen From

Re: Unsubscribe

2025-02-04 Thread Zhanghao Chen
Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org. Best, Zhanghao Chen From: Mujahid Niaz Sent: Wednesday, February 5, 2025 9:29 Cc: user@flink.apache.org Subject: Unsubscribe Unsubscribe

Re: Dead Letter Queue for FlinkSQL

2025-02-04 Thread Zhanghao Chen
You'll need to implement an custom sink for that. Best, Zhanghao Chen From: Ilya Karpov Sent: Monday, February 3, 2025 18:30 To: user Subject: Dead Letter Queue for FlinkSQL Hi there, Because sink connectors can throw exceptions in real time (for example

Re: Flink High Availability Data Cleanup

2025-02-04 Thread Zhanghao Chen
egies Best, Zhanghao Chen From: Chen Yang via user Sent: Wednesday, February 5, 2025 7:17 To: user@flink.apache.org Cc: Vignesh Chandramohan Subject: Flink High Availability Data Cleanup Hi Flink Community, I'm running the Flink jobs (standalone mode)

Re: Java 17 support in Flink

2025-01-23 Thread Zhanghao Chen
[1] has been merged. I'll try to work on it under [2], looking forward to more volunteers on it! [1] https://issues.apache.org/jira/browse/FLINK-30478 [2] https://issues.apache.org/jira/browse/FLINK-15736 Best, Zhanghao Chen From: Nikola Milutinovic

Re: table.exec.source.idle-timeout support

2025-01-23 Thread Zhanghao Chen
, Zhanghao Chen From: Nikola Milutinovic Sent: Friday, January 10, 2025 23:48 To: user@flink.apache.org Subject: Re: table.exec.source.idle-timeout support Hi Nic. I do not have a solution (soy), but have seen something similar. And have complained about it

Re: Object-Reuse with Table API / Flink SQL

2025-01-23 Thread Zhanghao Chen
Hi, In short, yes if without user-defined functions. For UDFs, you'll have to ensure that it does cache data internally (maintain a local hash map for example), otherwise downstream ops may change the cached data and breaks data integrity. Best, Zhanghao

Re: Java 17 support in Flink

2025-01-05 Thread Zhanghao Chen
[2]. [1] https://lists.apache.org/thread/qvw66of180t3425pnqf2mlx042zhlgnn [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-458%3A+Long-Term+Support+for+the+Final+Release+of+Apache+Flink+1.x+Line Best, Zhanghao Chen From: Anuj Jain Sent: Sunday, January 5

Re: How to programatically register a 3rd party serializer (Flink 1.20)

2024-12-14 Thread Zhanghao Chen
Hi Jean-Marc, Thanks for reporting this, this is a mistake. Your code is correct, I've created a issue [1] to fix that. [1] https://issues.apache.org/jira/browse/FLINK-36904 Best, Zhanghao Chen From: Jean-Marc Paulin Sent: Thursday, December 5, 2024 21:

Re: Recommendations for avoid Kryo

2024-09-17 Thread Zhanghao Chen
/datastream/fault-tolerance/serialization/types_serialization/#defining-type-information-using-a-factory Best, Zhanghao Chen From: Lasse Nedergaard Sent: Friday, September 13, 2024 14:27 To: user Subject: Recommendations for avoid Kryo Hi. I was wondering how others

Re: Flink 1.19.1 Java 17 Compatibility

2024-09-17 Thread Zhanghao Chen
In our production environment, it works fine. Best, Zhanghao Chen From: Sachin Sharma Sent: Friday, September 13, 2024 1:19 To: Oscar Perez via user Subject: Flink 1.19.1 Java 17 Compatibility Hi, We are planning to use Flink 1.19.1 with kubernetes operator, I

Re: TaskManager Using more CPU Cores in Kubernetes Deployment than configured

2024-09-17 Thread Zhanghao Chen
and the resources of the underlying host are returned instead. Check [1] for more information. [1] https://jvmaware.com/container-aware-jvm/ Best, Zhanghao Chen From: Oliver Schmied Sent: Wednesday, September 11, 2024 16:49 To: user@flink.apache.org Subject: Tas

Re: Managed memory and state size

2024-08-05 Thread Zhanghao Chen
full cp size counts all files. Best, Zhanghao Chen From: banu priya Sent: Monday, August 5, 2024 16:45 To: user@flink.apache.org Subject: Managed memory and state size Hi All, As my incremental rocksdb check point size is increasing continuously, I am trying to

Re: flight buffer local storage

2024-07-21 Thread Zhanghao Chen
t the presence of backpressure. Unaligned checkpoint is introduced to solve this problem, where in-flight buffers are stored in cp without the need of alignment. Best, Zhanghao Chen From: Enric Ott <243816...@qq.com> Sent: Wednesday, July 17, 2024 16:

Re: SavepointReader: Record size is too large for CollectSinkFunction

2024-07-21 Thread Zhanghao Chen
Hi, you could increase it as follows: Configuration config = new Configuration(); config.setString(collect-sink.batch-size.max, "10mb"); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config); From: Salva Alcántara Sent: Satur

Re: Buffer Priority

2024-07-12 Thread Zhanghao Chen
Hi Enric, It basically means the prioritized buffers can bypass all non-prioritized buffers at the input gate and get processed first. You may refer to https://issues.apache.org/jira/browse/FLINK-19026 for more details where it is firstly introduced. Best, Zhanghao Chen

Re: Understanding flink-autoscaler behavior

2024-06-24 Thread Zhanghao Chen
You can try session mode with only one job, but still with adaptive scheduler disabled. When stopping a session job, the TMs won't be released immediately and can be reused later. Best, Zhanghao Chen From: Chetas Joshi Sent: Tuesday, June 25, 2024 1:

Re: Flink Stateful Functions 3.4

2024-06-17 Thread Zhanghao Chen
un/commits/master/ Best, Zhanghao Chen From: L. Jiang Sent: Tuesday, June 18, 2024 4:57 To: user@flink.apache.org Subject: Flink Stateful Functions 3.4 Hi there, Anyone knows which Flink version that Flink Stateful Functions 3.4 is compatible with? https://nightlies.

Re: Understanding flink-autoscaler behavior

2024-06-12 Thread Zhanghao Chen
anagement/#stateful-and-stateless-application-upgrades Best, Zhanghao Chen From: Chetas Joshi Sent: Thursday, June 13, 2024 6:33 To: Zhanghao Chen Cc: Sachin Sharma ; Gyula Fóra ; Oscar Perez via user Subject: Re: Understanding flink-autoscaler behavior Hi Zhang

Re: Failed to resume from HA when the checkpoint has been deleted.

2024-06-11 Thread Zhanghao Chen
ernal job monitoring system to manually recover it. Best, Zhanghao Chen From: Jean-Marc Paulin Sent: Tuesday, June 11, 2024 16:04 To: Zhanghao Chen ; user@flink.apache.org Subject: Re: Failed to resume from HA when the checkpoint has been deleted. Thanks for you

Re: Failed to resume from HA when the checkpoint has been deleted.

2024-06-10 Thread Zhanghao Chen
Hi, In this case, you could cancel the job using the flink stop​ command, which will clean up Flink HA metadata, and resubmit the job. Best, Zhanghao Chen From: Jean-Marc Paulin Sent: Monday, June 10, 2024 18:53 To: user@flink.apache.org Subject: Failed to

Re: Setting uid hash for non-legacy sinks

2024-06-09 Thread Zhanghao Chen
e the corresponding uidHash for each suboperator. Maybe you can further investigate it and fire a JIRA issue on it. Best, Zhanghao Chen From: Salva Alcántara Sent: Sunday, June 9, 2024 14:49 To: Gabor Somogyi Cc: user Subject: Re: Setting uid hash for non-legacy

Re: Understanding flink-autoscaler behavior

2024-06-07 Thread Zhanghao Chen
tlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#reactive-mode [2] https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.8/docs/custom-resource/autoscaler/ Best, Zhanghao Chen From: Sachin Sharma Sent: Saturday, June

Re: Force to commit kafka offset when stop a job.

2024-06-06 Thread Zhanghao Chen
Yes, the exact offset position will also be committed when doing the savepoint. Best, Zhanghao Chen From: Lei Wang Sent: Thursday, June 6, 2024 16:54 To: Zhanghao Chen ; ruanhang1...@gmail.com Cc: user Subject: Re: Force to commit kafka offset when stop a job

Re: Force to commit kafka offset when stop a job.

2024-06-05 Thread Zhanghao Chen
Hi, you could stop the job with a final savepoint [1]. Flink which will trigger a final offset commit on the final savepoint. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/#stopping-a-job-gracefully-creating-a-final-savepoint Best, Zhanghao Chen

Re: Task Manager memory usage

2024-06-04 Thread Zhanghao Chen
Hi Sigalit, Yes. Here, most of your memory is consumed by JVM heap and Flink network memory, both are somewhat like a pre-allocated memory pool managed by JVM/Flink Memory Manager, which typically do not return memory to the OS even if there's some free space internally. Best, Zhanghao

Re: Java 17 incompatibilities with Flink 1.18.1 version

2024-05-30 Thread Zhanghao Chen
ation. The default configuration in the Flink distribution has already been configured such that Flink itself works on Java 17. Best, Zhanghao Chen From: Rajat Pratap Sent: Thursday, May 30, 2024 13:17 To: Zhanghao Chen Subject: Re: Java 17 incompatibilities with Fl

Re: Java 17 incompatibilities with Flink 1.18.1 version

2024-05-28 Thread Zhanghao Chen
Hi Rajat, Flink releases are compiled with JDK 8 but it is able to run on JDK 8-17. As long as your Flink runs on JDK17 on both server and client side, you are free to write your Flink jobs with Java 17. Best, Zhanghao Chen From: Rajat Pratap Sent: Tuesday

Re: Would Java 11 cause Getting OutOfMemoryError: Direct buffer memory?

2024-05-23 Thread Zhanghao Chen
, Zhanghao Chen From: John Smith Sent: Thursday, May 23, 2024 22:40 To: Zhanghao Chen Cc: Biao Geng ; user Subject: Re: Would Java 11 cause Getting OutOfMemoryError: Direct buffer memory? Based on these two settings... taskmanager.memory.flink.size: 16384m

Re: Task Manager memory usage

2024-05-23 Thread Zhanghao Chen
ing a larger taskmanager.memory.jvm-overhead memory, and monitor it for a long time. If that's not the case, then there might be native memory leakage somewhere, but that may not be related to the state. Best, Zhanghao Chen From: Sigalit Eliazov Sent: Thursd

Re: Parallelism for auto-scaling, memory for auto-tuning - Flink operator

2024-04-17 Thread Zhanghao Chen
fficient in most cases. Best, Zhanghao Chen From: Maxim Senin via user Sent: Thursday, April 18, 2024 5:56 To: user@flink.apache.org Subject: Parallelism for auto-scaling, memory for auto-tuning - Flink operator Hi. Does it make sense to specify `parallelism`

Re: Flink job performance

2024-04-15 Thread Zhanghao Chen
instability persists? Best, Zhanghao Chen From: Oscar Perez Sent: Monday, April 15, 2024 19:24 To: Zhanghao Chen Cc: Oscar Perez via user Subject: Re: Flink job performance Hei, ok that is weird. Let me resend them. Regards, Oscar On Mon, 15 Apr 2024 at 14:00, Zhanghao

Re: Flink job performance

2024-04-15 Thread Zhanghao Chen
Hi, there seems to be sth wrong with the two images attached in the latest email. I cannot open them. Best, Zhanghao Chen From: Oscar Perez via user Sent: Monday, April 15, 2024 15:57 To: Oscar Perez via user ; pi-team ; Hermes Team Subject: Flink job

Re: Pyflink Performance and Benchmark

2024-04-15 Thread Zhanghao Chen
the JSON processing scenario with UDFs in Java/Python under thread mode/Python under process mode. Best, Zhanghao Chen From: Niklas Wilcke Sent: Monday, April 15, 2024 15:17 To: user Subject: Pyflink Performance and Benchmark Hi Flink Community, I wanted to reach

Re: Flink job performance

2024-04-15 Thread Zhanghao Chen
Hi Oscar, The rebalance operation will go over the network stack, but not necessarily involving remote data shuffle. For data shuffling between tasks of the same node, the local channel is used, but compared to chained operators, it still introduces extra data serialization overhead. For data s

Re: How to enable RocksDB native metrics?

2024-04-11 Thread Zhanghao Chen
Add a space between -yD and the param should do the trick. Best, Zhanghao Chen From: Lei Wang Sent: Thursday, April 11, 2024 19:40 To: Zhanghao Chen Cc: Biao Geng ; user Subject: Re: How to enable RocksDB native metrics? Hi Zhanghao, flink run -m yarn-cluster

Re: How to enable RocksDB native metrics?

2024-04-11 Thread Zhanghao Chen
Hi Lei, You are using an old-styled CLI for YARN jobs where "-yD" instead of "-D" should be used. From: Lei Wang Sent: Thursday, April 11, 2024 12:39 To: Biao Geng Cc: user Subject: Re: How to enable RocksDB native metrics? Hi Biao, I tried, it doesn't work

Re: Debugging Kryo Fallback

2024-04-09 Thread Zhanghao Chen
Hi, you may first enable the Kryo fallback option first, submit the job, and search for "be processed as GenericType". Flink will print it in most cases where we fall back to Kryo (a few exceptions including type Class, Object, recursive types, interface). Best, Zha

Re: One query just for curiosity

2024-03-28 Thread Zhanghao Chen
ge). Best, Zhanghao Chen From: Ganesh Walse Sent: Friday, March 29, 2024 10:42 To: Zhanghao Chen Cc: user@flink.apache.org Subject: Re: One query just for curiosity You mean to say we can process 32767 records in parallel. And may I know if this is the case

Re: Flink cache support

2024-03-28 Thread Zhanghao Chen
thout duplication, you might set up a Redis service externally for that purpose. Best, Zhanghao Chen From: Ganesh Walse Sent: Friday, March 29, 2024 4:45 To: user@flink.apache.org Subject: Flink cache support Hi Team, In my project my requirement is to cache data

Re: One query just for curiosity

2024-03-28 Thread Zhanghao Chen
Flink can be scaled up to a parallelism of 32767 at max. And if your record processing is mostly IO-bound, you can further boost the throughput via Async-IO [1]. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/asyncio/ Best, Zhanghao Chen

Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

2024-03-28 Thread Zhanghao Chen
Congratulations! Best, Zhanghao Chen From: Yu Li Sent: Thursday, March 28, 2024 15:55 To: d...@paimon.apache.org Cc: dev ; user Subject: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project CC the Flink user and dev mailing list. Paimon originated

Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR

2024-03-12 Thread Zhanghao Chen
Hi Sachin, Flink 1.8 series have already been out of support, have you tried with a newer version of Flink? From: Sachin Mittal Sent: Tuesday, March 12, 2024 14:48 To: user@flink.apache.org Subject: Facing ClassNotFoundException: org.apache.flink.api.common.Exe

Re: Re: Running Flink SQL in production

2024-03-08 Thread Zhanghao Chen
r now, but there's some on-going effort [2]. Hopefully, it would be much easier to do so in the future. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/#application-mode [2] https://issues.apache.org/jira/browse/FLINK-26541 Best, Zha

Re: Question about time-based operators with RocksDB backend

2024-03-04 Thread Zhanghao Chen
an be satisfied with the reduce/aggregate function pattern, which is important for large windows. Best, Zhanghao Chen From: Gabriele Mencagli Sent: Monday, March 4, 2024 19:38 To: user@flink.apache.org Subject: Question about time-based operators with RocksDB ba

Re: Task Manager getting killed while executing sql queries.

2024-02-16 Thread Zhanghao Chen
Hi Kanchi, Could you provide with more information on it? Like at what stage this log prints (job recovering, running, etc), any more detailed job or stacktrace. Best, Zhanghao Chen From: Kanchi Masalia via user Sent: Friday, February 16, 2024 4:07 To: Neha

Re: Flink use case feedback request

2024-02-16 Thread Zhanghao Chen
with. Best, Zhanghao Chen From: Brent Sent: Saturday, February 17, 2024 3:01 To: user@flink.apache.org Subject: Flink use case feedback request Hey everyone, I've been looking at Flink to handle a fairly complex use case and was hoping for some feedback

Re: Parallelism and Target TPS

2024-02-01 Thread Zhanghao Chen
Hi Patricia, Flink will create one Kafka consumer per parallelism, however, you'll need some testing to measure the capability of a single task. Usu, one consumer can consume at a much higher rate than 1 record per second. Best, Zhanghao Chen From: patrici

Re: Apache Flink lifecycle and Java 17 support

2024-01-28 Thread Zhanghao Chen
d Java 17 in production, including ByteDance as mentioned by Xiangyu. Best, Zhanghao Chen From: Deepti Sharma S via user Sent: Friday, January 26, 2024 22:56 To: xiangyu feng Cc: user@flink.apache.org Subject: RE: Apache Flink lifecycle and Java 17 support Hel

Re: Query on using two sinks for a Flink job (Flink SQL)

2023-12-06 Thread Zhanghao Chen
Hi Elakiya, You can try executing TableEnvironmentImpl#executeInternal for non-insert statements, then using StatementSet.addInsertSql to add multiple insertion statetments, and finally calling StatementSet#execute. Best, Zhanghao Chen From: elakiya udhayanan

Re: Flink Kubernetes HA

2023-12-06 Thread Zhanghao Chen
lection is performed with a unified config map for doing that. Best, Zhanghao Chen From: Ethan T Yang Sent: Wednesday, December 6, 2023 5:40 To: user@flink.apache.org Subject: Flink Kubernetes HA Hi Flink users, After upgrading Flink ( from 1.13.1 -> 1.18.0), I n

Re: Flink 1.18 and java 17

2023-11-27 Thread Zhanghao Chen
Hi Lasse, The default flink-conf.yaml file bundled in the distribution should already have a preset env.java.opts.all config for Java 17. Have you tried that? Best, Zhanghao Chen From: Lasse Nedergaard Sent: Monday, November 27, 2023 21:20 To: user Subject

Re: Operator ids

2023-11-25 Thread Zhanghao Chen
It is not supported yet. Curious why do you need to get the operator IDs? They are usually only used internally. Best, Zhanghao Chen From: rania duni Sent: Saturday, November 25, 2023 20:44 To: user@flink.apache.org Subject: Operator ids Hello! I would like

Re: [DISCUSS][FLINK-33240] Document deprecated options as well

2023-11-01 Thread Zhanghao Chen
cated because the related API/module is entirely deprecated, e.g. DataSet API. In that case, users would not search for replacement on an individual option but rather need to migrate to a new API, and it would be better to move these options to a separate section. WDYT? Best, Zhanghao

Re: [DISCUSS][FLINK-33240] Document deprecated options as well

2023-11-01 Thread Zhanghao Chen
options. Best, Zhanghao Chen From: Alexander Fedulov Sent: Tuesday, October 31, 2023 18:12 To: d...@flink.apache.org Cc: user@flink.apache.org Subject: Re: [DISCUSS][FLINK-33240] Document deprecated options as well Hi Zhanghao, Thanks for the proposition. In general

[DISCUSS][FLINK-33240] Document deprecated options as well

2023-10-11 Thread Zhanghao Chen
s. I propose to document deprecated options as well, with a "(deprecated)" tag placed at the beginning of the option description to highlight the deprecation status [1]. Looking forward to your feedbacks on it. [1] https://issues.apache.org/jira/browse/FLINK-33240 Best, Zhanghao Chen

Re: Rescale Flink - Adjust Parallelism based on TPS

2023-01-28 Thread Zhanghao Chen
deployments. Too few resources lead to unstable Flink jobs which requires users to scale up their deployments manually. cwiki.apache.org Best, Zhanghao Chen From: Madan D via user Sent: Saturday, January 28, 2023 8:56 To: user@flink.apache.org Subject: Rescale Fli

Re: flink on yarn job always restart

2022-07-18 Thread Zhanghao Chen
Hi, could you provide the whole JM log? Best, Zhanghao Chen From: SmileSmile Sent: Monday, July 18, 2022 20:46 To: user Subject: flink on yarn job always restart hi all we meet a situation, parallelism 3000,the job contains multiple agg operation,the job

Re: Can FIFO compaction with RocksDB result in data loss?

2022-07-04 Thread Zhanghao Chen
d you manually enabled allow compaction option for FIFO? Best, Zhanghao Chen From: Alexander Fedulov Sent: Tuesday, July 5, 2022 5:49 To: Vishal Surana Cc: user Subject: Re: Can FIFO compaction with RocksDB result in data loss? Hi Vishal, I am not sure I get what you

Re: How to test Flink SQL UDF with open method?

2022-05-12 Thread Zhanghao Chen
Hi, What kind of parameters do you want to get, Flink global job parameters, or some other parameters? Best, Zhanghao Chen From: zhouhaifengmath Sent: Thursday, May 12, 2022 14:33 To: user@flink.apache.org Subject: How to test Flink SQL UDF with open method

Configuring heap size for Flink client

2022-05-11 Thread Zhanghao Chen
to be used (I'm mostly using Per-Job mode where job graph is generated on the client side)? I'd appreciate a lot if anyone could give some insights on this. Thanks, Zhanghao Chen

Re: About job execution

2022-05-03 Thread Zhanghao Chen
fecycle is bound to the job's lifecycle, so yes, after the job is finished, its jobmaster will be shutdown. Taskmanagers, on the other hand, are shared among jobs. Hope that helps. Best, Zhanghao Chen From: Ww J Sent: Monday, May 2, 2022 14:15 To: user Sub

Re: Adpative scheduler is not recognizing by Flink 1.14.0

2022-04-18 Thread Zhanghao Chen
t;flink run -m yarn-cluster -ynm jobName". I suggest you allocating enough resources first and lanuching the JM. Then check on the Flink WebUI - JobManager - Configuration page if the configurations are recognized and try again. Best, Zhanghao Chen

Re: java.lang.Exception: Job leader for job id 0efd8681eda64b072b72baef58722bc0 lost leadership.

2022-04-16 Thread Zhanghao Chen
Hi, to unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org<mailto:user-unsubscr...@flink.apache.org>. Best, Zhanghao Chen From: Samir Vasani Sent: Saturday, April 16, 2022 1:08:20 PM To: Jai Patel Cc: Nicolaus Weidner ; user ; Weixia

Re: Why first op after union cannot be chained?

2022-04-06 Thread Zhanghao Chen
Thanks Guowei! I'll check it out. Best, Zhanghao Chen From: Guowei Ma Sent: Wednesday, April 6, 2022 16:01 To: Zhanghao Chen Cc: user@flink.apache.org Subject: Re: Why first op after union cannot be chained? Hi Zhanghao AFAIK, you might to se

Re: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

2022-04-05 Thread Zhanghao Chen
Best, Zhanghao Chen From: Burcu Gul POLAT EGRI Sent: Tuesday, April 5, 2022 22:11 To: Zhanghao Chen Subject: RE: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way) Thank you very much Zhanghao. Are you planning to add capabili

Why first op after union cannot be chained?

2022-04-05 Thread Zhanghao Chen
in the code piece for checking the eligibility of chaining in JobGraphGenerator::createSingleInputVertex: "first op after union is stand-alone, because union is merged" that could be relevant, but I'm not sure what it means. Could anyone enlighten me how to understand this? Best, Zhanghao Chen

Re: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

2022-04-05 Thread Zhanghao Chen
FYI, I've just created a new JIRA ticket on extending /jars/:jarid/run API to support setting Flink configs: https://issues.apache.org/jira/browse/FLINK-27060. Best, Zhanghao Chen From: Burcu Gul POLAT EGRI Sent: Tuesday, April 5, 2022 18:20 To: Zhanghao

Re: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

2022-04-05 Thread Zhanghao Chen
s a REST-ful API that accepts HTTP requests and responds with JSON data. nightlies.apache.org  Hope that helps. Best, Zhanghao Chen From: Burcu Gul POLAT EGRI Sent: Tuesday, April 5, 2022 18:20 To: Zhanghao Chen ; user@flink.apache.org Subject: RE: Submit jobs vi

Re: Submit jobs via Rest API and deploy Flink on a running Kubernetes cluster (Native way)

2022-04-05 Thread Zhanghao Chen
or Flink client Java API and provides a RESTful API to users. You could also checkout the newly released Flink Kubernetes Operator (not sure if the doc is ready yet, you can search that in the mailing list) to see if it could be integrated it into your service.

Re: Naming sql_statment job

2022-03-30 Thread Zhanghao Chen
y to adjust certain parameters for optimization. For example, unbounded streaming programs may need to ensure that the required state size is capped (see streaming concepts). nightlies.apache.org Best, Zhanghao Chen From: lan tran Sent: Wednesday,

Re: Flink STOP with savepoint

2022-03-24 Thread Zhanghao Chen
Hi Diwakar, The client log doesn't contain much useful info except that the operation timed out. You could try: 1. Check the JM log to see if there is any relevant info. 2. Increase the client timeout to see if that helps. Best, Zhanghao Chen

Re: Use existing Kafka zookeeper for Flink

2022-03-14 Thread Zhanghao Chen
Hi, This will make the ZK cluster a single point of failure for your overall system. But other than that, as long as ZK is not overloaded (that will depend on the spec of your ZK cluster as well as the scale of your Kafka & Flink clusters), it's totally fine to do that. Best, Zhan

Re: Shaded zookeeper - curator mismatch?

2022-03-07 Thread Zhanghao Chen
classpath to see if a higher version of ZK client is mistakenly packaged. Hope that helps. Best, Zhanghao Chen From: Filip Karnicki Sent: Monday, March 7, 2022 16:49 To: Zhanghao Chen Cc: user Subject: Re: Shaded zookeeper - curator mismatch? Hi Zhanghao

Re: Question about Flink counters

2022-03-05 Thread Zhanghao Chen
series database) to do aggregations. Also, you cannot reference a counter from a different machine. I'm not sure if this is what you expected. Best, Zhanghao Chen From: Shane Bishop Sent: Saturday, March 5, 2022 23:22 To: Zhanghao Chen ; user@flink.apach

Re: Shaded zookeeper - curator mismatch?

2022-03-04 Thread Zhanghao Chen
Hi Filip, Could you share the version of the ZK server you are connecting to? Best, Zhanghao Chen From: Filip Karnicki Sent: Friday, March 4, 2022 23:12 To: user Subject: Shaded zookeeper - curator mismatch? Hi, I believe there's a mismatch in s

Re: Question about Flink counters

2022-03-04 Thread Zhanghao Chen
t the second example of Metrics | Apache Flink<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/#counter> for reference. Best, Zhanghao Chen From: Shane Bishop Sent: Saturday, March 5, 2022 5:24 To: user@flink.apache.org Subjec