Re: Urgent: Mitigating Slow Consumer Impact and Seeking Open-SourceSolutions in Apache Kafka Consumers

Hi Karthick, We’ve experienced the similar issue before. What we were doing at that time was to define multiple topics and each has a different # of partitions which means some of the topics with more partitions will have the high parallelisms for processing. And you can further divide the topic

A potential bug for Kubernetes HA in Flink 1.15.1

A, and the issue no longer persisted. Let me know what you think. Best regards, Wei

akka.remote.OversizedPayloadException after we upgrade to Flink 1.15

und 128 MB when we hit the akka error). I'd like to see if there's a known root cause for this problem and what can we do here to eliminate it? Best, Wei

Re: Can I setup standby taskmanagers while using reactive mode?

eactive mode doesn't support standby taskmanagers. As you said it >> always uses all available resources in the cluster. >> >> I can see it being useful though to not always scale to MAX but (MAX - >> some_offset). >> >> I'd suggest to file a

Can I setup standby taskmanagers while using reactive mode?

standby task managers as active task manager to process data. I wonder if you have any suggestion on this, should we avoid using Flink reactive mode together with standby task managers? Best, Wei

Re: is there any detrimental side-effect if i set the max parallelism as 32768

Hi Hangxiang, David, Thank you for your replies. Your responses are very helpful. Best regards, Tony Wei David Anderson 於 2023年3月14日 週二 下午12:12寫道： > I believe there is some noticeable overhead if you are using the > heap-based state backend, but with RocksDB I think the differe

is there any detrimental side-effect if i set the max parallelism as 32768

much between each subtask. I'm wondering if this is a good practice, because based on the official document it is not recommended actually. If possible, I would like to know the detail about this side-effect. Which state backend will have this issue? and Why? Please give me an advice. Thanks in

Re: OOM errors cause by the new KafkaSink API

e time, the control group is very stable. We have run it over 5 days and very smoothly. Best Regards, Hua Wei On Tue, May 10, 2022 at 7:16 PM Martijn Visser wrote: > Hi Hua Wei, > > Have you built your own Flink version? Since Flink doesn't support Scala > 2.12.12, the late

Re: OOM errors cause by the new KafkaSink API

ine which of the two steps cause the problem? Because the new Kafka API needs the new serializer (KafkaRecordSerializationSchema) and seems like cannot use the old one (KafkaSerializationSchema), we cannot separate the change into two steps. Best Regards, Hua Wei On Tue, Apr 26, 2022 at 5:03 P

Flink team staffing

ave a couple of Big Data and a couple of MLE engineers. Are we in good shape? Would love to hear your thoughts. -Wei -- 206-430-3317

Re: OOM errors cause by the new KafkaSink API

rsistent OOM, you can dump the memory and check which part > is taking up memory. > > > 2022年4月25日 上午11:44，Hua Wei Chen 写道： > > Hi all, > > Due to FlinkKafkaConsumer and FlinkKafkaProducer will be depreciated at > Flink 1.15*[1]*, we are trying to migrate the APIs to K

OOM errors cause by the new KafkaSink API

/java.io.ObjectOutputStream.close(Unknown Source) at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:635) ... 15 more ``` -- *Regards,* *Oscar / Chen Hua Wei*

How to accelerate state processor with a large savepoint

Hi team, We want to try to use state processor APIs[1] to clean up some legacy states. Here are our steps: 1. Create a new savepoint (~= 1.5TB) 2. Submit state processor jobs 3. Write results to a new savepoint We create 8 task managers with 120 slots to execute it. Here are the related configur

Re: Not cleanup Kubernetes Configmaps after execution success

like KubernetesApplicationClusterEntrypoint is re-started in >> the middle of shutdown and, as a result, the resources it (re)creates >> aren't clean up. >> >> Could you please also share Kubernetes logs and resource definitions >> to validate the above assumption? &

Not cleanup Kubernetes Configmaps after execution success

Hi all, We have Flink jobs run on batch mode and get the job status via JobHandler. onJobExecuted ()[

Re: confused about `TO_TIMESTAMP` document description

Hi Leonard, Thanks for confirmation. I have created the jira ticket [1]. The pull request will be submitted later. best regards, [1] https://issues.apache.org/jira/browse/FLINK-22970 Leonard Xu 於 2021年6月10日 週四 下午8:58寫道： > Hi，Tony > > > I found this code snippet [2] might be related to `TO_TIM

confused about `TO_TIMESTAMP` document description

Hi Expert, this document [1] said `TO_TIMESTAMP` will use the session time zone to convert date time string into a timestamp. If I understand correctly, when I set session time zone to `Asia/Shanghai` and query `SELECT TO_TIMESTAMP('1970-01-01 08:00:00');`, the result should be epoch timestamp `0`

Re: when should `FlinkYarnSessionCli` be included for parsing CLI arguments?

Hi Till, I have created the ticket to extend the description of `execution.targe`. https://issues.apache.org/jira/browse/FLINK-22476 best regards, Tony Wei 於 2021年4月26日 週一 下午5:24寫道： > Hi Till, Yangze, > > I think FLINK-15852 should solve my problem. > It is my fault that my flin

Re: when should `FlinkYarnSessionCli` be included for parsing CLI arguments?

t;> > Cheers, >> > Till >> > >> > On Mon, Apr 26, 2021 at 8:37 AM Yangze Guo wrote: >> >> >> >> Hi, Tony. >> >> >> >> What is the version of your flink-dist. AFAIK, this issue should be >> >> addressed in F

when should `FlinkYarnSessionCli` be included for parsing CLI arguments?

Hi Experts, I recently tried to run yarn-application mode on my yarn cluster, and I had a problem related to configuring `execution.target`. After reading the source code and doing some experiments, I found that there should be some room of improvement for `FlinkYarnSessionCli` or `AbstractYarnCli

Re: Install/Run Streaming Anomaly Detection R package in Flink

environment: https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/python/faq.html#preparing-python-virtual-environment <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/python/faq.html#preparing-python-virtual-environment> Best, Wei > 在 2021年2月24日，01:38，Roman Kha

Re: Initializing broadcast state

Hi guys, i meet the same question, but i use a different way to init: ``` val list = ... //i use jdbc to get the init data val dimensionInitStream = env.fromCollection(list) //the main stream and the `dimensionStream` is a stream from flink cdc val dimension = dimensionStream.union(dimensionInit

Re: [ANNOUNCE] Apache Flink 1.12.1 released

Thanks Xintong for the great work! Best, Wei > 在 2021年1月19日，18:00，Guowei Ma 写道： > > Thanks Xintong's effort! > Best, > Guowei > > > On Tue, Jan 19, 2021 at 5:37 PM Yangze Guo <mailto:karma...@gmail.com>> wrote: > Thanks Xintong for the great work! &g

Re: Urgent help on S3 CSV file reader DataStream Job

Hi Deep, You can try to change the `FileProcessingMode.PROCESS_ONCE` to `FileProcessingMode.PROCESS_CONTINUOUSLY`. Best, Wei > 在 2020年12月15日，20:18，DEEP NARAYAN Singh 写道： > > Hi Wei, > Could you please suggest , how to fix this below issues. > > Thanks & Regards, >

Re: [ANNOUNCE] Apache Flink 1.12.0 released

Congratulations! Thanks Dian and Robert for the great work! Best, Wei > 在 2020年12月10日，20:26，Leonard Xu 写道： > > > Thanks Dian and Robert for the great work as release manager ! > And thanks everyone who makes the release possible ! > > > Best, > Leonard >

Re: Urgent help on S3 CSV file reader DataStream Job

of the Row, you do not need to make the above changes, because at this time we really need the TypeInformation array to contain two StringTypeInfo elements. Best, Wei > 在 2020年12月8日，19:29，DEEP NARAYAN Singh 写道： > > Hi Wei, > > Also I need to know how to get file names al

Re: Urgent help on S3 CSV file reader DataStream Job

Hi Deep, Could you show your current code snippet? I have tried the Csv file data on my local machine and it works fine, so I guess what might be wrong elsewhere. Best, Wei > 在 2020年12月8日，03:20，DEEP NARAYAN Singh 写道： > > Hi Wei and Till, > Thanks for the quick reply. > >

Re: Urgent help on S3 CSV file reader DataStream Job

(); Then you can convert the content of the csv files to json manually. Best, Wei > 在 2020年12月7日，19:10，DEEP NARAYAN Singh 写道： > > Hi Guys, > > Below is my code snippet , which read all csv files under the given folder > row by row but my requirement is to read csv file a

Re: PyFlink - Scala UDF - How to convert Scala Map in Table API?

serializers. Best, Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/types.html#raw <https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/types.html#raw> [2] https://stackoverflow.com/questions/28157236/kryo-serialization-with-nested-hashmap-with-

Re: PyFlink - Scala UDF - How to convert Scala Map in Table API?

yMap") # prepare source and sink t = st_env.from_elements([(1, 'hi', 'hello'), (2, 'hi', 'hello')], ['a', 'b', 'c']) st_env.execute_sql("""create table mySink ( output_of_my_scala_udf ROW ) with ( 'con

Re: PyFlink - Scala UDF - How to convert Scala Map in Table API?

ON" sql statement, which accepts the type hint. Best, Wei > 在 2020年11月17日，19:29，Pierre Oberholzer 写道： > > Hi Wei, > > Thanks for your suggestion. Same error. > > Scala UDF > > @FunctionHint(output = new DataTypeHint("ROW")) > class du

Re: PyFlink - Scala UDF - How to convert Scala Map in Table API?

Hi Pierre, You can try to replace the '@DataTypeHint("ROW")' with '@FunctionHint(output = new DataTypeHint("ROW”))' Best, Wei > 在 2020年11月17日，15:45，Pierre Oberholzer 写道： > > Hi Dian, Community, > > (bringing the thread back to wider

Re: [ANNOUNCE] New PMC member: Dian Fu

Congratulations Dian! > 在 2020年8月28日，14:29，Jingsong Li 写道： > > Congratulations , Dian! > > Best, Jingsong > > On Fri, Aug 28, 2020 at 11:06 AM Walter Peng > wrote: > congrats! > > Yun Tang wrote: > > Congratulations , Dian! > > > -- > Best, Jingsong Lee

Flink maxrecordcount increase causing a few task manager throughput drops

hi, I change the following config from flink.shard.getrecords.maxrecordcount: 1000 flink.shard.getrecords.intervalmillis: 200 to flink.shard.getrecords.maxrecordcount: 1 flink.shard.getrecords.intervalmillis: 1000 and found a few task managers around (10/1000) are becoming very slow. We als

Re: [DISCUSS] FLIP-133: Rework PyFlink Documentation

n and progress of the redesigning in the Spark community. It is so similar to our working that there should be some ideas worthy for us. Best, Wei > 在 2020年8月5日，15:02，Xingbo Huang 写道： > > Hi, > > I found that the spark community is also working on redesigning pyspark > documentatio

is it possible one task manager stuck and still fetching data from Kinesis?

We are running Flink 1.10 about 900+ task managers with kinesis as an input stream. The problem we are having now is that only Max Age of kinesis shard is growing and the average age of that kinesis is very low meaning most of shards having very low age. We already checked the data skew issue but i

Re: PyFlink DDL UDTF join error

Hi Manas, It seems a bug of the create view operation. I have created a JIRA for it: https://issues.apache.org/jira/browse/FLINK-18750 <https://issues.apache.org/jira/browse/FLINK-18750> Before repairing, please do not use create view operation for udtf call. Best, Wei > 在 2020年7月

Re: PyFlink DDL UDTF join error

ll try to find out what caused this exception. Best, Wei > 在 2020年7月28日，18:33，Manas Kale 写道： > > Hi, > Using pyFlink DDL, I am trying to: > Consume a Kafka JSON stream. This has messages with aggregate data, example: > "data": > "{\"0001\":105.0,\

Re: [ANNOUNCE] Apache Flink 1.11.1 released

Congratulations! Thanks Dian for the great work! Best, Wei > 在 2020年7月22日，15:09，Leonard Xu 写道： > > Congratulations! > > Thanks Dian Fu for the great work as release manager, and thanks everyone > involved! > > Best > Leonard Xu > >> 在 2020年7月22日，14:52，

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

`pyflink-shell.sh local` and try the examples in the help message, run well and no exception. - Try a word count example in IDE with Python 2.7.15 and Python 3.7.5, run well and no exception. Best, Wei > 在 2020年2月10日，19:12，jincheng sun 写道： > > Hi everyone, > > Please review

Re: [DISCUSS] Upload the Flink Python API 1.9.x to PyPI for user convenience.

Hi Jincheng, Thanks for bring up this discussion! +1 for this proposal. Building from source takes long time and requires a good network environment. Some users may not have such an environment. Uploading to PyPI will greatly improve the user experience. Best, Wei jincheng sun 于2020年2月4日周二 上午

Re: [ANNOUNCE] Dian Fu becomes a Flink committer

Congrats Dian Fu! Well deserved! Best, Wei > 在 2020年1月16日，18:10，Hequn Cheng 写道： > > Congratulations, Dian. > Well deserved! > > Best, Hequn > > On Thu, Jan 16, 2020 at 6:08 PM Leonard Xu <mailto:xbjt...@gmail.com>> wrote: > Congratulations! Dian Fu >

Re: [ANNOUNCE] Apache Flink 1.8.3 released

Thanks Hequn for being the release manager. Great work! Best, Wei > 在 2019年12月12日，15:27，Jingsong Li 写道： > > Thanks Hequn for your driving, 1.8.3 fixed a lot of issues and it is very > useful to users. > Great work! > > Best, > Jingsong Lee > > On Thu, Dec 12

Re: ***UNCHECKED*** Error while confirming Checkpoint

Hi Piotrek, There was already an issue [1] and PR for this thread. Should we mark it as duplicated or related issue? Best, Tony Wei [1] https://issues.apache.org/jira/browse/FLINK-10377 Piotr Nowojski 於 2019年11月28日 週四 上午12:17寫道： > Hi Tony, > > Thanks for the explanation. Assumi

Re: ***UNCHECKED*** Error while confirming Checkpoint

esney said. The following checkpoint succeeded before the previous savepoint, handling both of their pending transaction, but savepoint still succeeded and sent the notification to each TaskManager. That led to this exception. Could you double check if this is the case? Thank you. Best, Tony Wei

Re: ***UNCHECKED*** Error while confirming Checkpoint

Hi, As the follow up, it seem that savepoint can't be subsumed, so that its notification could still be send to each TMs. Is this a bug that need to be fixed in TwoPhaseCommitSinkFunction? Best, Tony Wei Tony Wei 於 2019年11月27日 週三 下午3:43寫道： > Hi, > > I want to raise this questio

Re: ***UNCHECKED*** Error while confirming Checkpoint

ier and led to this exception happened, because there was no pending transaction in queue. Does anyone know the details about subsumed notifications mechanism and how checkpoint coordinator handle this situation? Please correct me if I'm wrong. Thanks. Best, Tony Wei Stefan Richter 於 2018年10

Questions about how to use State Processor API

Thanks in advance. Best, Tony Wei

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

action` list is empty before executing `enqueueNewPartitions` function. Am I right? If it can be confirmed as a bug, I would like to submit my patch to fix it. Thanks for your help. Best, Tony Wei [1] https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L20

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

x27;s coordinator, since the empty transaction won't make any request to server. The attachments are my simple producer code. Please help to verify what I thought is correct. Thanks. Best, Tony Wei [1] https://github.com/apache/kafka/blob/c0019e653891182d7a95464175c9b4ef63f8bae1/clients/src/mai

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

? Is there any expert who is familiar with both kafka and flink's kafka connector could help me solve this? Thanks very much. The attachment is my code to reproduce this problem. The cluster's versions are the same as I mentioned in my first email. Best, Tony Wei *flink taskmanager:*

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

r's behavior. I tried to use kafka java producer to reproduce the exception, but it worked well. Maybe my observation is not correct, but the experiment result seems like that. Do you have any thoughts on this? Best, Tony Wei Tony Wei 於 2019年9月19日 週四 上午11:08寫道： > Hi Becket, > > On

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

Hi Becket, One more thing, I have tried to restart other brokers without active controller, but this exception might happen as well. So it should be independent of the active controller like you said. Best, Tony Wei Tony Wei 於 2019年9月18日 週三 下午6:14寫道： > Hi Becket, > > I have reprod

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

8 07:13:42,779] DEBUG [TransactionCoordinator id=2] Returning > NOT_COORDINATOR error code to client for blacklist -> Sink: > kafka-sink--eba862242e60de7e4744f3307058f865-7's AddPartitions request > (kafka.coordinator.transaction.TransactionCoordinator) > [2019-09-18 07:13:43,633] DEBUG

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

to find out what's wrong about my kafka producer. Could someone help me please? Best, Tony Wei Fabian Hueske 於 2019年8月16日 週五 下午4:10寫道： > Hi Tony, > > I'm sorry I cannot help you with this issue, but Becket (in CC) might have > an idea what went wrong here. > > Best

Kafka producer failed with InvalidTxnStateException when performing commit transaction

nt checkpoint: 1 max checkpoint duration before and after the exception occurred: < 2 minutes Best, Tony Wei

Re: Kafka ProducerFencedException after checkpointing

Hi Piotr, Thanks a lot. I need exactly once in my use case, but instead of having the risk of losing data, at least once is more acceptable when error occurred. Best, Tony Wei Piotr Nowojski 於 2019年8月12日 週一 下午3:27寫道： > Hi, > > Yes, if it’s due to transaction timeout you will lose

Re: Kafka ProducerFencedException after checkpointing

Hi, I had the same exception recently. I want to confirm that if it is due to transaction timeout, then I will lose those data. Am I right? Can I make it fall back to at least once semantic in this situation? Best, Tony Wei Piotr Nowojski 於 2018年3月21日 週三 下午10:28寫道： > Hi, > > B

Re: How to make two SQLs use the same KafkaTableSource?

is fine with me. My original question just focused on reused nodes in SQL api, and seems Blink planner is what I need. Thanks for your help again. Best, Tony Wei Zhenghua Gao 於 2019年8月9日 週五 下午1:54寫道： > Blink planner support lazy translation for multiple SQLs, and the common > nodes will

Re: How to make two SQLs use the same KafkaTableSource?

forgot to send to user mailing list. Tony Wei 於 2019年8月9日 週五 下午12:36寫道： > Hi Zhenghua, > > I didn't get your point. It seems that `isEagerOperationTranslation` is > always return false. Is that > means even I used Blink planner, the sql translation is still in a lazy >

How to make two SQLs use the same KafkaTableSource?

ka. A workaround might be registering the same kafka topic twice with different name, group_id for two SQLs. But I would still like to know if there is any way to make two SQLs just read from the same KafkaTableSource? Thanks in advance. Best, Tony Wei

Re: Is it possible to decide the order of where conditions in Flink SQL

true > END Best regards, Tony Wei Hequn Cheng 於 2019年7月28日 週日 下午3:30寫道： > Hi Tony, > > There is no order guarantee for filter conditions. The conditions would be > pushed down or merged during query optimization. > > However, you can use the case when[1] to achieve what you want

Re: Is it possible to decide the order of where conditions in Flink SQL

operator of physical plan have any meaning to represent the execution order of `where` conditions? Best, Tony Wei sri hari kali charan Tummala 於 2019年7月27日 週六 上午3:02寫道： > try cte common table expressions if it supports or sql subquery. > > On Fri, Jul 26, 2019 at 1:00 PM Fanbin Bu w

Is it possible to decide the order of where conditions in Flink SQL

`!user.is_robot` first then executing UDF, it will reduce the number of database access. Those records with `true` in `user.is_robot` will be dropped earlier and don't need to access database. select * from users where !user.is_robot and UDF_NEED_TO_QUERY_DB(user) Thanks, Tony Wei

Re: left join failing with FlinkLogicalJoinConverter NPE

Hi, I also found the similar issue here [1]. Best, Tony Wei [1] https://issues.apache.org/jira/browse/FLINK-11433 Tony Wei 於 2019年7月19日 週五 下午5:38寫道： > Hi, > > Is there any update for this issue? I have had the same problem just like > Karl's. > After I remove query like

Re: left join failing with FlinkLogicalJoinConverter NPE

Hi, Is there any update for this issue? I have had the same problem just like Karl's. After I remove query like "select collect(data) ..." from one of the joined tables, the sql can be executed correctly without throwing any NPE. Best regards, Tony Wei Xingcan Cui 於 2019年2月2

Question about counting top k on streaming data

n't work here in streaming mode, is there any other optimization can apply in this case? It is not necessary to focus on SQL only. Any improvement on DataStream is also welcome. Thank you. Best Regards, Tony Wei

Re: What does "Continuous incremental cleanup" mean in Flink 1.8 release notes

red states should be clean up as well based on Flink 1.6's implementation. Am I right? Best, Tony Wei Konstantin Knauf 於 2019年3月9日 週六 上午7:00寫道： > Hi Tony, > > before Flink 1.8 expired state is only cleaned up, when you try to access > it after expiration, i.e. when user code tries

What does "Continuous incremental cleanup" mean in Flink 1.8 release notes

ecuting TTL mechanism. Could you give me more references to learn about it? A simple example to illustrate it is more appreciated. Thank you. Best, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-master/release-notes/flink-1.8.html#state

Re: Building Flink from source according to vendor-specific versionbut causes protobuf conflict

k it later. Thanks again. Best Regards Wei -- Original -- From: "Timo Walther";; Date: Jan 8, 2019 To: "user"; Cc: "gary"; Subject: Re: Building Flink from source according to vendor-specific versionbut causes protob

Building Flink from source according to vendor-specific version but causes protobuf conflict

Hi guys, Good day. I rebuilt flink from the source and specified the vendor specific Hadoop version. It works well when i just submit a streaming application without '-d'(--detached) option as follows: bin/flink run -m yarn-cluster -yqu root.streaming -yn 5 -yjm 2048 -ytm 3096 -ynm CJVForma

Re: how to override s3 key config in flink job

Hi Andrey, Thanks for your detailed answer, and I have created a JIRA issue to discuss it [1]. Please check the description and help me to fill the details, like component/s, since I'm not sure where it should be put. Thank you very much. Best, Tony Wei [1] https://issues.apache.org/jira/b

Re: how to override s3 key config in flink job

Hi yinhua, Our flink version is 1.6.0. Best, Tony Wei yinhua.dai 於 2018年11月27日 週二 下午2:32寫道： > Which flink version are you using. > I know how it works in yarn, but not very clear with standalone mode. > > > > -- > Sent from: > http://apache-flink-user-mailing

Re: how to override s3 key config in flink job

to submit different flink applications with different s3 key for flink presto s3 filesystem. Any other suggestions are also welcome. Thank you. Best, Tony Wei yinhua.dai 於 2018年11月27日 週二 上午11:37寫道： > Did you try "-Dkey=value"? > > > > -- > Sent from: > http

Re: Reset kafka offets to latest on restart

write your own implementation with kafka client and always seek to the latest position when the job begin to run. Best, Tony Wei Vishal Santoshi 於 2018年11月25日 週日 上午4:51寫道： > I think I can set . a new uuid but it seems `allowNonRestoreState` is a > CLI hint. I need the "automatic" restart

Re: how to override s3 key config in flink job

Hi, Is there anyone can answer me? Thanks, Tony Wei Tony Wei 於 2018年11月20日 週二 下午7:39寫道： > Hi, > > Is there any way to provide s3.access-key and s3.secret-key in flink > application, instead of setting > them in flink-conf.yaml? > > In our use case, my team provide a fli

Re: Reset kafka offets to latest on restart

the way that Rong provided to set the initial start position. cc. Gordon who know more about the details of kafka source. Best, Tony Wei Rong Rong 於 2018年11月22日 週四 上午8:23寫道： > Hi Vishal, > > You can probably try using similar offset configuration as a service > consumer. > May

how to override s3 key config in flink job

store checkpoints. So, we want to know if is it feasible to let users provide their checkpoint path and corresponding aws key to access their own s3 bucket? If not, could you show me why it doesn't work currently? And, is it possible to become a new feature? Thanks in advance for your help. Best, Tony Wei

sys.exist(1) led to standalonesession daemon closed

he.flink.runtime.blob.TransientBlobCache - Shutting > down BLOB cache > 2018-11-05 13:28:04,761 INFO org.apache.flink.runtime.blob.BlobServer > - Stopped BLOB server at 0.0.0.0:42075 Best, Tony Wei. [1] https://github.com/scallop/scallop

Re: How to get Cluster metrics in FLIP-6 mode

Hi Gary, Thanks for your information. Best, Tony Wei 2018-09-11 20:26 GMT+08:00 Gary Yao : > Hi Tony, > > You are right that these metrics are missing. There is already a ticket for > that [1]. At the moment you can obtain these information from the REST API > (/overview) [2]. &

How to get Cluster metrics in FLIP-6 mode

these metrics? Or did I miss something? Thank you. Best Regards, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html#cluster

Question about akka configuration for FLIP-6

that FLIP-6 tried to get rid of akka and use its own rpc interface. Please correct me if I misunderstood. Thanks. akka.watch.heartbeat.interval akka.watch.heartbeat.pause taskmanager.exit-on-fatal-akka-error Best Regards, Tony Wei

Re: checkpoint failed due to s3 exception: request timeout

Hi Andrey, Cool! I will add it in my flink-conf.yaml. However, I'm still wondering if anyone is familiar with this problem or has any idea to find the root cause. Thanks. Best, Tony Wei 2018-08-29 16:20 GMT+08:00 Andrey Zagrebin : > Hi, > > the current Flink 1.6.0 version uses

Re: checkpoint failed due to s3 exception: request timeout

Hi Vino, I thought this config is for aws s3 client, but this client is inner flink-s3-fs-presto. So, I guessed I should find a way to pass this config to this library. Best, Tony Wei 2018-08-29 14:13 GMT+08:00 vino yang : > Hi Tony, > > Sorry, I just saw the timeout, I thought

Re: checkpoint failed due to s3 exception: request timeout

filesystem and I thought it might have a simple way to support this setting like other s3.xxx config. Very much appreciate for your answer and help. Best, Tony Wei 2018-08-29 11:51 GMT+08:00 vino yang : > Hi Tony, > > A while ago, I have answered a similar question.[1] > > You

Re: Need a clarification about removing a stateful operator

Hi Stefan, Thanks for your detailed explanation. Best, Tony Wei 2018-08-17 15:56 GMT+08:00 Stefan Richter : > Hi, > > it will not be transported. The JM does the state assignment to create the > deployment information for all tasks. If will just exclude the state for > operato

Re: Need a clarification about removing a stateful operator

Hi Chesnay, Thanks for your quick reply. I have another question. Will the state, which is ignored, be transported to TMs from DFS? Or will it be detected by JM's checkpoint coordinator and only those states reuired by operators be transported to each TM? Best, Tony Wei 2018-08-17 14:38 G

Need a clarification about removing a stateful operator

ator? And could this behavior differ between different state backend (Memory, FS, RocksDB) ? Many thanks, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology

Re: flink_taskmanager_job_task_operator_records_lag_max == -Inf on Flink 1.4.2

`records_lag_max` is the maximum lag in terms of number of records for any partition in this "window". I'm not sure what this "window" means and if it is configurable. If it is configurable, then you can directly pass the config argument to Flink-Kafka-Connector to set kafka consumer.

Re: Conceptual question

generated by another state descriptor. Please correct me if I misunderstood. Thank you. Best Regards, Tony Wei 2018-06-09 9:45 GMT+08:00 TechnoMage : > Thank you all. This discussion is very helpful. It sounds like I can > wait for 1.6 though given our development status. > >

Can not submit flink job via IP or VIP of jobmanager

using the hostname (ip-10-0-1-95.eu-central-1.compute.internal) of the job manager and not the IP or the VIP? Thank you! Best regards Wei Cluster configuration: Standalone cluster with JobManager at flink.marathon.l4lb.thisdcos.directory/***.***.***.***:6123 Usi

Re: Conceptual question

Wei 2018-06-07 21:43 GMT+08:00 Piotr Nowojski : > Hi, > > Oh, I see now. Yes indeed getKeyedStateBackened() is not exposed to the > function and you can not migrate your state that way. > > As far as I know yes, at the moment in order to convert everything at once > (witho

Re: Conceptual question

7;t look up all keys and migrate the entire previous states to the new states in `ProcessFunction#open()`? As I said, do I need to port `ProcessFunction` to `KeyedProcessOperator` to migration state like the manner showed in `WindowOperator`? Best Regards, Tony Wei 2018-06-07 20:28 GMT+08:00

Re: Conceptual question

was implemented state operator by `ProcessFunction` API, is it possible to port it to `KeyedProcessOperator` and do the state migration that you mentioned? And are there something concerned and difficulties that will leads to restored state failed or other problems? Thank you! Best Regards, Tony Wei

Re: Question about the behavior of TM when it lost the zookeeper client session in HA mode

st ZK connection. So, as Piotr said, it looks like an error in Kafka producer and I will pay more attention on it to see if there is something unexpected happens again. Best Regards, Tony Wei 2018-05-15 19:56 GMT+08:00 Piotr Nowojski : > Hi, > > It looks like there was an error in asynchronous

Question about the behavior of TM when it lost the zookeeper client session in HA mode

ny times during the last weekend and made my kafka log delay grew up. Please help me. Thank you very much! Best Regards, Tony Wei

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

and enabled incremental checkpoint mechanism as well. Our job has run healthily for more than two weeks. Thank you all for helping me to investigate and solve this issue. Best Regards, Tony Wei [1] EBS: I/O Credits and Burst Performance <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVo

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi Stefan, We prepared to run it on local SSDs yesterday. However, as I said, the problem just disappeared. Of course we will replace it to local SSDs, but I'm afraid that I might be able to reproduce the same situation for both environments to compare the difference. Best Regards, Ton

Re: Fwd: checkpoint stuck with rocksdb statebackend and s3 filesystem

root cause or stop the investigation and make the conclusion in this mailing thread. What do you think? Best Regards, Tony Wei 2018-03-06 15:13 GMT+08:00 周思华 : > Hi Tony, > > Sorry for missing the factor of cpu, I found that the "bad tm"'s cpu load > is so much high

Fwd: checkpoint stuck with rocksdb statebackend and s3 filesystem

Sent to the wrong mailing list. Forward it to the correct one. -- Forwarded message -- From: Tony Wei Date: 2018-03-06 14:43 GMT+08:00 Subject: Re: checkpoint stuck with rocksdb statebackend and s3 filesystem To: 周思华 , Stefan Richter Cc: "user-subscr...@flink.apache.org&q

  1   2   >