Re: datadog metrics

Hi Fanbin, I have commented on the ticket: https://issues.apache.org/jira/browse/FLINK-16611. - Steve On Mon, Mar 16, 2020 at 6:39 PM Fanbin Bu wrote: > Hi Steve, > > could you please share your work around solution in more detail in the > above ticket? > > Thanks, > Fanbin > > On Mon, Mar 16,

Re: Re: flink sql-client read hive orc table no result

Seems the root cause is "transactional"='true'. After remove this，the table can be queryed from flink sql-client，even i add "clustered by (robot_id) into 3 buckets" again. Thanks, Lei wangl...@geekplus.com.cn From: Kurt Young Date: 2020-03-18 18:04 To: wangl...@geekplus.com.cn CC: lirui;

Re: FlinkCEP - Detect absence of a certain event

Hi Humberto, I guess Fuji is familiar with Flink CEP and he can answer your proposed question. I already cc him. Best, Zhijiang -- From:Humberto Rodriguez Avila Send Time:2020 Mar. 18 (Wed.) 17:31 To:user Subject:FlinkCEP - Dete

Re: Upgrade flink fail from 1.9.1 to 1.10.0

Hey Andrey, Thanks for your answer. I know use savepoint to upgrade the flink cluster is the available, but that mean when I upgrading my flink cluster I need to cancel all jobs from JM. And the stream process will be stopped, that will have an impact for the production system which is time sensi

Can't create a savepoint with State Processor API

Hi everyone, I'm looking for a way to modify state inside an operator in Flink. I found State Processor API that allows to modify savepoints, which looks great. But I can't mak

Re: Timestamp Erasure

Hi Dom, If you are converting a DataStream to a Table with a rowtime attribute, then the DataStream should hold event-time timestamp. For example, call `assignTimestampsAndWatermarks` before converting to table. You can find more details in the doc [1]. Best, Jark [1]: https://ci.apache.org/pr

Re: Flink Schema Validation - Kafka

Hi Robert, Thanks for your reply. This helps, was looking into similar direction. Thanks, Hemant On Wed, 18 Mar 2020 at 8:44 PM, Robert Metzger wrote: > Hi Hemant, > > you could let the Kafka consumer just deserialize your JSON data as into a > DataStream, then you use a custom processFunction

Timestamp Erasure

Hey, I just wanted to ask about one thing about timestamps. So, currently If I have a KeyedBroadcastProcess function followed by Temporal Table Join, it works like a charm. But, say I want to delay emitting some of the results due to any reason. So If I *registerProcessingTimeTimer* and any elemen

Flink Release Security Workflow

Are there any docs/links that describe the security workflow for a Flink release? For instance, the static code scan workflow; pen test workflow; security review of new features; etc. The reason for the question is to better understand how to include Flink within the security workflow of a prod

Re: Flink checkStateMappingCompleteness doesn't include UserDefinedOperatorIDs

Thanks a lot for reporting this potential issue. The ticket looks good. I would suggest to keep the discussion on the ticket. Otherwise, information will potentially be split between this list and the ticket. I'm sure a committer will soon look at it. On Wed, Mar 18, 2020 at 5:19 AM Bashar Abdul-

Re: time-windowed joins and tumbling windows

Hi Vinod, thanks for answering my questions. The == Optimized Logical Plan == looks as expected. However, the == Physical Execution Plan == seems to be quite complex. Are you sure that watermarks don't get lost in some of those custom operators before entering the SQL part of the pipeline? I

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

Hi Becket/Till, Thanks for the detail explanation. Just to confirm: the issue in FLINK-8093 refers to multiple Kafka consumer within the same TM - thus the fix should be to make consumer client.id unique for different tasks ? and the issue here is an issue internal to the Kafka consumer, where bot

Re: Flink Schema Validation - Kafka

Hi Hemant, you could let the Kafka consumer just deserialize your JSON data as into a DataStream, then you use a custom processFunction to parse the string to JSON. In your custom function, you can handle the error more flexibly (like outputting erroneous records through a side output). I hope th

Re: Custom Exception Handling

Hi Anil, You are never throwing any exception outside of your process function?, so your Flink job is not restarting because of a failure (your Flink job would restart because of a failure if you would throw an exception out of your user code). If you can rule-out a job restart (check the logs fo

Re: Upgrade flink fail from 1.9.1 to 1.10.0

Hi Reo, I do not think this is always guaranteed by Flink API. The usual supported way is to: - take a savepoint - upgrade the cluster (JM and TM) - maybe rebuild the job against the new Flink version - start the job from the savepoint [1] The externalised checkpoints also do not have to be alwa

Upgrade flink fail from 1.9.1 to 1.10.0

Hi all, I encountered a problem when I upgrade flink from 1.9.1 to 1.10.0. At first, my job is running on flink stably which JM and TM is flink 1.9.1. And then I try to upgrade to 1.10.0. I stop the JM progress and start another JM progress. At this time, the JM is 1.10.0 and the TM is 1.9.1,

Re: [DISCUSS] FLIP-111: Docker image unification

Thanks for creating this FLIP Andrey. I like the general idea pretty much. I tried to group some of the above-mentioned points in order to give my 50 cent. # Configuration How to configure the Flink process seems to be the biggest question. Due to historical reasons we have a plethora of differe

Re: Setting the operator-id to measure percentile latency over several jobs

I talked to Chesnay about this offline. Shipping the operator names with the latency markers would significantly increase their size (potentially affecting performance) There is no global lookup from operatorId to operatorName. On Fri, Mar 6, 2020 at 5:34 PM Robert Metzger wrote: > @Chesnay Sche

Flink Schema Validation - Kafka

Hi Users, Is there a way I can do a schema validation on read from Kafka in a Flink job. I have a pipeline like this Kafka Topic Raw(json data) -> Kafka Topic Avro(avro data) -> Kafka Topic Transformed(avro data) -> Sink While reading from Raw topic I wanted to validate the schema so that in ca

Re: Re: flink sql-client read hive orc table no result

also try to remove "transactional"='true'? Best, Kurt On Wed, Mar 18, 2020 at 5:54 PM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > > Tried again. Even i remove the "clustered by (robot_id) into 3 buckets" > statement, no result from flink sql-client > > Thanks, > Lei > >

Re: Re: flink sql-client read hive orc table no result

Tried again. Even i remove the "clustered by (robot_id) into 3 buckets" statement, no result from flink sql-client Thanks, Lei wangl...@geekplus.com.cn From: Kurt Young Date: 2020-03-18 17:41 To: wangl...@geekplus.com.cn; lirui CC: user Subject: Re: flink sql-client read hive orc table no

Re: flink sql-client read hive orc table no result

My guess is we haven't support hive bucket table yet. cc Rui Li for confirmation. Best, Kurt On Wed, Mar 18, 2020 at 5:19 PM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > Hive table store as orc format: > CREATE TABLE `robot_tr`(`robot_id` int, `robot_time` bigint, > `linear_

FlinkCEP - Detect absence of a certain event

In the documentation of FlinkCEP, I found that I can enforce that a particular event doesn't occur between two other events using notFollowedBy or notNext. However, I was wondering If I could detect the absence of a certain event after a time X. For example, if an event A is not followed by anot

flink sql-client read hive orc table no result

Hive table store as orc format: CREATE TABLE `robot_tr`(`robot_id` int, `robot_time` bigint, `linear_velocity` double, `track_side_error` int) partitioned by (warehouseid STRING) clustered by (robot_id) into 3 buckets stored as orc tblproperties("transactional"='true'); Under hive client,

Re: How do I get the outPoolUsage value inside my own stream operator?

Hi Felipe, I checked the code path, and the metric of outPoolUsage is under the following layer: TaskMetricGroup -> TaskIOMetricGroup -> "buffers" group -> "outPoolUsage". It seems that you missed the `TaskIOMetricGroup` from below samples. You can get it from TaskMetricGroup. Hope it solve yo

Re: Help me understand this Exception

Thanks, Zhijiang and Gordon. I will see the logs to find out more. On Wed, Mar 18, 2020 at 1:44 PM Zhijiang wrote: > Agree with Gordon's below explanation! > > Besides that, maybe you can also check the job master's log which might > probably show the specific exception to cause this failure. >

Re: Help me understand this Exception

Agree with Gordon's below explanation! Besides that, maybe you can also check the job master's log which might probably show the specific exception to cause this failure. I was thinking whether it is necessary to improve ExceptionInChainedOperatorException to also provide the message from the

Re: JobMaster does not register with ResourceManager in high availability setup

It seems that your zookeeper service is not stable. From the the log i find that resourcemanager leader is granted and taskmanager could register to resourcemanager successfully. That means the resourcemanager address has been published to the ZK successfully. Also a ZooKeeperLeaderRetrievalServic