Flink SQL 1.12 How to implement query Hbase table on secondary index

Hi, now my data store hbase and I want use flink to implement kafka table temproal join hbase table , but condtion is not rowkey , I realize hbase secondary index, how can I implement this function what can use flink sql first query secondary index(such as es) and then use rowkey query hba

Potential Bug with Date Serialization for Table Stream

Per the docs , I'm hoping to confirm whether or not an error we are seeing is a bug with Flink. We have a job that uses a Kafka source to read Avro records. The kafka source is converted into a StreamTableSource. We are using the new Blink tabl

Stateful function with GCP Pub/Sub ingress/egress

The statefun docs have some nice examples of how to use Kafka and Kinesis for ingress/egress in conjunction with a function. Is there some documentation or example code I could reference to do the same with a GCP Pub/Sub topic? Thanks. Dave

RE: Watermarks event time vs processing time

Hi Hanspeter, Event time mode should work just the same … for your example below you your need only one single arbitrary event per kafka partition that has a timestamp > 1646992800560 + sessionWindowGap + outOfOrderness in order for the session window to be triggered. I’m not sure why processi

Re: Watermarks event time vs processing time

Hi Matthias and others Thanks for the answer. I will remove the Idleness. However I am not doing max/min etc. Unfortunately most examples are about aggregations. The inputs are like this {"handling_time":1646992800260,"transaction_id":"017f6af1548e-119dfb",} {"handling_time":164699280

RE: Watermarks event time vs processing time

Hi Hanspeter, Let me relate some hints that might help you getting concepts clearer. From your description I make following assumptions where your are not specific enough (please confirm or correct in your answer): 1. You store incoming events in state per transaction_id to be sorted/aggreg

Adding a custom Kafka deserializer to Statefun Job

Hi, I am writing a StateFun application using remote functions, where I want to include a custom Kafka deserializer that adds the timestamp of the Kafka ingress messages to the messages sent between my remote functions. I can't seem to find a solution to this using remote functions, as the cur

Re: Rescaling REST API not working

HI, Aarsh Shah By consulting the information, I found that this API has been temporarily deprecated, and there is no API to deal with rescaling. Maybe you can try to use the reactive scheduler. > 2022年3月16日 下午2:15，Aarsh Shah 写道： > > Hello, > I tried to call the rescaling api with patch to aut

Re: RocksDB metrics for effective memory consumption

Hi Donatien, The managed memory in Flink actually locates off-heap and stays as native memory, e.g memory consumed by RocksDB, python. In other words, JVM cannot know how much the memory that third-party software used. Thus, Flink will just set the upper limit for the managed memory and let thi

Watermarks event time vs processing time

Hi, I read from a Kafka topic events that are in JSON format These event contain a handling time (aka event time) in epoch milliseconds, a transaction_id and a large nested JSON structure. I need to group the events by transaction_id, order them by handling time and calculate the differences in ha

Re: [ANNOUNCE] Apache Flink 1.1.4.4 released

Hi Konstantin, I have installed the wheel packages of 1.13.6 and 1.14.4 respectively, and tested them through some examples. Thanks a lot for your work. Best, Xingbo Konstantin Knauf 于2022年3月16日周三 15:29写道： > Hi Xingbo, > > you are totally right. Thank you for noticing. This also affected Flink

Re: Setting S3 as State Backend in SQL Client

Hi dz902, I actually can't find that sentence on the website you've linked to. It does state "The following sections list all available options that can be used to adjust Flink Table & SQL API programs.". So that list are the available options that you can use. The options that you're trying are n

Re: Setting S3 as State Backend in SQL Client

Hi, Per SQL Lite doc ( https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/sqlclient/) I see this: > SQL Client Configuration > You can configure the SQL client by setting the options below, or any valid Flink configuration entry: So any valid Flink configuration should wor

Re: [ANNOUNCE] Apache Flink 1.1.4.4 released

Hi Xingbo, you are totally right. Thank you for noticing. This also affected Flink 1.13.6, the other release I was recently managing. I simply skipped a step in the release guide. It should be fixed now. Could you double-check? Cheers, Konstantin On Wed, Mar 16, 2022 at 4:07 AM Xingbo Huang w

Re: Setting S3 as State Backend in SQL Client

Hi, If I remember correctly, set operations supports only a limited set of configurations. Most of them are table options that are listed on table configuration [1] plus some pipeline options. State backend options are not likely one of them. [1] https://nightlies.apache.org/flink/flink-docs