Re: [Flink SQL] Lookup join hbase problem

Hi houyin, > It maybe because the conditions in where clause, being pushed down as a > predicate into join clause ? Yes, after pushdown, hbase lookupkeys are `rowKey` and `city_code`, which trigger above exception. > How can I solve this problem ? Because only constant value and field input ref o

Event Time Timers in Process functions when time characteristic is ProcessingTime

Hi, In environment configuration when set the time characteristics to ProcessingTime via setStreamTimeCharacteristic(...) call, I cannot see watermarks in the Flink UI. I think this is expected, because watermarks disabled in the source (using Kafka as source)? Another point here is, can I use Ev

Re: Exception in snapshotState suppresses subsequent checkpoints

Hi Tao, I'm afraid that your Flink job continues to be in high backpressued and all subsequent checkpoints did not ever run 'FromElementsFunctionT#snapshotState' which means your code to throw exception never be executed. You could check those expired checkpoints to see whether your tasks conta

Re: [Flink SQL] Lookup join hbase problem

Yes. Currently, the HBase lookup source only supports lookup on rowkey. If there is more than one join on condition, it may fail. We should support lookup HBase on multiple fields (by Get#setFilter). Feel free to open issues. Best, Jark On Mon, 28 Jun 2021 at 12:48, 纳兰清风 wrote: > Hi, > > Whe

Re: Questions about UPDATE_BEFORE row kind in ElasticSearch sink

UPDATE_BEFORE is required in cases such as Aggregation with Filter. For example: SELECT * FROM ( SELECT word, count(*) as cnt FROM T GROUP BY word ) WHERE cnt < 3; There is more discussion in this issue: https://issues.apache.org/jira/browse/FLINK-9528 Best, Jark On Mon, 28 Jun 2021 at 13

Questions about UPDATE_BEFORE row kind in ElasticSearch sink

Hi team, We notice the UPDATE_BEFORE row kind is not dropped and treated as DELETE as in code

Re: [Question] Why doesn't Flink use Calcite adapter?

Hi guangyuan, The question is an interesting and broad topic. I try to give my opinion based on my limited knowledge. Flink introduces dynamic sources to read from an external system[1]. Flink connector modules are completely decoupled with Calcite. There are two benefits: (1) If users need to dev

[Flink SQL] Lookup join hbase problem

Hi, When I was using hbase table as my lookup table, I got this error: Caused by: java.lang.IllegalArgumentException: Currently, HBase table can only be lookup by single row key. at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138) My SQL is insert into si

Re: [Question] Why doesn't Flink use Calcite adapter?

Maybe this question was better addressed to the DEV list. On Fri, Jun 25, 2021 at 11:57 PM guangyuan wang wrote: > > > > I have read the design doc of the Flink planner recently. I've found the > Flink only uses Calcite as an SQL optimizer. It

Re: Yarn Application Crashed?

Just found some additional info. It looks like one of the EC2 instances got terminated at the time the crash happened and this job had 7 Task Managers running on that EC2 instance. Now I suspect it's possible that when Yarn tried to migrate the Task Managers, there were no idle containers as this j

Yarn Application Crashed?

Hi, I recently experienced a job crash due to the underlying Yarn application failing for some reason. Here is the only error message I saw. It seems I can no longer see any of the Flink job logs. Application application_1623861596410_0010 failed 1 times (global limit =2; local limit is =1) due t