Re: Flink SQL continuous join checkpointing

2020-12-23 Thread Taras Moisiuk
Hi Leonard, Thank you for answer, in fact I used regular join because my interval condition was based on wrong column. I extended my join with attribute column condition and it solved the problem: ... FROM table_fx fx LEFT JOIN table_v v ON v.active = fx.instrument_active_id

Re: Flink SQL continuous join checkpointing

2020-12-22 Thread Leonard Xu
Hi, Taras > But checkpoint data size for join task is permanently increasing despite the > watermarks on the tables and "Low watermark" mark in UI. > As far as I understand outdated records from both tables must be dropped from > checkpoint after 2 hours, but looks like it holds all job state s

Flink SQL continuous join checkpointing

2020-12-22 Thread Taras Moisiuk
Hi everyone! I'm using Flink *1.12.0* with SQL API. My streaming job is basically a join of two dynamic tables (from kafka topics) and insertion the result into PostgreSQL table. I have enabled watermarking based on kafka topic timestamp column for each table in join: CREATE TABLE table1 (