Ramin Gharib created FLINK-38532:
------------------------------------
Summary: FLIP-551: Make FRESHNESS Optional for Materialized Tables
Key: FLINK-38532
URL: https://issues.apache.org/jira/browse/FLINK-38532
Project: Flink
Issue Type: New Feature
Components: Table SQL / API
Reporter: Ramin Gharib
The {{FRESHNESS}} clause, as introduced in
[FLIP-435|https://cwiki.apache.org/confluence/display/FLINK/FLIP-435%3A+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines],
is a mandatory part of the {{CREATE MATERIALIZED TABLE}} syntax. While this
forces users to be explicit about their data recency requirements, it
introduces friction, especially for new users and everyday use cases.
The primary motivations for making this clause optional are:
# *Reduce Boilerplate:* For many users, the goal is to create a continuously
updating materialized table with a low-latency, "near real-time" refresh.
Requiring them to specify {{FRESHNESS = INTERVAL '...' SECOND/MINUTE}} in every
single statement is redundant for this common pattern.
# *Lower Barrier to Entry:* The {{FRESHNESS}} concept, while powerful,
currently forces new users to immediately understand the distinction between
{{CONTINUOUS}} and {{FULL}} refresh modes. More importantly, to choose a
sensible {{{}FRESHNESS{}}}, they implicitly need to understand that this value
becomes the job's _checkpoint interval_ in the default streaming mode.
By providing a sensible default, we remove this requirement. A new user can get
a working, continuous pipeline without needing to know about the underlying
checkpointing mechanism upfront. They can start with a simple, functional table
and learn about performance tuning concepts like checkpointing later, as their
needs become more advanced.
# *Enable Platform-Level Intelligence:* A more powerful architectural approach
is to allow the underlying {{Catalog}} to determine the default freshness,
enabling "smart" catalogs that can implement context-aware logic.
More information to be found on the FLIP:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-551%3A+Make+FRESHNESS+Optional+for+Materialized+Tables
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
