Hi Jark,
+1 for your suggestion. I think it will simplify the design a lot if we
serialize all expressions as SQL strings and will avoid duplicate parser
code. Initially, I had concerns that there might be expressions in the
future that cannot be represented in SQL. But currently I cannot come up
with a counter example.
Table operations will be a different topic that will require a custom
string syntax. But expressions as SQL expressions sounds good to me.
@Jingsong: Jark is right. Table API expression strings are outdated and
error-prone. They will be removed at some point.
Regards,
Timo
On 24.10.19 10:50, Jark Wu wrote:
Thanks Jingsong,
As discussed in Java Expression DSL, we are planning to drop the Java
Expression string API.
So I think we don’t have a plan to unify #1 and #2. And in the future, we may
only have SQL parser to parse a string expression.
The only thing to consider is, whether can all the resolved expression be
converted to SqlNode.
AFAIK, currently, after expression resolving, all the expressions can be
converted to SqlNodes.
Best,
Jark
[1]:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Introduction-of-a-Table-API-Java-Expression-DSL-td27787.html
<http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Introduction-of-a-Table-API-Java-Expression-DSL-td27787.html>
在 2019年10月24日，13:30，Jingsong Li <jingsongl...@gmail.com> 写道：
Thanks Jark for your proposal.
If we introduce a new kind of string presentation for expression, we will
have 3 string presentation now:
1. Java expression string api. We have PlannerExpressionParser to parse
string to Expressions.
2. Sql string presentation, as you said, we can use calcite classes to
parse and unparse.
3. New kind of string presentation for serialize.
From this point of view, I prefer not to introduce a new kind of string
presentation to reduce the complexity.
There are some differences between #1 and #2:
- method invoking: "f0.substring(1, f7)" and "SUBSTRING(f0, 1, f7)"
- bigint literal: "1L" and "cast(1 as BIGINT)"
Now it is two completely independent sets., Whether we can unify #1 and #2
into one set, and we all use one parser?
Best,
Jingsong Lee
On Tue, Oct 22, 2019 at 7:57 PM Jark Wu <imj...@gmail.com> wrote:
Hi Timo,
I think it's a good idea to use `SqlParser#parseExpression()` to parse
literals.
That means the string format of literal is SQL compatible.
After some discussion with Kurt, we think why not one more step forward,
i.e. convert the whole expression to SQL format.
For example, the above expression will be converted to:
`cat`.`db`.`func`(`cat`.`db`.`f0`, TIMESTAMP '2019-10-21 12:12:12')
There are some benefits from this:
0) the string representation is more readable, and can be manually typed
more easily.
1) the string format is SQL syntax, not customized, which means it can be
integrated by third party projects.
2) we can reuse Calcite's SqlParser to parse string and SqlNode#unparse to
generate string, this can avoid introducing duplicate code and a custom
parser.
3) no compatible problems.
Regarding to how Expression can be converted into a SQL string, I think we
can leverage some Calcite utils:
ResolvedExpression ---(ExpressionConverter)---> RexNode
----(RexToSqlNodeConverter)---> SqlNode --> SqlNode#unparse()
What do you think?
Best,
Jark
On Mon, 21 Oct 2019 at 22:08, Timo Walther <twal...@apache.org> wrote:
Hi Jark,
thanks for the proposal. This is a great effort to finalize the new API
design.
I'm wondering if we could simply use the SQL parser like
`org.apache.calcite.sql.parser.SqlParser#parseExpression(..)` to parse
an expression that contain only literals. This would avoid any
discussion as the syntax is already defined by the SQL standard. And it
would also be very unlikely to have a need for a version.
For example:
CALL('FUNC', FIELD('f0'), VALUE('TIMESTAMP(3)', TIMESTAMP '2019-10-21
12:12:12'))
Or even further if the SQL parser allows that:
CALL('FUNC', `cat`.`db`.`f0`, TIMESTAMP '2019-10-21 12:12:12')
I would find it confusing if we use different representation for
literals such as intervals and timestamps in the properties. This would
also reduce code duplication as we reuse logic for parsing identifiers
etc.
What do you think?
Regards,
Timo
On 18.10.19 12:28, Jark Wu wrote:
Hi everyone,
I would like to start a discussion[1] about how to make Expression
string
serializable and deserializable. Expression is the general interface
for
all kinds of expressions in Flink Table API & SQL, it represents a
logical
tree for producing a computation result. In FLIP-66[2] and FLIP-70[3],
we
introduced watermark and computed column syntax in DDL. The watermark
strategy and computed column are both represented in Expression. In
order
to persist watermark and computed column information in catalog, we
need
to
figure out how to persist and restore Expression.
FLIP-80:
https://docs.google.com/document/d/1LxPEzbPuEVWNixb1L_USv0gFgjRMgoZuMsAecS_XvdE/edit?usp=sharing
Thanks for any feedback!
Best,
Jark
[1]:
https://docs.google.com/document/d/1LxPEzbPuEVWNixb1L_USv0gFgjRMgoZuMsAecS_XvdE/edit?usp=sharing
[2]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL
[3]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design
--
Best, Jingsong Lee
