wangzhihao created FLINK-23989:
----------------------------------
Summary: Flink SQL visibility
Key: FLINK-23989
URL: https://issues.apache.org/jira/browse/FLINK-23989
Project: Flink
Issue Type: New Feature
Components: Runtime / Queryable State
Reporter: wangzhihao
It’s desired to inspect into the internal states generated by SQL especially
for debug purpose. We propose to add a new feature in Flink UI console, when
the users click on a Vertex of the Job DAG. We shall list all states the vertex
has in a separate panel (let’s call *states panel*). On this panel users can
query the states with some keys. The returned value share be a human readable
string instead of opaque binary.
Particularly, we need expose the states as queryable. But currently the user
experience of queryable states is cumbersome. Only the serialized value is
returned to client and users need to handle deserialization by themselves.
What’s worse, the client need to construct the serializer and type manually. To
improve this situation. We propose:
# Have a new API to find all queryable states associated to a job vertex. This
can be done to check against the KvStateLocationRegistry, which store the
mapping between JobVertexId and states.
# Have a new API to allow users get the types of queryable states: For a
register name (String), Queryable Server will return the type of key and value
([LogicalType|https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/LogicalType.java]).
# To generate human readable string with API in step 2, we can 1) generate
[TypeSerializer|https://github.com/apache/flink/blob/6c9818323b41a84137c52822d2993df788dbc9bb/flink-core/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializer.java]
from the LogicalType, so as to handle Serde automatically. 2) to convert
internal data structures to external data structures to generate printable
string. (with converters
[DataStructureConverter|https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/table/data/conversion/DataStructureConverter.html]
)
With all these steps and some modifications to Web UI/ Rest, we can enable
users to query SQL internal states.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
