[
https://issues.apache.org/jira/browse/FLINK-23989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihao Wang updated FLINK-23989:
--------------------------------
Description:
It’s desired to inspect into the internal states generated by SQL especially
for debug purpose. We propose to add a new feature in Flink UI console, when
the users click on a Vertex of the Job DAG. We shall list all states the vertex
has in a separate panel (let’s call *states panel*). On this panel users can
query the states with some keys. The returned value share be a human readable
string instead of opaque binary.
Particularly, we need to expose the states as queryable. But currently the user
experience of [queryable
states|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/fault-tolerance/queryable_state/]
is cumbersome. Only the serialized value is returned to client and users need
to handle deserialization by themselves. What’s worse, the client need to
construct the serializer and type manually. To improve this situation. We
propose:
# Have a new API to find all queryable states associated to a job vertex. This
can be done to check against the
[KvStateLocationRegistry|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/query/KvStateLocationRegistry.java],
which store the mapping between JobVertexId and states.
# Have a new API to allow users get the types of queryable states: For a
register name (String), Queryable Server will return the type of key and value
([LogicalType|https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/LogicalType.java]).
# To generate human readable string automatically with API in step 2, we can
1) generate
[TypeSerializer|https://github.com/apache/flink/blob/6c9818323b41a84137c52822d2993df788dbc9bb/flink-core/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializer.java]
from the LogicalType, so as to handle Serde automatically. 2) to convert
internal data structures to external data structures to generate printable
string. (with converters
[DataStructureConverter|https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/table/data/conversion/DataStructureConverter.html]
)
With all these steps and some modifications to Web UI/ Rest, we can enable
users to query SQL internal states.
was:
It’s desired to inspect into the internal states generated by SQL especially
for debug purpose. We propose to add a new feature in Flink UI console, when
the users click on a Vertex of the Job DAG. We shall list all states the vertex
has in a separate panel (let’s call *states panel*). On this panel users can
query the states with some keys. The returned value share be a human readable
string instead of opaque binary.
Particularly, we need to expose the states as queryable. But currently the user
experience of [queryable
states|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/fault-tolerance/queryable_state/]
is cumbersome. Only the serialized value is returned to client and users need
to handle deserialization by themselves. What’s worse, the client need to
construct the serializer and type manually. To improve this situation. We
propose:
# Have a new API to find all queryable states associated to a job vertex. This
can be done to check against the KvStateLocationRegistry, which store the
mapping between JobVertexId and states.
# Have a new API to allow users get the types of queryable states: For a
register name (String), Queryable Server will return the type of key and value
([LogicalType|https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/LogicalType.java]).
# To generate human readable string automatically with API in step 2, we can
1) generate
[TypeSerializer|https://github.com/apache/flink/blob/6c9818323b41a84137c52822d2993df788dbc9bb/flink-core/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializer.java]
from the LogicalType, so as to handle Serde automatically. 2) to convert
internal data structures to external data structures to generate printable
string. (with converters
[DataStructureConverter|https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/table/data/conversion/DataStructureConverter.html]
)
With all these steps and some modifications to Web UI/ Rest, we can enable
users to query SQL internal states.
> Flink SQL visibility
> ---------------------
>
> Key: FLINK-23989
> URL: https://issues.apache.org/jira/browse/FLINK-23989
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Queryable State
> Reporter: Zhihao Wang
> Priority: Major
>
> It’s desired to inspect into the internal states generated by SQL especially
> for debug purpose. We propose to add a new feature in Flink UI console, when
> the users click on a Vertex of the Job DAG. We shall list all states the
> vertex has in a separate panel (let’s call *states panel*). On this panel
> users can query the states with some keys. The returned value share be a
> human readable string instead of opaque binary.
>
> Particularly, we need to expose the states as queryable. But currently the
> user experience of [queryable
> states|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/fault-tolerance/queryable_state/]
> is cumbersome. Only the serialized value is returned to client and users
> need to handle deserialization by themselves. What’s worse, the client need
> to construct the serializer and type manually. To improve this situation. We
> propose:
> # Have a new API to find all queryable states associated to a job vertex.
> This can be done to check against the
> [KvStateLocationRegistry|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/query/KvStateLocationRegistry.java],
> which store the mapping between JobVertexId and states.
> # Have a new API to allow users get the types of queryable states: For a
> register name (String), Queryable Server will return the type of key and
> value
> ([LogicalType|https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/LogicalType.java]).
> # To generate human readable string automatically with API in step 2, we can
> 1) generate
> [TypeSerializer|https://github.com/apache/flink/blob/6c9818323b41a84137c52822d2993df788dbc9bb/flink-core/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializer.java]
> from the LogicalType, so as to handle Serde automatically. 2) to convert
> internal data structures to external data structures to generate printable
> string. (with converters
> [DataStructureConverter|https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/table/data/conversion/DataStructureConverter.html]
> )
> With all these steps and some modifications to Web UI/ Rest, we can enable
> users to query SQL internal states.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
