[jira] [Created] (FLINK-36320) Use Protobuf Nested timestamp field as event time column and watermark
haiqingchen created FLINK-36320:
---
Summary: Use Protobuf Nested timestamp field as event time column
and watermark
Key: FLINK-36320
URL: https://issues.apache.org/jira/browse/FLINK-36320
Project: Flink
Issue Type: Improvement
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: haiqingchen
Hi I'm using protobuf format to read from kafka and there's a message type Test
in the proto which is also a nested field in another message type TestObject
{code:java}
message Test {
int64 startTime,
int32 id
}
message TestObject {
repeated Test tests = 3
}{code}
Here's how I define the ddl.
{code:java}
create table test_pb (
Test Array>
) with (
"connector"="kafka",
'format' = 'protobuf',
'protobuf.message-class-name'= '.TestObject'
..
)
{code}
I would like to use startTime field in the nested Test object as event time
column and define watermark for it , but as it is a nested field in an Array, I
can't simply use it. I would like to flatten the TestObject column but it
doesn't work for DDL.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-36321) Execute read/write state request in different executor
Yanfei Lei created FLINK-36321: -- Summary: Execute read/write state request in different executor Key: FLINK-36321 URL: https://issues.apache.org/jira/browse/FLINK-36321 Project: Flink Issue Type: Sub-task Reporter: Yanfei Lei -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36322) Fix compile error of flink benchmark caused by breaking changes
Zakelly Lan created FLINK-36322: --- Summary: Fix compile error of flink benchmark caused by breaking changes Key: FLINK-36322 URL: https://issues.apache.org/jira/browse/FLINK-36322 Project: Flink Issue Type: Technical Debt Components: Benchmarks Reporter: Zakelly Lan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36314) Support state V1 interface
Yanfei Lei created FLINK-36314: -- Summary: Support state V1 interface Key: FLINK-36314 URL: https://issues.apache.org/jira/browse/FLINK-36314 Project: Flink Issue Type: Sub-task Reporter: Yanfei Lei -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36319) FAIL behavior on non-retriable write errors causes an infinite loop when restarting from checkpoint
Lorenzo Nicora created FLINK-36319:
--
Summary: FAIL behavior on non-retriable write errors causes an
infinite loop when restarting from checkpoint
Key: FLINK-36319
URL: https://issues.apache.org/jira/browse/FLINK-36319
Project: Flink
Issue Type: Sub-task
Reporter: Lorenzo Nicora
The {{FAIL}} (default) error handling behavior when a write request is rejected
as non-retriable ({{onPrometheusNonRetriableError}}), causes the job to fail
and restart.
Restarting from checkpoint causes some out-of-order (duplicate) writes, that by
default Prometheus rejects as non-retrable.
As a consequence, when {{onPrometheusNonRetriableError}} = {{FAIL}} any
restarts from checkpoint puts the job in an infinite loop.
Changes:
1. default {{onPrometheusNonRetriableError}} should be {{DISCARD_AND_CONTINUE}}
2. {{onPrometheusNonRetriableError}} cannot be set to {{FAIL}}
We can keep the rest of the implementation as-is for the moment, and just
prevent from setting {{FAIL}} for this behaviour, as we may expand handling
this error with a different behaviour
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-36315) The flink-cdc-base module supports source metric statistics
liuxiaodong created FLINK-36315: --- Summary: The flink-cdc-base module supports source metric statistics Key: FLINK-36315 URL: https://issues.apache.org/jira/browse/FLINK-36315 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: liuxiaodong The MySQL source already supports embedding observability metrics, but this feature cannot be reused by other source types. Therefore, we hope to port this feature to the base for easy reuse by other types -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36316) Move PortRange util to flink-core
Ferenc Csaky created FLINK-36316:
Summary: Move PortRange util to flink-core
Key: FLINK-36316
URL: https://issues.apache.org/jira/browse/FLINK-36316
Project: Flink
Issue Type: Improvement
Components: API / Core
Reporter: Ferenc Csaky
Fix For: 2.0-preview
The only thing what the {{PortRange}} utility does is wrapping a
{{NetUtils#getPortRangeFromString}} to help keeping track of the original
config string.
Considering this, it makes sense to move it to the same module, as currently
{{PortRange}} is in {{flink-runtime}}, which limits its usage to that module,
or requiring to add it as a dependency, which is not a very good option.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-36317) Populate the ArchivedExecutionGraph with CheckpointStatsSnapshot data if in WaitingForResources state with a previousExecutionGraph being set
Matthias Pohl created FLINK-36317:
-
Summary: Populate the ArchivedExecutionGraph with
CheckpointStatsSnapshot data if in WaitingForResources state with a
previousExecutionGraph being set
Key: FLINK-36317
URL: https://issues.apache.org/jira/browse/FLINK-36317
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Affects Versions: 2.0-preview
Reporter: Matthias Pohl
in FLINK-36295 we noticed an issue with the WaitingForResources state that
follows a restartable failure. The CheckpointStatistics are present but not
exposed through the ArchivedExecutionGraph despite being available.
We should think about adding these stats in {{WaitingForResources#getJob}} to
have them accessible even if the job isn't running at the moment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
