[
https://issues.apache.org/jira/browse/FLINK-31159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691304#comment-17691304
]
Hong Liang Teoh edited comment on FLINK-31159 at 2/20/23 9:35 PM:
------------------------------------------------------------------
h2. Test Outline
We want to submit a job that has parallelism > 1 and only 1 subtask is busy.
We want to verify FlameGraph correctly reflects the busy subtask.
h2. Test Setup
Flink job does the following:
* Source from a Kinesis stream
* Map with busy loop to calculate factorials
* Discarding sink
{code:java}
public static long factorialUsingStreams(int n) {
return LongStream.rangeClosed(1, n)
.reduce(1, (long x, long y) -> x * y);
}
public static void main(String[] args) throws Exception {
// ...
env.addSource(new FlinkKinesisConsumer<>(
STREAM, new SimpleStringSchema(), consumerConfig))
.map((MapFunction<String, String>) value -> {
for (int i = 0; i < 10; i++) {
factorialUsingStreams(25);
}
return value;
})
.addSink(new DiscardingSink<>());
env.execute("Busy loop");
}{code}
Flink job has parallelism of 4
Kinesis source stream only has 1 shard, so only 1 subtask will be reading
records.
This means only 1 subtask will be busy.
h2. Results
Verified that the Flame graph accurately reflects this. See snapshots below
- Only subtask 3 has records
!Map_record_distribution.png|width=1090,height=550!
* Flame graph with "all" setting
!all_subtasks.png|width=1083,height=562!
* FlameGraph with subtask 0 shows nothing on CPU
!subtask0.png|width=1080,height=550!
* FlameGraph from subtask 3 shows CPU usage as shown on "all"
!subtask3.png|width=1080,height=572!
was (Author: JIRAUSER292614):
h2. Test Outline
We want to submit a job that has parallelism > 1 and only 1 subtask is busy.
We want to verify FlameGraph correctly reflects the busy subtask.
h2. Test Setup
Flink job does the following:
* Source from a Kinesis stream
* Map with busy loop to calculate factorials
* Discarding sink
{code:java}
public static long factorialUsingStreams(int n) {
return LongStream.rangeClosed(1, n)
.reduce(1, (long x, long y) -> x * y);
}
public static void main(String[] args) throws Exception {
// ...
env.addSource(new FlinkKinesisConsumer<>(
STREAM, new SimpleStringSchema(), consumerConfig))
.map((MapFunction<String, String>) value -> {
for (int i = 0; i < 10; i++) {
factorialUsingStreams(25);
}
return value;
})
.addSink(new DiscardingSink<>());
env.execute("Busy loop");
}{code}
Flink job has parallelism of 4
Kinesis source stream only has 1 shard, so only 1 subtask will be reading
records.
This means only 1 subtask will be busy.
h2. Results
Verified that the Flame graph accurately reflects this. See snapshots below
- Only subtask 3 has records
!Map_record_distribution.png!
* Flame graph with "all" setting
!all_subtasks.png!
* FlameGraph with subtask 0 shows nothing on CPU
!subtask0.png!
* FlameGraph from subtask 3 shows CPU usage as shown on "all"
!subtask3.png!
> Release Testing: Verify FLINK-30583 Provide the flame graph to the subtask
> level
> --------------------------------------------------------------------------------
>
> Key: FLINK-31159
> URL: https://issues.apache.org/jira/browse/FLINK-31159
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / REST, Runtime / Web Frontend
> Affects Versions: 1.17.0
> Reporter: Hong Liang Teoh
> Priority: Major
> Fix For: 1.17.0
>
> Attachments: Map_record_distribution.png, all_subtasks.png,
> subtask0.png, subtask3.png
>
>
> The issue aims to verify FLINK-30583.
> Please verify:
> # When below conditions are met:
> ** Job has more than 1 parallel subtask
> ** Some subtasks are busy, whilst others are not (e.g. some receive records,
> some don't)
> ** The FlameGraph accurately reflects the busy subtask
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
