zhuzhurk commented on code in PR #24475:
URL: https://github.com/apache/flink/pull/24475#discussion_r1519708804
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/AbstractMultipleInputTransformation.java:
##########
@@ -76,7 +76,7 @@ public StreamOperatorFactory<OUT> getOperatorFactory() {
}
@Override
- public List<Transformation<?>> getTransitivePredecessors() {
+ protected List<Transformation<?>> getTransitivePredecessorsInternal() {
return inputs.stream()
.flatMap(input -> input.getTransitivePredecessors().stream())
Review Comment:
One transformation, and all its predecessors, can appear multiple times if
it is the head of a diamond DAG structure, like the case reported in
FLINK-32513. This may results in lots of memory consumption.
Maybe we can introduce an `LinkedHashSet` to do deduplication.
##########
flink-core/src/test/java/org/apache/flink/api/dag/TransformationTest.java:
##########
@@ -42,10 +43,21 @@
public class TransformationTest extends TestLogger {
private Transformation<Void> transformation;
+ private Transformation<Void> transformationWithInput;
@Before
public void setUp() {
transformation = new TestTransformation<>("t", null, 1);
+ transformationWithInput =
+ new TestTransformationWithInput<>(
+ "t", null, 1, Arrays.asList(transformation,
transformation));
+ }
+
+ @Test
+ public void testPredecessorCache() throws Exception {
+ transformationWithInput.getTransitivePredecessors();
+ assertEquals(transformationWithInput.getPredecessorsCache().size(), 1);
+ assertEquals(((TestTransformation<?>)
transformation).getNumGetTransitivePredecessor(), 1);
}
Review Comment:
Could you construct a case that is demonstrated in FLINK-32513 and verify
the traversing count and predecessor count?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
