zhijiangW commented on a change in pull request #7631: [FLINK-11391][shuffle]
Introduce PartitionShuffleDescriptor and ShuffleDeploymentDescriptor
URL: https://github.com/apache/flink/pull/7631#discussion_r260636298
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/InputChannelDeploymentDescriptor.java
##########
@@ -52,30 +52,38 @@
/** The ID of the partition the input channel is going to consume. */
private final ResultPartitionID consumedPartitionId;
- /** The location of the partition the input channel is going to
consume. */
- private final ResultPartitionLocation consumedPartitionLocation;
+ /** The location type of the partition the input channel is going to
consume. */
+ private final LocationType locationType;
+
+ /** The connection to use to request the remote partition. */
+ private final Optional<ConnectionID> connectionId;
Review comment:
If I understand correctly, the above comments can be summarized into three
points:
1. `LocationType` can be decided by `ShuffleService` by comparing
`ResourceID` between producer and consumer. And the `consumerResourceID` could
be covered in `IGDD`. Regarding with `ICDD=SDD=PartitionInfo`, I only concern
the only different field `IntermediateDataSetID` existing in `PartitionInfo`
for finding the proper `SingleInputGate` to update partition info on `Task`
side. I would think through this step by step.
2. `registerPartition` and `cachePartition` are triggered in
`allocateAndAssignSlotForExecution` and return future, then we can confirm
during deploying the consumer in eager mode, we can always get corresponding
registered/cached `SDD` of producers.
3. `UnknownShuffleDeploymentDescriptor` should also be introduced for lazy
deployment mode to indicate there would be updated `SDD` later.
I agree with the above points, especially for the point 2 which solves my
previous concern naturally. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
