Sorry, let me edit my first point. We can just create addendums for YARN-6616 in branch-2.7 and branch-2.8 to edit the submitTime field to the correct id 28. We don’t need to revert YARN-6616 from these branches completely.
Jonathan ________________________________ From: Jonathan Hung <jyhung2...@gmail.com> Sent: Tuesday, September 24, 2019 11:38 AM To: Eric Badger Cc: Hadoop Common; yarn-dev; mapreduce-dev; Hdfs-dev Subject: Re: Incompatible changes between branch-2.8 and branch-2.9 Hi Eric, thanks for the investigation. * For YARN-6616, for branch-2.8 and below, it was only committed to 2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can revert YARN-6616 from branch-2.7 and branch-2.8. * For YARN-6050, there's a bit here: https://developers.google.com/protocol-buffers/docs/proto that says "optional is compatible with repeated", so I think we should be OK there. * For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or 2.8.5 to a 2.9+ version will be an issue. One option could be to move the intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to release this), then upgrade from 2.8.6 to 2.9+. Jonathan Hung On Tue, Sep 24, 2019 at 9:23 AM Eric Badger <ebad...@verizonmedia.com.invalid> wrote: We (Verizon Media) are currently moving towards upgrading our clusters from our internal fork of branch-2.8 to an internal fork of branch-2. During this process, we have found multiple incompatible changes in protobufs between branch-2.8 and branch-2. These incompatibilities were all introduced between branch-2.8 and branch-2.9. I did a git diff over all .proto files across the branch-2.8 and branch-2.9 and found 3 instances of incompatibilities from 3 separate commits. All of the incompatibilities are in yarn_protos.proto I would like to discuss how to fix these incompatible changes. Otherwise, rolling upgrades will not be supported between branch-2.8 (or below) and branch-2.9 (or beyond). We could revert the incompatible changes, but then the new releases would be incompatible with the releases that have these incompatible changes. If we do nothing, then rolling upgrades won't work between 2.8- and 2.9+. Thanks, Eric ------------------------------------------------------------------- git diff branch-2.8..branch-2.9 $(find . -name '*\.proto') https://issues.apache.org/jira/browse/YARN-6616 - Trunk patch (applied through branch-2.9) differs from branch-2.8 patch @@ -211,7 +245,20 @@ message ApplicationReportProto { optional PriorityProto priority = 23; optional string appNodeLabelExpression = 24; optional string amNodeLabelExpression = 25; - optional int64 submitTime = 26; + repeated AppTimeoutsMapProto appTimeouts = 26; + optional int64 launchTime = 27; + optional int64 submitTime = 28; https://issues.apache.org/jira/browse/YARN-6050 - Trunk and branch-2 patches both change the protobuf type in the same way. @@ -356,7 +416,22 @@ message ApplicationSubmissionContextProto { optional LogAggregationContextProto log_aggregation_context = 14; optional ReservationIdProto reservation_id = 15; optional string node_label_expression = 16; - optional ResourceRequestProto am_container_resource_request = 17; + repeated ResourceRequestProto am_container_resource_request = 17; + repeated ApplicationTimeoutMapProto application_timeouts = 18; https://issues.apache.org/jira/browse/YARN-7813 - Trunk (applied through branch-3.1) and branch-3.0 (applied through branch-2.9) patches differ from branch-2.8 patch @@ -425,7 +501,21 @@ message QueueInfoProto { optional string defaultNodeLabelExpression = 9; optional QueueStatisticsProto queueStatistics = 10; optional bool preemptionDisabled = 11; - optional bool intraQueuePreemptionDisabled = 12; + repeated QueueConfigurationsMapProto queueConfigurationsMap = 12; + optional bool intraQueuePreemptionDisabled = 13;
