TLDR: Pipelines should be relatively faster soon
---
As you know, PR gates can take a long time in edk2. While there are
various improvements that could be made, I want to highlight a change in
the queue now, it's impact, and how it will help.
Pipelines are composed of jobs which are composed of steps. Pipelines
are scheduled onto build machines (agents) at job granularity. Today, a
matrix of build configurations kicks off many jobs for a single pipeline
run.
Examples of pipelines that contain jobs:
- Windows VS2019 PR
- https://dev.azure.com/tianocore/edk2-ci/_build?definitionId=34
- Number of jobs spawned: 14
- Ubuntu GCC5 PR
- https://dev.azure.com/tianocore/edk2-ci/_build?definitionId=33
- Number of jobs spawned: 15
- PlatformCI_OvmfPkg_Windows_VS2019_PR
- https://dev.azure.com/tianocore/edk2-ci/_build?definitionId=51
- Number of jobs spawned: 11
- PlatformCI_OvmfPkg_Ubuntu_GCC5_PR
- https://dev.azure.com/tianocore/edk2-ci/_build?definitionId=49
- Number of jobs spawned: 19
- PlatformCI_EmulatorPkg_Windows_VS2019_PR
- https://dev.azure.com/tianocore/edk2-ci/_build?definitionId=45
- Number of jobs spawned: 12
- PlatformCI_EmulatorPkg_Ubuntu_GCC5_PR
- https://dev.azure.com/tianocore/edk2-ci/_build?definitionId=42
- Number of jobs spawned: 6
- PlatformCI_ArmVirtPkg_Ubuntu_GCC5_PR
- https://dev.azure.com/tianocore/edk2-ci/_build?definitionId=47
- Number of jobs spawned: 18
- tianocore.PatchCheck
- https://dev.azure.com/tianocore/edk2-ci/_build?definitionId=35
- Number of jobs spawned: 1
Total Jobs Per PR: 96
Each job must acquire an "agent" of which 30 are available to the
project. So you can see, how several PRs with recent pushes can lead to
outcomes like this:
This agent request is not running because you have reached the maximum
number of requests that can run for parallelism type 'Microsoft-Hosted
Public'. Current position in queue: 617. Current parallelism: 30. Max
Parallelism for parallelism type 'Microsoft-Hosted Public': 30
There is a lot that could be discussed about reducing jobs,
conditionalizing builds on others, etc. But, that's out of the scope of
this email.
---
Another issue exacerbates the problem. It comes from a single job in two
of those pipelines ("Ubuntu GCC5 PR" and " Windows VS2019 PR") whose
task is to wait for every other job in the pipeline to finish, so it can
download the code coverage results from each and merge them into a
single code coverage report to publish for the pipeline run.
That job is queued after the last job from the original matrix of all
other jobs in the pipeline is done. This means that those two pipelines
(therefore the PR) are essentially being thrown back to the beginning of
the build queue when they should have completed.
For example, if 2 more PRs had been pushed while the matrix jobs for a
pipeline are running, that single code coverage job which takes about 3
to 5 minutes itself to run, is now going to cause the entire PR pipeline
results to be queued behind 2 x ~96 = ~192 other jobs where some jobs
take up to 30 minutes. This doesn't include CI pipelines that also add
to the build queue when a change is merged.
---
PR 5978 (https://github.com/tianocore/edk2/pull/5978) disables the code
coverage job as a near term solution until someone has enough time to
test a new approach that is more efficient.
This means the code coverage report will no longer be available in
pipeline runs. However, the individual coverage.xml files will still be
published in the artifacts for each job, and you can create a converged
report locally from those pipeline artifacts if needed.
The pipeline itself has an example of how to do that:
https://github.com/tianocore/edk2/blob/master/.azurepipelines/templates/pr-gate-build-job.yml#L100
Thanks,
Michael
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#120053): https://edk2.groups.io/g/devel/message/120053
Mute This Topic: https://groups.io/mt/107567922/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
