Jens Geyer created THRIFT-6324:
----------------------------------
Summary: Cut the AppVeyor build matrix and build in parallel
Key: THRIFT-6324
URL: https://issues.apache.org/jira/browse/THRIFT-6324
Project: Thrift
Issue Type: Improvement
Components: Build Process
Reporter: Jens Geyer
h3. Problem
Six jobs, roughly 3 h 07 min of job time squeezed into 1 h 45 min of wall
clock. Measured on build 0.26.0.10733 and cross-checked against 0.26.0.10724, a
different pull request with near-identical timings.
*The Python matrix is five nearly identical builds.* Five of the six jobs are
MSVC2022 builds that differ only in {{PYTHON_VERSION}} (3.10 through 3.14).
Each one builds the whole C++ library, the compiler and every test executable
-- 21 min 27 s -- in order to run 12 Python tests that account for 112 s of the
248 s {{ctest}} run. The other 54 tests are C++ and identical across all five
jobs.
GitHub Actions already runs the full 3.10 to 3.14 matrix twice, in
{{lib-python}} on ubuntu-24.04 and in {{lib-python-macos}}, both off the same
YAML anchor in {{.github/workflows/build.yml}}. What AppVeyor adds on top is
the MSVC build of the {{fastbinary}} C extension ({{lib/py/setup.py}}) and the
Windows behaviour of the pure-Python tests. Neither of those changes between
3.11, 3.12 and 3.13 -- breakage shows up at the edges of the range, not in the
middle.
*Nothing is built in parallel.* {{cmake --build}} is invoked without
{{--parallel}} in both scripts:
* MSVC: MSBuild builds one project at a time. The log shows some 40 test
executables completing strictly in sequence, 3 to 15 s apart.
* MinGW: {{mingw32-make}} runs single-threaded for 34 min 13 s of that job's 46
min.
h3. Change
* {{appveyor.yml}}: keep Python 3.11 and 3.14, drop 3.10, 3.12 and 3.13. 3.11
rather than 3.10 as the lower edge because 3.10 reaches end of life on
2026-10-31, so the matrix holds for a year; 3.14 becomes 3.15 once the AppVeyor
image carries it.
* {{build/appveyor/MSVC-appveyor-full.bat}} and
{{build/appveyor/MINGW-appveyor-full.bat}}: pass {{--parallel
%NUMBER_OF_PROCESSORS%}} to {{cmake --build}}. An explicit number rather than a
bare {{--parallel}}, which for the Makefile generator means an unbounded {{make
-j}}.
* {{build/appveyor/win_showenv.bat}}: report {{NUMBER_OF_PROCESSORS}}, so the
logs say how much parallelism a worker actually offers.
Expected effect: job time down from about 3 h 07 min to about 1 h 42 min from
the matrix change alone, plus whatever the parallel build adds on top of that.
h3. Not in scope
Running time is the smaller half of the problem. Over the last 23 finished
builds the median time spent waiting in the queue was 6 h 26 min, against 1 h
43 min of actual building. That is a project setting on the AppVeyor side and
is being raised separately with ASF Infra.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
