on Thu May 11 2017, Pavol Vaskovic <swift-dev-AT-swift.org> wrote: > Hi Mishal, > > Douglas Gregor suggested in SR-4814 (that was merged as dupe into > SR-4669) that I get in contact with you because I’m working on issue > SR-4669 Add a Benchmark_Driver --rerun N option > <https://bugs.swift.org/browse/SR-4669> and I should make sure that I > don’t break the Swift-CI. > > The point of that is to deal with false changes (regressions and > improvements) that are often seen > in benchmark reports from CI. > > I have run Benchmark_O with --num-iters=100 on my machine for the the > whole performance test suite, to get a feeling for the distribution of > benchmark samples, because I also want to move the Benchmark_Driver to > use MEAN instead of MIN in the analysis.
I'm concerned about that, especially for microbenchmarks; it seems to me as though MIN is the right measurement. Can you explain why MEAN is better? > There was a tremendous difference on the quality of measured samples > depending on how busy my machine was. When I had idle browser windows > open in the background, or Xcode open, the results had very high > variance. When I purged all I could the results were very stable. > > First I want to check with you that the Macs that run the CI and > benchmarks do have turned off the Spotlight indexing on the folder > that they check out the project into > (/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift). Just > making sure that `mds`, `mds_stores` and `mds_worker` processes don’t > unnecessarily impact the benchmark runs. How about Time Machine's > `backupd`? > > --- > > Also I have recently filed SR-4659 Benchmark logs should be tied to > tested tree version <https://bugs.swift.org/browse/SR-4659> and > implemented a fix that got merged in PR #8978 > <https://github.com/apple/swift/pull/8978>, adding short hash of the > commit to the Benchmark log file name. But that works only while you > don’t make local changes. In the end I would like to have fully > identified versions of benchmarks in the comparison report. Have you thought about using git notes to attach results to commits? That has always seemed to me like a good way to manage test results. > In order to make that work, I’ve filed SR-4675 Report detailed build > version from Benchmark drivers > <https://bugs.swift.org/browse/SR-4675>, that calls for having > something akin to `swift —version` be baked into the > Benchmark_O{Onone, Ounchecked}. > > Given we are on git now, I think the solution is to modify cmake to > run `git describe` and generate some kind of version file that will be > included during build and reported by the driver when running > benchmarks. > > I thought it would be nice touch to report also the branch names like one > that is cited in the PRs > on github. For example: > > apple:master vs palimondo:SR-4659 > > I found that getSVN.cmake > <https://github.com/apple/swift-llvm/blob/stable/cmake/modules/GetSVN.cmake#L73> > from llvm has all the commands to extract the needed information… But, > whenI looked at full console logs from swift-ci to see if that would > work with the way how the branches are checked out on the build > machine, I’m a bit confused: > >> Cloning the remote Git repository >> Cloning repository g...@github.com:apple/swift.git >> > git init /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift # >> timeout=10 >> Fetching upstream changes from g...@github.com:apple/swift.git >> > git --version # timeout=10 >> > git fetch --tags --progress g...@github.com:apple/swift.git >> +refs/heads/*:refs/remotes/origin/* >> > git config remote.origin.url g...@github.com:apple/swift.git # timeout=10 >> > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* >> # timeout=10 >> > git config remote.origin.url g...@github.com:apple/swift.git # timeout=10 >> Cleaning workspace >> > git rev-parse --verify HEAD # timeout=10 >> No valid HEAD. Skipping the resetting >> > git clean -fdx # timeout=10 >> Fetching upstream changes from g...@github.com:apple/swift.git >> > git fetch --tags --progress g...@github.com:apple/swift.git >> +refs/pull/*:refs/remotes/origin/pr/* >> > git rev-parse refs/remotes/origin/pr/9330/merge^{commit} # timeout=10 >> > git rev-parse refs/remotes/origin/origin/pr/9330/merge^{commit} # >> timeout=10 >> Checking out Revision 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 >> (refs/remotes/origin/pr/9330/merge) >> > git config core.sparsecheckout # timeout=10 >> > git checkout -f 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 >> Using 'Changelog to branch' strategy. >> [swift-PR-osx-perf] $ /bin/bash -e > /var/folders/_8/79jmzf2142z2xydc_01btlx00000gn/T/hudson7057602820198606732.sh >> Cloning into 'ninja'... >> Branch: master >> Found related pull requests: [] > > I don’t quite understand how is the PR branch cloned from github. It > definitely looks different from > the situation wehn I build on my local machine. > > What would be the way to extract remote repository and branch name for the PR > being merged? > > When I’ll be working on these changes, It would be able to build just > the benchmark suite, but SR-4593 Following instructions to > independently build benchmarks fails to do so > <https://bugs.swift.org/browse/SR-4593>. Who can help me here? (So far > I have always been using ./swift/utils/build-script -R, but that > always takes ages, and often rebuilds whole compiler infrastructure.) > > Best regards > Pavol Vaskovic > > _______________________________________________ > swift-dev mailing list > swift-dev@swift.org > https://lists.swift.org/mailman/listinfo/swift-dev > -- -Dave _______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
