On Fri, Jul 22, 2016 at 9:39 PM, Don Lewis <truck...@apache.org> wrote:
> On 22 Jul, Damjan Jovanovic wrote: > > I've progressed much further, and openoffice-fbsd-nightly, > > openoffice-linux32-nightly, openoffice-linux64-nightly, and > > openoffice-linux64-rat are now building, while > openoffice-linux32-snapshot > > is only temporarily breaking due to SourceForge issues. I've also made > some > > interesting discoveries about the Windows buildbots. > > FreeBSD still needs LWP::Protocol::https. It's only working because of > the non-https fallback to a specific SourceForge mirror. > > No, it's only working because of --with-system-<everything> Also LWP::Protocol::https is dead now, please see my upcoming email. > > I had to revert r1736692 (in r1753709), setting main/extensions.lst back > to > > generic https:// SourceForge URLs, as the mirror-specific http:// URLs > are > > now broken, which was causing all buildbots that use > > --enable-bundled-dictionaries to fail. Enough buildbots support https now > > to make this a net benefit. > > The iweb mirror is out of service, but switching the main SourceForge > URL to pilotfiber (in r1752780) fixed that. > > > Also had to upload openssl-0.9.8zg to ooo-extras for the > > openoffice-linux32-snapshot, but the most recent build failed because it > > couldn't download some dictionaries/languages from SourceForge, which I > am > > generally finding to be too flaky. > > > > I think we should either run ./bootstrap multiple times on the buildbots, > > or list SourceForge URLs several times in external_deps.lst and > > extensions.lst, to compensate for SourceForge's unreliability. Buildbots > > should immediately fail the build if ./bootstrap fails, as there is no > > hope: ./bootstrap only downloads dependencies needed for the build, and > if > > any are missing, the build will definitely fail, and burning CPU for up > to > > 5 hours is pure waste. Alternatively we should find a more reliable > > ooo-extras hosting solution than SourceForge. We could also cache > > dependencies offline between builds, but I am guessing that has licensing > > issues? > > Yes, there is no sense in continuing if bootstrap fails. > > > That leaves the Windows bots. > > aoo-w7snap is still missing LWP::Protocol::https. > > aoo-win7 was failing to delete the old build files (rm: cannot remove > > `build/ext_libraries/apr/wntmsci12.pro/misc/build/apr-1.4.5/Makefile.win > ': > > Device or resource busy). > > Something seems to be keeping that file open even after the previous > builds > > are over. > > According to ext_libraries/apr/makefile.mk, the BUILD_ACTION on Windows > is: > > INCLUDE="$(INCLUDE);./include" nmake -f Makefile.win buildall > > so I suspected nmake hangs. > > > > I patched the build script to run "ps -W" (results at > > https://ci.apache.org/builders/aoo-win7/builds/348/steps/ps/logs/stdio) > > which showed 4 nmake processes, 2 from March 30, 2 from April 4, part of > > orphaned trees also containing perl, sh, and dmake. > > Killing nmake (through hacks to the buildbot script, as I still don't > have > > remote access) had no effect - deleting apr-1.4.5/Makefile.win was still > > giving the same error. > > Even killing dmake, sh and perl (which had to be done in creative ways on > > that dodgy OS - some through taskkill, some through Cygwin's kill) still > > had no effect. > > After all Cygwin processes were dead, that error was still coming up! > > > > So I started looking at non-Cygwin processes. DEVENV.EXE and DEVENV.COM > had > > the same March 30 / April 4 start times as the hung process trees, and > > killing them *finally* allowed apr-1.4.5/Makefile.win to be deleted. > > > > I'll experiment a lot more over the weekend, but right now I think the > > problem could be that the buildbot runs VSVARS.BAT to set up the Visual > > Studio environment, which (presumably) contains the evil DEVENV that > break > > the build and locks files while hung. On my own Windows VM I don't run > > VSVARS.BAT, and I can't reproduce the problem. Do the rest of you that > > build on Windows use it? > > Since VSVARS.BAT was not listed in the step by step guide, I haven't > been running it. It sounds a lot like it just sets some environment > variables to find where the various bits of VS are installed, so I would > think it would be generally harmless. I've been running builds from the > desktop. VSVARS.BAT might be needed when running headless ... > > I did some searches and saw some mention that DEVENV.* hangs might be > caused by anti-virus software. Is that running on the windows buildbot? > > I caught AOO building ext_libraries/apr on my own setup (where the buildbot usually fails and locks files), and quickly ran "ps -W" and "tasklist". DEVENV wasn't running. According to Infra, that buildbot is running "the basic Windows Defender stuff". > > That buildbot is currently running out of disk space, and it doesn't help > > that we "svn co" and then "svn export" a second copy. Originally the > > buildbot script used other tricks, like "svn switch", or keeping an SVN > > checkout across builds that was just updated and then exported from for > > each build, but some time ago I switched to a full "svn co" because it > was > > too unreliable (eg. files can get locked and need "svn cleanup"). With a > > full checkout there is no need to export, as we get a fresh copy each > time. > > I'll overhaul that buildbot script and try make it simpler and cleaner. > > Doing an update followed by an export of that would be less resource > intensive, though it adds 50% (since .svn isn't copied) to the space > consumed by the source. The write-locked file problem should only occur > if the update is interrupted, and since there is so little activity in > the repo, the updates should be pretty fast have a low probability of > failing. A full checkout would be much more likely to fail in the > middle. You could always run "svn cleanup" before "svn update". > > A checked out source tree for 4.1.2 is about 725 MB. A second exported > copy would bring the total up to about 1090 MB, which is still fairly > small compared to the overall build size. If space is an issue and load > on the svn server is not, then doing a fresh export (vs a checkout) from > the server each time would be the best. > > Space was an issue because I was using the small C: drive instead of E:. As the buildbots run automatically and we usually don't initiate builds and wait for results, little is lost by doing SVN update the slowest way, and much is gained through the increased reliability and simplicity.
