Re: Code Review Systems Was: [gentoo-dev] Git Migration: launch plan & schedule
Hi! This got a bit rambly, sorry 'bout that. tl;dr: Continuous Integration is a neat idea if you have the Hardware. The off-mainstream arch teams don't. On Tue, 07 Jul 2015, Ciaran McCreesh wrote: > On Tue, 07 Jul 2015 08:04:47 +0800 > Patrick Lauer wrote: > > I'll laugh about it next time I update OpenFOAM: > > > > Fri Jun 27 12:52:19 2014 >>> sci-libs/openfoam-2.3.0 > >merge time: 1 hour, 5 minutes and 8 seconds. > > or > > > > Sun Jun 29 20:36:09 2014 >>> sci-mathematics/nusmv-2.5.4 > >merge time: 2 hours, 58 minutes. > > What's the problem? Better the build bot wastes that time once than a > whole bunch of users do. Just be aware that that Just Won't Happen for the off-mainstream archs. We just don't have the hardware to keep up with something like that. Unless you want to employ several people reordering the CI checks so a test for glibc doesn't make mutt, util-linux, iptables and 55 other packages wait for hours. 'Cause invariably, if the CI system is delayed beyond the expectation for the package itself, it will be ignored. It would also mean that the other purposes the devboxes are used for will be delayed non-trivially. In the case of Alpha, that would for example be upstream gcc testing, and stage3 building. I am sure arches like PPC64, IA64 and so on have the same problem. > > That's without dependencies, so from a clean minimal starting point > > (once you figure out what useflags are needed) you're looking at 12+ > > hours of walltime to get that checked. (On a reasonably modern Xeon > > with SSDs ...) > > Uh, no, because you use binary packages for the dependencies, and you > use a package mangler that can figure out the use flags for you. Even with binary package usage (which is not as efficient as it could be due to USE flags), testing Qt4 (as in compile it and run the test suite, which is the bare minimum) took me close to 10 hours the other day. Not including fixing fetch problems halfway through. During this time, very little else got done on the one devbox we have. > > So thanks for your intentional comedy, but let's be serious here. > > Exherbo's been doing all this for years, and it works rather well. The > only comedy is that Gentoo doesn't even realise this is trivial to do > nowadays. Exherbo is not supporting as many off-mainstream architectures as we do, so the comparison is flawed at best. Yes, sure, let's run CI for amd64 and maybe x86 and whoever else has the hardware. But it will not speed up stabilization bugs regarding the long tail. It might even lead to people completely ignoring the slow architectures and the rift between amd64 and the rest becoming yet wider. Don't get me wrong, abandoning Alpha and other architectures is an option, but remember that for some of these, we are the last distro that still provides install media and a decent (as in: current and complete-ish) set of packages. The very fact that we are able to let amd64 and the rest drift a bit is the reason why we can still pull it off. I have considered dropping Alpha entirely (for myself, that is; I'd resign as the arch team lead and stop working on it; the devbox, which is soft-donated through me, would stay available). But I would rather not, for various reasons, none of which are technical. Then again, I'm not a Gentoo dev because it makes sense rationally (it does, but it's not my reason). In essence, assuming we can "just scale" to make CI work is ignoring the matter of the slower archs. And I suspect the "it works on amd64, fuck everyone else" is not something we want to pick as a motto. Regards, Tobias` -- Sent from aboard the Culture ship GSV (System Class) Empiricist
Re: Code Review Systems Was: [gentoo-dev] Git Migration: launch plan & schedule
On Wed, 8 Jul 2015 20:07:34 +0200 Tobias Klausmann wrote: > In essence, assuming we can "just scale" to make CI work is > ignoring the matter of the slower archs. And I suspect the "it > works on amd64, fuck everyone else" is not something we want to > pick as a motto. "It works on amd64 on a clean build system" is a heck of a lot better than "it may or may not work on one developer's heavily modified box"... The point of testing is not to catch all problems. It's just there to catch some simple screwups, cheaply. -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] signatures in git work flow
On Sun, Jul 05, 2015 at 09:05:12PM -0400, Rich Freeman wrote: > All the gpg stuff really exposes the weakness of git being based on > sha1 though. I wouldn't think that it would be that hard to change > git's hash function, with the caveat that the resulting repositories > might not be backwards-compatible… If you want a Merkle DAG with flexible hashes, you may be interested in IPFS [1]. It doesn't have Git's source-control tooling yet, but it does a good job at passing around generic Merkle objects with self-identifying hashes [2]. That way you can make your hash as strong as you like, although you'll still have public-rebase style incompatibilities if you want to go back and republish an older part of the DAG with a stronger hash. Cheers, Trevor [1]: http://ipfs.io/ [2]: https://github.com/jbenet/multihash/ -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature
Re: Code Review Systems Was: [gentoo-dev] Git Migration: launch plan & schedule
On Wed, Jul 8, 2015 at 11:07 AM, Tobias Klausmann wrote: > Hi! > > This got a bit rambly, sorry 'bout that. > > tl;dr: Continuous Integration is a neat idea if you have the > Hardware. The off-mainstream arch teams don't. > Clearly because we cannot be perfect, we should not even try! Realistically aside from the major arches none of the other arches have large userbases. I think quality for the sake of quality is pretty silly; but if we can improve the quality of arches where hardware is readily available we should do so (regardless of minority architecture objections.) -A > > > On Tue, 07 Jul 2015, Ciaran McCreesh wrote: > > On Tue, 07 Jul 2015 08:04:47 +0800 > > Patrick Lauer wrote: > > > I'll laugh about it next time I update OpenFOAM: > > > > > > Fri Jun 27 12:52:19 2014 >>> sci-libs/openfoam-2.3.0 > > >merge time: 1 hour, 5 minutes and 8 seconds. > > > or > > > > > > Sun Jun 29 20:36:09 2014 >>> sci-mathematics/nusmv-2.5.4 > > >merge time: 2 hours, 58 minutes. > > > > What's the problem? Better the build bot wastes that time once than a > > whole bunch of users do. > > Just be aware that that Just Won't Happen for the off-mainstream > archs. We just don't have the hardware to keep up with something > like that. Unless you want to employ several people reordering > the CI checks so a test for glibc doesn't make mutt, util-linux, > iptables and 55 other packages wait for hours. > > 'Cause invariably, if the CI system is delayed beyond the > expectation for the package itself, it will be ignored. > > It would also mean that the other purposes the devboxes are used > for will be delayed non-trivially. In the case of Alpha, that > would for example be upstream gcc testing, and stage3 building. I > am sure arches like PPC64, IA64 and so on have the same problem. > > > > That's without dependencies, so from a clean minimal starting point > > > (once you figure out what useflags are needed) you're looking at 12+ > > > hours of walltime to get that checked. (On a reasonably modern Xeon > > > with SSDs ...) > > > > Uh, no, because you use binary packages for the dependencies, and you > > use a package mangler that can figure out the use flags for you. > > Even with binary package usage (which is not as efficient as it > could be due to USE flags), testing Qt4 (as in compile it and run > the test suite, which is the bare minimum) took me close to 10 > hours the other day. Not including fixing fetch problems halfway > through. During this time, very little else got done on the one > devbox we have. > > > > So thanks for your intentional comedy, but let's be serious here. > > > > Exherbo's been doing all this for years, and it works rather well. The > > only comedy is that Gentoo doesn't even realise this is trivial to do > > nowadays. > > Exherbo is not supporting as many off-mainstream architectures as > we do, so the comparison is flawed at best. Yes, sure, let's run > CI for amd64 and maybe x86 and whoever else has the hardware. But > it will not speed up stabilization bugs regarding the long tail. > It might even lead to people completely ignoring the slow > architectures and the rift between amd64 and the rest becoming > yet wider. > > Don't get me wrong, abandoning Alpha and other architectures is > an option, but remember that for some of these, we are the last > distro that still provides install media and a decent (as in: > current and complete-ish) set of packages. The very fact that we > are able to let amd64 and the rest drift a bit is the reason why > we can still pull it off. > > I have considered dropping Alpha entirely (for myself, that is; > I'd resign as the arch team lead and stop working on it; the > devbox, which is soft-donated through me, would stay available). > But I would rather not, for various reasons, none of which are > technical. Then again, I'm not a Gentoo dev because it makes > sense rationally (it does, but it's not my reason). > > In essence, assuming we can "just scale" to make CI work is > ignoring the matter of the slower archs. And I suspect the "it > works on amd64, fuck everyone else" is not something we want to > pick as a motto. > > Regards, > Tobias` > > > > -- > Sent from aboard the Culture ship > GSV (System Class) Empiricist > >
Re: Code Review Systems Was: [gentoo-dev] Git Migration: launch plan & schedule
Hi! On Wed, 08 Jul 2015, Ciaran McCreesh wrote: > On Wed, 8 Jul 2015 20:07:34 +0200 > Tobias Klausmann wrote: > > In essence, assuming we can "just scale" to make CI work is > > ignoring the matter of the slower archs. And I suspect the "it > > works on amd64, fuck everyone else" is not something we want to > > pick as a motto. > > "It works on amd64 on a clean build system" is a heck of a lot better > than "it may or may not work on one developer's heavily modified > box"... The point of testing is not to catch all problems. It's just > there to catch some simple screwups, cheaply. Agreed. Still, in my experience, problems that are not seen on amd64 are the vast majority of timesinks. Usually, by the time non-amd64 shows up on a non-security bug, the Basic Bullshit™ has been sorted. And even if it isn't: Arches are usually quick to point it out and the rest of arches move on to the next bug. The truly arch-dependent bugs are what wastes my time: For example: - dependencies not being keyworded for arch or ~arch but only amd64/~amd64 - dependencies not even building on ~arch, but on ~amd64 - package assuming availability of gcc/ghc/Python-X.Y when arch/~arch only has something for
Re: Code Review Systems Was: [gentoo-dev] Git Migration: launch plan & schedule
Hi! On Wed, 08 Jul 2015, Alec Warner wrote: > On Wed, Jul 8, 2015 at 11:07 AM, Tobias Klausmann > wrote: > > > tl;dr: Continuous Integration is a neat idea if you have the > > Hardware. The off-mainstream arch teams don't. > > Clearly because we cannot be perfect, we should not even try! > Realistically aside from the major arches none of the other arches have > large userbases. I think quality for the sake of quality is pretty silly; > but if we can improve the quality of arches where hardware is readily > available we should do so (regardless of minority architecture objections.) Read what I wrote further down. I explicitly point out that amd64 is welcome to do whatever they want regarding CI, but that I suspect that the rift between it and the "lesser" architectures will become wider. Regards, Tobias -- Sent from aboard the Culture ship GSV (System Class) Empiricist
Re: [gentoo-dev] A question to Russian Gentoo Developers Community about import software substitution
How can I unsucribe from this mailing list On May 9, 2015 1:25 AM, "Dmitry Yu Okunev" wrote: > 08.05.2015 23:05, Vadim A. Misbakh-Soloviov пишет: > >> I had been trying to push the idea of creating an united FOSS community >>> to solve problems of the higher school of the Russian Federation. But >>> such initiatives faded due to absence of support of top executives… And >>> now (according to e.g. [1]) it won't be a problem, IMHO. >>> >> >> It would. Just because of the fact, that "top executives" (in Education >> Dept, >> in regional ministerys and so on) is that kind of clerks, that WOULDN'T >> accept >> anything if it is impossible to get (read as "steal") some money in >> personal. >> > > The link [1] shows that the _can_ get money in personal. > > And I'd prefer >>> make a distribution for the higher shool based on Gentoo Linux. >>> >> >> I'd also prefer, but it is not that possible as you imagine. There is such >> thing as certification in Federal Security Service (FSB) and so on. >> > > 1. It's not a big problem. We can certify a release of our Gentoo based >distribution. It's just need a money to pay to laboratories. Plus >big universities (like my) have enough social ties to fast push >such things through Russian bureaucracy machines, IMO. And we >can return the spent money the same way as ALT Linux (by selling >tech support coupons). > > 2. The Federal Service for Technical and Export Control (ФСТЭК) and >Federal Security Service (ФСБ) certificates required only to process >big systems with personal data (ФЗ-152). For the rest systems (like >ordinal desktop) it's not required at all. > > So >>> here's the Question: >>> >>> Does anybody interested in creating a consortium to send an application >>> to the Ministry of Communications? >>> >> >> I'm partially interested to mentally maintain that, but I have to free >> time to >> activelly do anything in personal (but, probably, will be fine in a >> group). >> > > I just offer to unite efforts. You don't need to free an additional time. > We just need to find common problems and solve it together as a project for > Ministry of Communications. Other good people will connect to us and > meanwhile the support of the ministry will stimulate the top executives. > > Specifically, I'm interested in making a distribution for the higher > school of Russian Federation. > > But, as I said, I doubt in success of that operation. But... Let the >> Force be >> with us... >> > > There was a lot of doubtable (but good) projects in my life. However few > of them successfully started and completed. I think we _must_ try if we > believe in FOSS. Sorry for this pathos :) > > P.S. I'd not use politic-related phrases (like "import software >> substitution") >> in international communities at all and here in particular. >> > > Sorry for that. I was thinking that it's better to make subject more > concrete. > > > Best regards, Dmitry. > >
Re: Code Review Systems Was: [gentoo-dev] Git Migration: launch plan & schedule
On 07/08/2015 09:14 PM, Tobias Klausmann wrote: > > I explicitly point out that amd64 > is welcome to do whatever they want regarding CI, but that I > suspect that the rift between it and the "lesser" architectures > will become wider. > That is technically correct, but it's not really an argument. The quality of the lesser arches will be the same as it was before, it will not get worse. The quality of the different arches has _never_ been consistent. Not even between x86 and amd64. It always depends on manpower, resources and how much it is used in the community. I feel we are drifting a bit offtopic.
Re: [gentoo-dev] A question to Russian Gentoo Developers Community about import software substitution
Tyler Pohl wrote: > > How can I unsucribe from this mailing list > > https://gentoo.org/get-involved/mailing-lists/instructions.html Dale :-) :-)
Re: Code Review Systems Was: [gentoo-dev] Git Migration: launch plan & schedule
On Wed, 2015-07-08 at 21:11 +0200, Tobias Klausmann wrote: > Hi! > > On Wed, 08 Jul 2015, Ciaran McCreesh wrote: > > On Wed, 8 Jul 2015 20:07:34 +0200 > > Tobias Klausmann wrote: > > > In essence, assuming we can "just scale" to make CI work is > > > ignoring the matter of the slower archs. And I suspect the "it > > > works on amd64, fuck everyone else" is not something we want to > > > pick as a motto. > > > > "It works on amd64 on a clean build system" is a heck of a lot better > > than "it may or may not work on one developer's heavily modified > > box"... The point of testing is not to catch all problems. It's just > > there to catch some simple screwups, cheaply. > > Agreed. Still, in my experience, problems that are not seen on > amd64 are the vast majority of timesinks. Usually, by the time > non-amd64 shows up on a non-security bug, the Basic Bullshit™ has > been sorted. And even if it isn't: Arches are usually quick to > point it out and the rest of arches move on to the next bug. The > truly arch-dependent bugs are what wastes my time: > > For example: > > - dependencies not being keyworded for arch or ~arch but only > amd64/~amd64 > - dependencies not even building on ~arch, but on ~amd64 > - package assuming availability of gcc/ghc/Python-X.Y when > arch/~arch only has something for - my favourite: assuming udev is at version X for arch/~arch when > it has been broken there for roughly twelve kernel versions due > to udev/systemd upstream not caring. The information is one > equery-y command line away, but no, let's file that bug. > That's a failing of the arch team or the committer then - or whomever keyworded the package without testing the dependencies. That's why the keywording bugs happen when dependencies change - and yes, occasionally a dependency loses it's keywords and that may be where the issue comes from, I'm not sure. This used to be watched very closely, but I believe the tool being used broke at some point. What exactly do you mean, it's just one command line away but let's file the bug - yes a bug SHOULD be filed, so that the people know of the issue so it doesn't get repeated, over and over. > Having every arch team chase the deps for every package is very, > very frustrating, since it is so trivial a problem. We need a > tool that answers the question: "I want to mov cat/pkg-1.2.3 to > stable for arches a, b and c, what do I need?" for _every > package_. Some groups seem to have tooling for this, but it is > far from easily (obviously?) available, so every team gets to > re-discover dependency hell. Ruby is especially bad since > FEATURES=test make the depgraph explode (and sometimes circular). > > Regards, > Tobias > > The only fear I have about CI, is that we turn into every other distro out there where "it builds, ship it!" - I know I prefer to have our arm team actually test the packages more than just doing FEATURES=test (assuming the tests aren't broken on arm) emerge, although I know there are some people on the team who feel that it's too much to actually test all of the arm boards. For a long time, certain binary distros were compiling anything and everything, and if it built for other arches it was available. Even if there was no way it would possibly work on that arch. Regards, Steev
Re: Code Review Systems Was: [gentoo-dev] Git Migration: launch plan & schedule
On Wed, Jul 8, 2015 at 10:11 PM, Steev Klimaszewski wrote: > On Wed, 2015-07-08 at 21:11 +0200, Tobias Klausmann wrote: > > Hi! > > > > On Wed, 08 Jul 2015, Ciaran McCreesh wrote: > > > On Wed, 8 Jul 2015 20:07:34 +0200 > > > Tobias Klausmann wrote: > > > > In essence, assuming we can "just scale" to make CI work is > > > > ignoring the matter of the slower archs. And I suspect the "it > > > > works on amd64, fuck everyone else" is not something we want to > > > > pick as a motto. > > > > > > "It works on amd64 on a clean build system" is a heck of a lot better > > > than "it may or may not work on one developer's heavily modified > > > box"... The point of testing is not to catch all problems. It's just > > > there to catch some simple screwups, cheaply. > > > > Agreed. Still, in my experience, problems that are not seen on > > amd64 are the vast majority of timesinks. Usually, by the time > > non-amd64 shows up on a non-security bug, the Basic Bullshit™ has > > been sorted. And even if it isn't: Arches are usually quick to > > point it out and the rest of arches move on to the next bug. The > > truly arch-dependent bugs are what wastes my time: > > > > For example: > > > > - dependencies not being keyworded for arch or ~arch but only > > amd64/~amd64 > > - dependencies not even building on ~arch, but on ~amd64 > > - package assuming availability of gcc/ghc/Python-X.Y when > > arch/~arch only has something for > - my favourite: assuming udev is at version X for arch/~arch when > > it has been broken there for roughly twelve kernel versions due > > to udev/systemd upstream not caring. The information is one > > equery-y command line away, but no, let's file that bug. > > > > That's a failing of the arch team or the committer then - or whomever > keyworded the package without testing the dependencies. That's why the > keywording bugs happen when dependencies change - and yes, occasionally > a dependency loses it's keywords and that may be where the issue comes > from, I'm not sure. This used to be watched very closely, but I believe > the tool being used broke at some point. > > What exactly do you mean, it's just one command line away but let's file > the bug - yes a bug SHOULD be filed, so that the people know of the > issue so it doesn't get repeated, over and over. > > > > Having every arch team chase the deps for every package is very, > > very frustrating, since it is so trivial a problem. We need a > > tool that answers the question: "I want to mov cat/pkg-1.2.3 to > > stable for arches a, b and c, what do I need?" for _every > > package_. Some groups seem to have tooling for this, but it is > > far from easily (obviously?) available, so every team gets to > > re-discover dependency hell. Ruby is especially bad since > > FEATURES=test make the depgraph explode (and sometimes circular). > > > > Regards, > > Tobias > > > > > > The only fear I have about CI, is that we turn into every other distro > out there where "it builds, ship it!" - I know I prefer to have our arm > team actually test the packages more than just doing FEATURES=test > (assuming the tests aren't broken on arm) emerge, although I know there > are some people on the team who feel that it's too much to actually test > all of the arm boards. For a long time, certain binary distros were > compiling anything and everything, and if it built for other arches it > was available. Even if there was no way it would possibly work on that > arch. > > So don't keyword or stabilize packages you don't test thoroughly. I see how CI *could* be used to do bad things; but I don't see anyone proposing it be used in that way; nor do you have to accept such suggestions (its your arch, after all.) -A > Regards, > Steev > > >
