Re: Amazon AWS credits for Debian QA work; help needed!
On Sun, Oct 23, 2011 at 02:29:31PM +0200, Lucas Nussbaum wrote:
> I've been willing to move my archive rebuilds work outside Grid'5000 for
> a while, since it would allow others to participate in that work as
> well.
>
> At this point, I'm fairly confident that we will get sponsored by Amazon
> to be able to run those tests on the AWS Cloud. I don't have the exact
> amount yet, but it's likely that we get enough credits to run ~60 full
> archive rebuilds.
That's *great* news, thanks for working on it. The work you've been
doing on rebuilds is invaluable and generalizing it so that it can be
done on other infrastructures out there is even more so. Kudos.
> There's some work to do to convert the existing scripts to leverage
> Amazon infrastructure. For example, we could use Amazon SQS for the
> queueing, etc. Is someone interested in working on that?
I can't offer coding time myself, but I'd like to discuss the
requirements you've in mind.
For instance, given we're generalizing the code, it'd be nice to have it
not too much tied to Amazon-specific technologies. 60 full rebuilds are
quite a bit, but won't last forever and I'd oppose betting on the fact
that Amazon will be kind to Debian forever in the future. I bet that
targeting Amazon's API would make it possible to run the rebuilds also
on Eucalyptus private clouds, is that correct? Having a simple
indirection layer that enables to use other infrastructures
(e.g. OpenStack) would be lovely.
Cheers.
--
Stefano Zacchiroli zack@{upsilon.cc,pps.jussieu.fr,debian.org} . o .
Maître de conférences .. http://upsilon.cc/zack .. . . o
Debian Project Leader... @zack on identi.ca ...o o o
« the first rule of tautology club is the first rule of tautology club »
signature.asc
Description: Digital signature
Re: Load on udd.d.o
On Mon, Oct 24, 2011 at 08:48:16AM +0200, Stefano Zacchiroli wrote: > > Is this something that the publicity team could help with; putting out > > a call for donations like was done for snapshot.debian.org? > > Before going down that path, DSA and/or HW coordination decides a) that > they want a new machine to that and b) its configuration. Doing a call > for (*hardware*) donation without having decided that could confuse > things up. Once that settled, -publicity help would indeed be useful (as > an idea: we can mention in it what UDD is used for and how it's useful > for the project). Whatever can be done - I'm in favour of it because Blends tools heavily relay on UDD and are currently beaten very hard by slow response. The tools I'm running are delayed by a factor of at least 20 compared to the situation about three months ago. Due to a rewrite of the Blends code only half of the stuff I planned to run is currently active. The cron job for this half used to run about 1 hour when it was implemented and currently we are facing the situation that it is not finishing on one day (and I need to prevent that a competing job will be startet the next day). So I can not even seriosely think of doing even more queries against UDD which would be needed to bring back all functionality we had about one year ago. Kind regards Andreas. -- http://fam-tille.de -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024070053.ga26...@an3as.eu
Re: Amazon AWS credits for Debian QA work; help needed!
On 24/10/11 at 09:04 +0200, Stefano Zacchiroli wrote: > For instance, given we're generalizing the code, it'd be nice to have it > not too much tied to Amazon-specific technologies. 60 full rebuilds are > quite a bit, but won't last forever and I'd oppose betting on the fact > that Amazon will be kind to Debian forever in the future. I bet that > targeting Amazon's API would make it possible to run the rebuilds also > on Eucalyptus private clouds, is that correct? Having a simple > indirection layer that enables to use other infrastructures > (e.g. OpenStack) would be lovely. I agree, but I'm not too worried about that. AWS provides two things: - a IaaS cloud: provisioning of virtual machines (Amazon EC2) - additional services to ease application development: storage (S3), distributed nosql DB (SimpleDB), SQL DB (RDS), distributed cache (ElastiCache), queueing (SQS), etc. In that sense, it's in the middle between a IaaS cloud and a PaaS cloud. Free Software Cloud stacks such as Eucalyptus and OpenStack are usually restricted to replacing EC2 and S3. However, we have plenty of packages that could replace the other Amazon services. Sometimes they are even API-compatible with Amazon services: ElastiCache uses the same API as memcached. I agree that it would be a bad thing to design an infrastructure that relies deeply AWS services. But I think that it's fine to use them as long as it's not too hard to replace them by services provided by Debian packages. I know that this is also a compromise between efficiency of development and freeness of the resulting infrastructure. In the end, it should probably be a decision that is taken by the one writing the code. Lucas -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024103523.ga8...@xanadu.blop.info
Re: Load on udd.d.o
Hi, [Please note that this is my very personal opinion!] On Mon Oct 24, 2011 at 10:06:11 +0800, Paul Wise wrote: > On Sun, Oct 23, 2011 at 7:46 PM, Peter Palfrader wrote: > > On Sun, 23 Oct 2011, Lucas Nussbaum wrote: > > > >> But the main problem, anyway, is that samosa is a bit low on RAM (only 6 > >> GB), and a bit slow on I/O. In the long term, it would be useful to move > >> UDD to a faster box... > > > > We probably don't have anything like that tho. > > Is this something that the publicity team could help with; putting out > a call for donations like was done for snapshot.debian.org? Throwing money/newer hardware on this will not help for something that is literally broken by design. You can not expect, if everyone[1] can do any SQL query on UDD, this will perform nicely. You will never be able to do proper indexing for every possible query. Users are and ever will be insane. If persons really want to do special queries, UDD offers the SQL dump for download so they could run this on their local machine. While investing in new hardware might make UDD a bit faster, it will not solve the underlying problem and performance issues will either not go away at all or return very quickly. If Debian wants to invest some of its funds in renewing our gear there probably are more important services to us and our users that could benefit from new HW. For instance putting bugs.debian.org on SSDs might be a really nice idea. Having it on machines covered by warranty would certainly not be wrong either. Cheers, Martin [1] who has an alioth account, which is easy to get. -- Martin Zobel-Helas | Debian System Administrator Debian & GNU/Linux Developer | Debian Listmaster GPG key http://go.debian.net/B11B627B | GPG Fingerprint: 6B18 5642 8E41 EC89 3D5D BDBB 53B1 AC6D B11B 627B -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024121625.gb1...@ftbfs.de
Re: Load on udd.d.o
On Mon, 24 Oct 2011, Andreas Tille wrote: > On Mon, Oct 24, 2011 at 08:48:16AM +0200, Stefano Zacchiroli wrote: > > > Is this something that the publicity team could help with; putting out > > > a call for donations like was done for snapshot.debian.org? > > > > Before going down that path, DSA and/or HW coordination decides a) that > > they want a new machine to that and b) its configuration. Doing a call > > for (*hardware*) donation without having decided that could confuse > > things up. Once that settled, -publicity help would indeed be useful (as > > an idea: we can mention in it what UDD is used for and how it's useful > > for the project). > > Whatever can be done - I'm in favour of it because Blends tools heavily > relay on UDD and are currently beaten very hard by slow response. You probably picked the wrong tool/service to rely on. UDD cannot be fast by design. It's a data dump that tries to collect as much information as possible, not facilitate fast and efficient queries. Cheers, -- | .''`. ** Debian ** Peter Palfrader | : :' : The universal http://www.palfrader.org/ | `. `' Operating System | `-http://www.debian.org/ -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024121705.gl8...@anguilla.noreply.org
Re: Load on udd.d.o
On 24/10/11 at 14:16 +0200, Martin Zobel-Helas wrote: > Hi, > > [Please note that this is my very personal opinion!] > > On Mon Oct 24, 2011 at 10:06:11 +0800, Paul Wise wrote: > > On Sun, Oct 23, 2011 at 7:46 PM, Peter Palfrader wrote: > > > On Sun, 23 Oct 2011, Lucas Nussbaum wrote: > > > > > >> But the main problem, anyway, is that samosa is a bit low on RAM (only 6 > > >> GB), and a bit slow on I/O. In the long term, it would be useful to move > > >> UDD to a faster box... > > > > > > We probably don't have anything like that tho. > > > > Is this something that the publicity team could help with; putting out > > a call for donations like was done for snapshot.debian.org? > > Throwing money/newer hardware on this will not help for something that is > literally broken by design. You can not expect, if everyone[1] can do any > SQL query on UDD, this will perform nicely. You will never be able to do > proper indexing for every possible query. Users are and ever will be > insane. If persons really want to do special queries, UDD offers the > SQL dump for download so they could run this on their local machine. > > While investing in new hardware might make UDD a bit faster, it will > not solve the underlying problem and performance issues will either not > go away at all or return very quickly. > > If Debian wants to invest some of its funds in renewing our gear there > probably are more important services to us and our users that could > benefit from new HW. For instance putting bugs.debian.org on SSDs might > be a really nice idea. Having it on machines covered by warranty would > certainly not be wrong either. I'm not going to argue the respective importances of the BTS and UDD. But saying that UDD is broken by design, thus should not be given faster hardware, and then saying that the BTS should get faster hardware, is a bit strange. The BTS uses a myriad of small files to store the bugs data, and using a proper database could probably remove the need for faster hardware, as well as bring a lot of new features (proper multi-criteria search, for example). Now, I understand that this requires quite large changes in the BTS, and that we don't have anybody willing to work on them. Anyway, I agree that UDD cannot be fast by design. We just need to make it sufficiently fast to be useful. One way to move forward could be to have two instances of UDD: - one hosted on samosa, that only does the importing and the execution of "official" services that run reasonable queries for which UDD has been optimized (by adding indexes, etc). - one on another machine (alioth?), that would be available to everyone to hack on UDD-based tools. That one could have a strict query duration limit. Also, wouldn't it be possible to just add more RAM to samosa ? It only has 6 GB, and RAM isn't that expensive nowadays. With <$1000, we could probably get a significant performance increase. Note that the recent load problems were apparently caused by a lack of VACUUMing in the DB. I'm not sure why auto-vacuum doesn't work as expected, but I'm going to cron a VACUUM FULL once per month. This doesn't solve the root issue, though. (Andreas, could you confirm that the situation improved for you too?) Lucas -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024125918.ga12...@xanadu.blop.info
Re: Load on udd.d.o
On Mon, Oct 24, 2011 at 02:17:05PM +0200, Peter Palfrader wrote: > On Mon, 24 Oct 2011, Andreas Tille wrote: > > > Whatever can be done - I'm in favour of it because Blends tools heavily > > relay on UDD and are currently beaten very hard by slow response. > > You probably picked the wrong tool/service to rely on. UDD cannot be > fast by design. It's a data dump that tries to collect as much > information as possible, not facilitate fast and efficient queries. Hmmm, for what purpose do we collect these data if it is not designed for getting data efficiently out of it? Your response opens more questions than it answers. Kind regards Andreas. -- http://fam-tille.de -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024130543.ga11...@an3as.eu
Re: Load on udd.d.o
On Mon, 24 Oct 2011, Lucas Nussbaum wrote: > Also, wouldn't it be possible to just add more RAM to samosa ? It only > has 6 GB, and RAM isn't that expensive nowadays. With <$1000, we could > probably get a significant performance increase. I don't think it'd be easy, no. Many DDs seem to not understand that most machines debian has are several years old by now. 6 gigs on a i386 machine already is an awful lot. We just don't have enough new hardware to go around. Samosa for instance is a DL380-g3, a model that was *retired* by its manufaturer in January 2005. Maximum memory configuration on this particular model is 6 gig, the machine is maxxed out. > Note that the recent load problems were apparently caused by a lack of > VACUUMing in the DB. I'm not sure why auto-vacuum doesn't work as > expected, but I'm going to cron a VACUUM FULL once per month. > This doesn't solve the root issue, though. CLUSTERing specific tables might be cheaper in my experience. -- | .''`. ** Debian ** Peter Palfrader | : :' : The universal http://www.palfrader.org/ | `. `' Operating System | `-http://www.debian.org/ -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024132445.gm8...@anguilla.noreply.org
Re: Load on udd.d.o
On Mon, Oct 24, 2011 at 02:59:18PM +0200, Lucas Nussbaum wrote: > Anyway, I agree that UDD cannot be fast by design. We just need to make > it sufficiently fast to be useful. One way to move forward could be to > have two instances of UDD: > - one hosted on samosa, that only does the importing and the execution > of "official" services that run reasonable queries for which UDD has > been optimized (by adding indexes, etc). > - one on another machine (alioth?), that would be available to everyone > to hack on UDD-based tools. That one could have a strict query > duration limit. Depending on this limit it would make sense for the purpose of blends.alioth.debian.org which would end up in accessing a local database and should speed up things even more. > Also, wouldn't it be possible to just add more RAM to samosa ? It only > has 6 GB, and RAM isn't that expensive nowadays. With <$1000, we could > probably get a significant performance increase. See private mail. > Note that the recent load problems were apparently caused by a lack of > VACUUMing in the DB. I'm not sure why auto-vacuum doesn't work as > expected, but I'm going to cron a VACUUM FULL once per month. > This doesn't solve the root issue, though. > > (Andreas, could you confirm that the situation improved for you too?) Yes, drastically: $ time ./tasks.py debian-med real6m12.815s user1m22.989s sys 0m0.964s These are the "old" numbers, before samosa went slowly! This is not the longest running Blend but it is a very good sign that for all Blends the job will run less than one hour again. If we could keep this status it would be sufficient for my purposes. In the "slow times" the time above was about 300min which is definitely inacceptable. Kind regards Andreas. -- http://fam-tille.de -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024141313.gb12...@an3as.eu
Re: Load on udd.d.o
On 24/10/11 at 15:24 +0200, Peter Palfrader wrote: > On Mon, 24 Oct 2011, Lucas Nussbaum wrote: > > > Also, wouldn't it be possible to just add more RAM to samosa ? It only > > has 6 GB, and RAM isn't that expensive nowadays. With <$1000, we could > > probably get a significant performance increase. > > I don't think it'd be easy, no. Many DDs seem to not understand that > most machines debian has are several years old by now. 6 gigs on a i386 > machine already is an awful lot. We just don't have enough new hardware > to go around. > > Samosa for instance is a DL380-g3, a model that was *retired* by its > manufaturer in January 2005. > > Maximum memory configuration on this particular model is 6 gig, the > machine is maxxed out. OK, I see. But given that Debian has some spare money, why don't we renew those such machines that are very old, out of warranty, etc? Lucas -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024144506.ga8...@xanadu.blop.info
UDD: statement_timeout for guest user set to 120s
Hi, Martin Zobel-Helas just set statement_timeout to 120s on UDD for the guest user. If you are affected by that (i.e, your UDD-based application can no longer work properly), please talk to me, so we can figure out if the DB is sufficiently optimized for your queries. There's also the option of using another user account for those queries. But I would prefer if you did not do that without asking first, so we can understand the use cases where UDD doesn't perform correctly. Lucas -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024144918.gb8...@xanadu.blop.info
Re: Load on udd.d.o
On Mon, 24 Oct 2011, Lucas Nussbaum wrote: On Mon, 24 Oct 2011, Lucas Nussbaum wrote: > OK, I see. But given that Debian has some spare money, why don't we > renew those such machines that are very old, out of warranty, etc? We are purchasing a new lists and new syncproxy.eu/ftp.d.o box at this moment. For various reasons these things take longer than expected and they are way, way more painful than they have any right to be. Currently, we seem to get little to no support from people outside of DSA on this. Maybe Debian doesn't want us spending money on hardware. So one part seems to be that buying new gear isn't easy or fast and instead is very painful. The other problem is that ever since the debconf and debian accounts got merged, we have no clue whatsoever about how much money we actually have for debian and for hardware. It's certainly not an awful lot. So even when we actually manage to get to buy hardware, we don't really buy the gear we like, but instead compromise and hope that what we purchase will be up for the job. Which brings us to a third part. If we actually wanted to replace everything that's older than say 3 or even 5 years with new systems, we couldn't afford it. Not by a long shot. -- | .''`. ** Debian ** Peter Palfrader | : :' : The universal http://www.palfrader.org/ | `. `' Operating System | `-http://www.debian.org/ -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024151718.gn8...@anguilla.noreply.org
Re: Load on udd.d.o
Hi, On Mon Oct 24, 2011 at 17:17:19 +0200, Peter Palfrader wrote: > Which brings us to a third part. If we actually wanted to replace everything > that's older than say 3 or even 5 years with new systems, we couldn't afford > it. Not by a long shot. Debian got a lot of new machines by donations in the past, machines that had been stocked with the best and newest parts at that time. The time we got those donations is over and will most probably never ever come back. These machines don't come cheap if you actually have to buy them one at a time. You get four or six gigs pretty quickly, but if you want what counts of decent amounts of ECC RAM, say 24 or 32 gigs - which isn't too crazy, it's going to set you back around a grand just for memory. And that's nothing compared to storage. You may think disk space is inexpensive these days, and maybe it is if you put your data on 2tb consumer disks you buy from the discounter around the corner, but storage in servers still costs a small fortune (~1-2 USD per gig, on 2.5" 10k SAS disks, and you don't just want a single one of them). Throw in another half grand to a grand for the raid controller or the "optional" battery to enable the existing controller's write cache. Add CPUs, chassis, extended warranty and it's peanuts no more. So far we've bought maybe two and a half machines from Debian money. And that's in all of our history. Maybe we could buy more, but do we have the funds? And if we do, can we justify spending that amount of money if the existing gear still mostly does its job? Cheers, Martin PS: now, you can try to cut down on costs of individual machines by for instance moving storage out of the server itself and making it part of the local infrastructue. I.e. provide storage via iscsi/fc/nfs etc. But these things again don't come for free. You have to get that infrastructure, and it suddenly is a spof. And it requires that new machines get hosted at the place that has the storage. And then it probably isn't as fast as a local raid10. Oh well. Nobody said it was easy :) -- Martin Zobel-Helas | Debian System Administrator Debian & GNU/Linux Developer | Debian Listmaster GPG key http://go.debian.net/B11B627B | GPG Fingerprint: 6B18 5642 8E41 EC89 3D5D BDBB 53B1 AC6D B11B 627B -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024160033.gd1...@ftbfs.de
Re: Load on udd.d.o
On Mon, Oct 24, 2011 at 05:17:19PM +0200, Peter Palfrader wrote:
> The other problem is that ever since the debconf and debian accounts got
> merged, we have no clue whatsoever about how much money we actually have for
> debian and for hardware.
This should not be your problem.
The interface for spending Debian money is the DPL, just tell him/her
that you need to buy something and say how much it will cost. It is then
up to the DPL to decide whether Debian has enough money for that or not,
taking into account all future expected expenses (DebConf included), and
discussing it with the community (DebConf organizers included) if
needed.
You seem to be assuming that, in the past, looking at Debian's balance
(and excluding DebConf balance from it) was enough to quantify how much
money were available to buy hardware. That assumption was flawed in its
own right also before the DebConf/Debian merge, because there might be
other non-hardware expenses you were not aware of. An important example
of that are sprints; one year of sprints can easily dominate hardware
expenses or not, depending on the year.
Cheers.
--
Stefano Zacchiroli zack@{upsilon.cc,pps.jussieu.fr,debian.org} . o .
Maître de conférences .. http://upsilon.cc/zack .. . . o
Debian Project Leader... @zack on identi.ca ...o o o
« the first rule of tautology club is the first rule of tautology club »
signature.asc
Description: Digital signature
Re: Load on udd.d.o
On Mon, Oct 24, 2011 at 06:00:33PM +0200, Martin Zobel-Helas wrote: > Debian got a lot of new machines by donations in the past, machines that > had been stocked with the best and newest parts at that time. The time > we got those donations is over and will most probably never ever come > back. It would be interesting to learn more about this statement. I guess you are basing it on the numbers donated hardware / year. But where do you see the reasons which let you assume that they most probably never ever come back. Are we doing something wrong / not so good as in those days when we got the donations? Kind regards Andreas. -- http://fam-tille.de -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024163850.ga16...@an3as.eu
Re: Load on udd.d.o
Hi, On Montag, 24. Oktober 2011, Peter Palfrader wrote: > The other problem is that ever since the debconf and debian accounts got > merged, we have no clue whatsoever about how much money we actually have > for debian and for hardware. Uhm, why? The DebConf budget is well known, so I don't see why this merge complicated things. Could you be a bit more specific what the problem is? I'd like to help you here. cheers, Holger -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201110241859.47983.hol...@layer-acht.org
Re: UDD: statement_timeout for guest user set to 120s
On Mon, Oct 24, 2011 at 04:49:18PM +0200, Lucas Nussbaum wrote: > Martin Zobel-Helas just set statement_timeout to 120s on UDD for the > guest user. Seems this limit is sufficient for the queries issued by the Blends tools - provided samosa remains at the current speed. Kind regards Andreas. -- http://fam-tille.de -- To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20111024200940.gb2...@an3as.eu
