2009.04.17 12:21, Joost Andrae rašė: > Hi Rimas, > >> So, in theory, it shouldn't be that hard to have one or two master >> servers hosting let's say, a terabyte of data, right? > > It is not about having one or two servers hosting the files. It is > about more than 100 servers hosting files for millions of users (> > 50000000 downloads since version 3).
This was covered in my further text. My point was that those hundreds of servers don't really have to (and maybe even shouldn't) have all the data that is present on masters. >> I think what would make sense is the following: >> 1) to have a consistent directory structure (i.e. the location of this >> version builds in this language should be predictable) on master >> servers. While looking at our project page >> [http://distribution.openoffice.org/files.html#sets], everything seems >> very clear already: ALL QA'd localized binaries should go to >> /localized/, while unQA'd files go to /extended/localized/. That makes > > Unfortunately that's not true. All files released by native language > teams are distributed via ../localized or via ../extended/localized. I > have to use ../extended/localized because the regular mirror network > lacks about 10 GB to host these files. Bouncer has been modified to be > able to handle ../extended content. That's precisely what I meant – the stuff isn't laid according to the plan anymore. It shouldn't be that way though. >> sense. I guess our project's problem is that we failed to accept the >> fact that with growing number of supported languages, the size of >> /localized/ (and thus sets that include it) also grows. Maybe it's time >> to face it... >> By the way, currently /localized/ uses 19GB, meanwhile at least for the >> last year the aforementioned page says it is about 30GB. I wonder how > > The size of the regular mirror network has a size of about 30 GB. The > extended mirror network has more disk space. According to the table on http://distribution.openoffice.org/files.html#sets, the size of category 2 mirror is about 37GB, and category 1 is 43GB. 30GB in that table is the size of /localized/ alone. >> QA'd builds got into /extended/localized/ at all... I just adjusted my >> script to fetch this whole folder, so I hope to see how much all our >> QA'd builds actually take up now. > > Thank you. That's good. So, let's get the facts: r...@faktas:/mnt/data/ftp/OpenOffice.org$ du -hs localized/ 19G localized/ r...@faktas:/mnt/data/ftp/OpenOffice.org$ du -hs extended/localized/ 9,9G extended/localized/ So, the sum is roughly 30GB – just as big as /localized/ alone should have been. >> 2) for space-concerned mirrors to mirror localized builds selectively. >> <snip> To help reduce the headache for mirror admins, we could group >> localizations logically into sets (and maybe subsets). E.g., the set >> I host >> could be a part of "Eastern Europe" set. >> 3) this should work very well when combined with geographical mirror >> allocation, since local mirrors would be getting most of the request for >> certain language builds anyway. I think the number of servers in a given >> country is proportional to the number of users in that country, so for >> smaller countries it's OK to have less/slower servers, meanwhile bigger >> and more advanced countries would have more and faster servers anyway. > > Bouncer 2.0 has no Geo IP support. We're currently evaluating a > different load balancing solution (MirrorBrain) as an alternative. Doesn't Bouncer 2.0 know which servers host which files, as well as MirrorBrain? But anyway, I only mentioned GeoIP because I am actually reading the mirrors list and I know that MirrorBrain is just about to be used and that it has GeoIP support. It's not a requirement at all. The requirement is for the load balancer to know where to find what files. And MirrorBrain does that AFAIK. >> From my POV, language packs are second class citizens. They take >> additional time and effort to discover and install, and should not be >> the only way to get OOo in your language. >> > > If there were enough disk space on all mirror servers then this > wouldn't be a problem at all. We could easily provide more than 160 GB > for each milestone but that's not realistic. Didn't I offer a solution for exactly this problem? Please re-read point 2) above, I just removed the unnecessary example from it. Best regards, Rimas --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
