On Sat, May 9, 2020 at 12:34 AM Valdis Klētnieks <valdis.kletni...@vt.edu> wrote: > > On Fri, 08 May 2020 16:40:20 -0700, Dave Taht said: > > > while I'm referring to stuff that's actually fun and off-topic... I've long > > off been watching progress in this area, knowing that > > "fixing bufferbloat" is a key requirement for technologies like this to > > succeed. > > Amen to that. Probably around a decade ago, $DAYJOB at the time was > approached > by a vendor who had a very compelling and cost-effective erasure-coded storage > product that would have allowed us to have the data at 3 sites, and total loss > of any one would be survivable. Two sites were about 3 cable miles apart, and > the third was 95 cable miles and could only run at 10Gig speed back then. > > And what sank the deal was uncertainty if we could meet throughput > requirements > under heavy load with one site that distant. (You'd be *amazed* what happens > in the first 3 minutes after semester add/drop goes live at midnight, when you > have 35,000 students. :) > > The other great bugaboo - How do you back up a half billion files that total > 12+ petabytes? (It doesn't help the solution space when the majority of it is > one research group that (a) will probably eventually data-mine their data for > $100M+ in research grants and also (b) the terms of some of the grants > specified a 30 year retention on the data *and* some wicked nasty PII issues, > because the data includes identifiable video of people who had not consented > to > be part of the research study...
Well, someone(s) are probably doing that with government money anyway. > Storage - it turns out you can't just throw 10 terabyte drives at the > problem. :) I still rather like bittorrrent. I've come to understand, over the years, why it got structured the way it did. The typical behavior of having 5 open streams and switching between them every 15 seconds or so is a direct outgrowth of bufferbloat, compensating for slow start in an overbuffered universe, and maxing out at 100ms observed induced delay, even with the latecomer problem, works as best as it can. the further, more recent adopion of vpns for it add intrinsic delay, and that also makes the impact of the protocol on edge networks less. It's not quite a dead protocol, and the implementations could still be better of course, But in no case can I come up with some way to keep file integrity for 10PB with even thousands of volunteer users providing chunks of storage. Hard problem! That said, thousands of nodes providing resiliency for the semester end/drop - far worse than mothers day - problem - does makes sense. the whole covid-19 thing - trying to match available hospital resources against the potential growth in infection rate - is bufferbloat in the real world - covidbloat - and I wish more folk understood that the queue theory problems are the same. -- Make Music, Not War Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-435-0729 _______________________________________________ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
