Tom Wijsman posted on Sun, 16 Jun 2013 23:24:27 +0200 as excerpted: > On Sun, 16 Jun 2013 19:33:53 +0000 (UTC) > Duncan <1i5t5.dun...@cox.net> wrote: > >> TL;DR: SSDs help. =:^) > > TL;DR: SSDs help, but they don't solve the underlying problem. =:-(
Well, there's the long-term fix to the underlying problem, and there's coping strategies to help with where things are at now. I was simply saying that an SSD helps a LOT in dealing with the inefficiencies of the current code. See the "quite apart... practical question of ... dealing with the problem /now/" bit quoted below. > I have one; it's great to help make my boot short, but it isn't really a > great improvement for the Portage tree. Better I/O isn't a solution to > computational complexity; it doesn't deal with the CPU bottleneck. But here, agreed with ciaranm, the cpu's not the bottleneck, at least not from cold-cache. It doesn't even up the cpu clocking from minimum as it's mostly filesystem access. Once the cache is warm, then yes, it ups the CPU speed and I see the single-core behavior you mention, but cold- cache, no way; it's I/O bound. And with an ssd, the portage tree update (the syncs both of gentoo and the overlays) went from a /crawling/ console scroll, to scrolling so fast I can't read it. >> Quite apart from the theory and question of making the existing code >> faster vs. a new from-scratch implementation, there's the practical >> question of what options one can actually use to deal with the problem >> /now/. > > Don't rush it: Do you know the problem well? Does the solution properly > deal with it? Is it still usable some months / years from now? Not necessarily. But first we must /get/ to some months / years from now, and that's a lot easier if the best is made of the current situation, while a long term fix is being developed. >> FWIW, one solution (particularly for folks who don't claim to have >> reasonable coding skills and thus have limited options in that regard) >> is to throw hardware at the problem. > > Improvements in algorithmic complexity (exponential) are much bigger > than improvements you can achieve by buying new hardware (linear). Same song different verse. Fixing the algorithmic complexity is fine and certainly a good idea longer term, but it's not something I can use at my next update. Throwing hardware at the problem is usable now. >> --- >> [1] I'm running ntp and the initial ntp-client connection and time sync >> takes ~12 seconds a lot of the time, just over the initial 10 seconds >> down, 50 to go, trigger on openrc's 1-minute timeout. > > Why do you make your boot wait for NTP to sync its time? Well, ntpd is waiting for the initial step so it doesn't have to slew so hard for so long if the clock's multiple seconds off. And ntpd is in my default runlevel, with a few local service tasks that are after * and need a good clock time anyway, so... > How could hardware make this time sync go any faster? Which is what I said, that as a practical matter, my boot didn't speed up much /because/ I'm running (and waiting for) the ntp-client time- stepper. Thus, I'd not /expect/ a hardware update (unless it's to a more direct net connection) to help much. >> [2] ... SNIP ... runs ~1 hour ... SNIP ... > > Sounds great, but the same thing could run in much less time. I have > worse hardware, and it doesn't take much longer than yours do; so, I > don't really see the benefits new hardware bring to the table. And that > HDD to SSD change, that's really a once in a lifetime flood. I expect I'm more particular than most about checking changelogs. I certainly don't read them all, but if there's a revision-bump for instance, I like to see what the gentoo devs considered important enough to do a revision bump. And I religiously check portage logs, selecting mentioned bug numbers probably about half the time, which pops up a menu with a gentoo bug search on the number, from which I check the bug details and sometimes the actual git commit code. For all my overlays I check the git whatchanged logs, and I have a helper script that lets me fetch and then check git whatchanged for a number of my live packages, including openrc (where I switched to live-git precisely /because/ I was following it closely enough to find the git whatchanged logs useful, both for general information and for troubleshooting when something went wrong -- release versions simply didn't have enough resolution, too many things changing in each openrc release to easily track down problems and file bugs as appropriate), as well. And you're probably not rebuilding well over a hundred live-packages (thank $DEITY and the devs in question for ccache!) at every update, in addition to the usual (deep) @world version-bump and newuse updates, are you? Of course maybe you are, but I did specify that, and I didn't see anything in your comments indicating anything like an apples to apples comparision. >> [3] Also relevant, 16 gigs RAM, PORTAGETMPDIR on tmpfs. > > Sounds all cool, but think about your CPU again; saturate it... > > Building the Linux kernel with `make -j32 -l8` versus `make -j8` is a > huge difference; most people follow the latter instructions, without > really thinking through what actually happens with the underlying data. > The former queues up jobs for your processor; so the moment a job is > done a new job will be ready, so, you don't need to wait on the disk. Truth is, I used to run a plain make -j (no number and no -l at all) on my kernel builds, just to watch the system stress and then so elegantly recover. It's an amazing thing to watch, this Linux kernel thing and how it deals with cpu oversaturation. =:^) But I suppose I've gotten more conservative in my old age. =:^P Needlessly oversaturating the CPU (and RAM) only slows things down and forces cache dump and swappage. These days according to my kernel-build- script configuration I only run -j24, which seems a reasonable balance as it keeps the CPUs busy but stays safely enough within a few gigs of RAM so I don't dump-cache or hit swap. Timing a kernel build from make clean suggests it's the same sub-seconds range from -j10 or so, up to (from memory) -j50 or so, after which build time starts to go up, not down. > Something completely different; look at the history of data mining, > today's algorithms are much much faster than those of years ago. > > Just to point out that different implementations and configurations have > much more power in cutting time than the typical hardware change does. I agree and am not arguing that. All I'm saying is that there are measures that a sysadmin can take today to at least help work around the problem, today, while all those faster algorithms are being developed, implemented, tested and deployed. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
