On Wednesday, August 10, 2016 10:13:29 AM james wrote: > On 08/10/2016 07:45 AM, Michael Mol wrote: > > On Tuesday, August 09, 2016 05:22:22 PM james wrote:
> >> > >> I did a quick test with games-arcade/xgalaga. It's an old, quirky game > >> with sporadic lag variations. On a workstation with 32G ram and (8) 4GHz > >> 64bit cores, very lightly loaded, there is no reason for in game lag. > >> Your previous settings made it much better and quicker the vast majority > >> of the time; but not optimal (always responsive). Experiences tell me if > >> I can tweak a system so that that game stays responsive whilst the > >> application(s) mix is concurrently running then the quick > >> test+parameter settings is reasonably well behaved. So thats becomes a > >> baseline for further automated tests and fine tuning for a system under > >> study. > > > > What kind of storage are you running on? What filesystem? If you're still > > hitting swap, are you using a swap file or a swap partition? > > The system I mostly referenced, rarely hits swap in days of uptime. It's > the keyboard latency, while playing the game, that I try to tune away, > while other codes are running. I try very hard to keep codes from > swapping out, cause ultimately I'm most interested in clusters that keep > everything running (in memory). AkA ultimate utilization of Apache-Spark > and other "in-memory" techniques. Gotcha. dirty_bytes and dirty_background_bytes won't apply to anything that doesn't call mmap() with a file backing or perform some other file I/O. If you're not doing those things, they should have little to no impact. Ideal values for dirty_bytes and dirty_background_bytes will depend heavily on the nature of your underlying storage. Dozens of other things might be tweaked depending on what filesystem you're using. Which is why I was asking about those things. > > > Combined codes running simultaneously never hits the HD (no swappiness) > but still there is keyboard lag. Where are you measuring this lag? How much lag are we talking about? > Not that it is actually affecting the > running codes to any appreciable degree, but it is a test I run so that > the cluster nodes will benefit from still being (low latency) quickly > attentive to interactions with the cluster master processes, regardless > of workloads on the nodes. Sure its not totally accurate, but so far > this semantical approach, is pretty darn close. It's not part of this > conversation (on VM etc) but ultimately getting this right solves one of > the biggest problems for building any cluster; that is workload > invocation, shedding and management to optimize resource utilization, > regardless of the orchestration(s) used to manage the nodes. Swapping to > disc is verbotim, in my (ultimate) goals and target scenarios. > > No worries, you have given me enough info and ideas to move forward with > testing and tuning. I'm going to evolve these into more precisely > controlled and monitored experiments, noting exact hardware differences; > that should complete the tuning of the Memory Management tasks, within > acceptable confine . Then automate it for later checking on cluster > test runs with various hardware setups. Eventually these test will be > extended to a variety of memory and storage hardware, once the > techniques are automated. No worries, I now have enough ideas and > details (thanks to you) to move forward. You've got me curious, now you're going to go run off and play with your thought problems and not share! Tease! > > >> Perhaps Zabbix +TSdB can get me further down the pathway. Time > >> sequenced and analyzed data is over kill for this (xgalaga) test, but > >> those coalesced test-vectors will be most useful for me as I seek a > >> gentoo centric pathway for low latency clusters (on bare metal). > > > > If you're looking to avoid Zabbix interfering with your performance, > > you'll > > want the Zabbix server and web interface on a machine separate from the > > machines you're trying to optimize. > > agreed. > > Thanks Mike, > James np -- :wq
signature.asc
Description: This is a digitally signed message part.
