Bill Kenworthy <billk <at> iinet.net.au> writes:
> rpm is just a wrapper around a an archive with instructions on how to > build and or install it. I have more experience with rpm's but I > believe debs are the same. Just unwrap your .rpm/.deb file of choice > and install it manually (the binaries/code are usually in a zip inside > the rpm). You should in most cases also be able to get a source rpm > (which I suspect you are talking about anyway, but binaries do work deps > permitting. Yes the point is to be able to rapidly/easily be able to test and evaluate many cluster/hpc codes, and other codes too, before I open them up and look at them in detail, manually. > you can install rpm and then install your package via rpm - seem to > remember doing this with .debs somehow too. deps are a problem but > usually workable. This is the argument for setting up an entire RH or debian workstation to quickly evaluate the codes to find that less than %5 which are fast and worthy of a deeper look. There are dozens of 'schedulers' for clusters and then the concept of a 'framework' is even more loose or unconstrained and the myriad of associated codes (modules) that one can use. It fact many are pushing 'write your own framework'. It's hard to categorize codes into a discernible 'stack' when it comes to HPC or cluster offerings. Apache says one thing, a guy writing a book says another. It's like the wild wild west with lots of folks (mostly large corps) shoot from the hip and shooting off at the mouth. Many idiots and deceivers in this space; and I intend to benchmark, profile and expose some of the myths. So then I'll unpack the sources onto a gentoo system for deep examination and low level (profile ) testing and evaluation and maybe creating an ebuild if I like what I see. Distributing bloatware across a cluster or on a HPC system, is a fools errand, imho, but that is exactly what many are doing. It's very hard to figure out if they are 'flim_flam' artists or sincerely belief their own musings. > and why set up a workstation? - this sort of thing is tailor made for > vm's. Create a base for your experiments with essential packages, > settings etc, snapshot it (golden master) and then throwaway->restore > when finished with that iteration. Ultimately, I think a VM or containers environment would be keen. I liked docker, initially, but, like systemd, it is now bloating so much that it is becoming an entire operating system on it's own. Docker is too much for me to worry about and learn. CoreOS is still attractive to me, but it probably too much for me to divert into, at this time. I'm really more after HPC (high performance computing of scientific codes) than clustering of a bizzilion processes (like log/web file analytics). That sort of stuff is not hard to distribute, hadoop is sufficient for those routine, boring codes and processes that are not time_critical. I'll do some of those ordinary tasks but many cluster/hpc solutions already work good enough for routine admin style processes, imho. Big Science solutions using cluster or HPC are still very fragmented if you look at some low level performance data. It takes a very specialized focus to solve a given category of Big Science codes if you want to approach any sort of reasonable robustness. It's like you need a custom designed (HPCC) High Performance Computational Cluster with a unique (at least highly customized and tweaked) implementation for each of the many different possible configuration solutions. This part is evolving on a daily basis, so apologies if it seems sketchy because it is sketchy at best. I think where I'll end up is a myriad of HPCC, architecturally unique, running along side of each other, mostly asleep until needed. It's even more 'fluid' when you consider the GPU resources, arm64 processors and the new integrated CISC chips like the amd APU and the Intel+fpga chips. > There are package managers besides gentoo/portage that can do a source > build/install and track the files on gentoo - though portage will not > know about it (rpm is one :) Hmmmm. Where do I read more about this? > and lastly, what do mean error prone? - to me a manual install is the > ONLY way you can build a proper ebuild that catches most of the > problems. In the (admittedly few) ebuilds I have done an occasional > human error is nothing compared to system problems for a difficult > package. Error prone with manual steps, due to the large amount of codes I am evaluating. Once I get down to less than 5% then the manual processes are fine, if not superior, as I'm also reviewing the codes of interest. It's also me, if I do not impose self_structure, I get sloppy with trying to manually speed things up. Also, if you go back to the days of .configure and look at many different make files, there is little coding consistency among different codes; boost and other lib codes are the best you can hope for on consistency, imho. ymmv. OK? Keep in mind that I'm an EE, hardware guy that started with discrete transistors, native assemble and state machines; and C was a luxury, so my perspectives do not often 'jive' with the entire OO world, but, I am trying to get along....... [1] > BillK James [1] http://www.phy.duke.edu/~rgb/Beowulf/c++_interview/c++_interview.html
