Hi Christian Thanks a lot for all the info. Sorry to acknowledge your email a bit later than I should have. Best E
On 21/06/10 16:18, Christian Thaeter wrote: > E Chalaron wrote: > >> Hi all >> >> I am thinking of building a machine to do exclusively numbers munching . >> I need a lot of post processing (sometimes 1 to 2 TB of data) with >> colour correction through histogram, unsharp masking, stabilisation etc ... >> >> So ... >> >> 1. I am quite found of AMDs but eventually what I need is not a >> multitasking box, I need a calculator. >> > as long you don't processing in the temporal domain but only on single > frames, things are easily parallelizeable so multicore is an option > 6x2ghz is still (way) more than 1x3ghz, the different stages of > processing can be even assigned to single cores (or do batch processing > in stipes, first only color correction, save to disk, then unsharp > masking). But caveat, bandwidth may become a limiting factor too, first > disk bandwidth (and latency) and next, when you do just simple not very > compute intensive transformations memory bandwidth might become a > limiting factor too. > > For disk you certaily want some fast raid (raid 10) if you do the batch > rendering then possibly even 2 arrays that you can play ping-pong > rendering from one to the other back and forth, the 2nd one can then be > just raid0 on the expense that you may loose some (temporary) data in > case of a harddisk error, and be sure that you finally end with the > results on the array with redundance. Since this processing is highly > linear, much ram won't help much with caching, while when you have > enough ram you can tweak the disk accesses quite a bit (readahead, delay > writes, getting overall better performance because of less seeks) this > makes mostly sense when you have only one array where you read from and > write to. I highly recommend one of the newer/better filesystems which > can defer writes more aggressively and join writes (ext4) and some > hardcore kernel tuneing when you have to do this processing often. > > For memory bandwidth it will only help to estimate (benchmark) what you > really need and then choose cpu/memory accordingly. IIRC AMD Opterons > are still best there because they have a NUMA architecture where each > socket has its own memory, but that of course requires a mainboard with > memory slots for each socket and equipped with sufficient memory > (dual/triple channel). > > > >> 2. I wish I could use Cinelerra on a PS3 with Yellow Dog but not >> sure if that will ever be possible. >> > I somehow doubt that and even then, the plugins are not CELL aware, that > would need some serious coding efforts. We have some ideas for Lumiera > in that area but this isnt priority there either. > > >> 3. I don't know much about Intel these days. >> > iirc the current chips are damn fast (and cost a lot) the memory > bottleneck got improved but i think its still not as scalable as > opterons (which are pricey too). > > When you really need a box to do this things (only) and often then it > makes sense to evaluate and benchmark your exact demands and choose the > hardware which fits it best. > > >> My idea is to have a core Cinelerra box in batch mode that will just go >> through the data located on other boxes as fast as possible. >> Eventually using rendering farm is an option too, but then the process >> will be as long as the slowest box... >> so ... not sure where from there... >> Any ideas? suggestions ? etc ... >> > dunno if i would really use cinelerra for it, it doesn't scale that well > and when something gets wrong you have to start over again. If its only > this batch rendering as you described i'd look into some commandline > tools to build the rendering pipe. ffmpeg, mencoder, gstreamer, AML.. if > you have enough disk bandwidth as i saied above then you may even do the > batch rendering with temporary files on disk, the advantage is that when > you do less at once you get overall better performance (due caching) and > can easily recover/restart when something broke in the middle. But this > really stands and fails with the disk bandwidth, if the cpu's start > waiting on the disks it will not work. > > > > Christian > > >> Thanks a lot >> Edouard >> >> > > _______________________________________________ > Cinelerra mailing list > [email protected] > https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra > > _______________________________________________ Cinelerra mailing list [email protected] https://init.linpro.no/mailman/skolelinux.no/listinfo/cinelerra
