On 01/04/2016 09:23 AM, Jack J. Woehr wrote: > Jerry Callen wrote: >> Jack J. Woehr wrote: >> >>> Not sure how relevant that this is to mainframe programming, but >>> years ago >>> when I designed and executed with a team of nine a data-heavy server in >>> Unix optimized for multiple cores, what we found was that reroutable >>> queuing >>> of data from one simplistic processing engine to the next (with >>> reservoirs >>> for data accumulation) got the most performance. >> I'm not sure I grok this. Are you talking about hardware or software >> (or a combination) > > Software running on multithreaded Linux on 8-16 cores. > > We had all sorts of data processing that happened on an steady > hi-volume stream of incoming data before the massaged data got parked > in the database. > > Our software architecture was based on that fine principle of Naval > Engineering enunciated so many decades ago: > > "All machines are the same. There's a gozinta, a gozouta, and in > the middle there's a pocketa-pocketa." > > The pocketa-pocketas were simple threads on cores. The gozintas and > gozoutas were MQSeries queues. > > Incoming data was dealt into various MQSeries queues. > > Threads were doing simple processing steps. > > Each thread took from a queue, processed, and then wrote to one of > several queues as appropriate, where the next thread for the next > appropriate processing step did the same etc., until finally written > to the database. > > Worked well with Linux multithreading architecture on multiple cores. > Keep all the cores balanced in load. > The basic underlying concept here seems to be that data that needs to be touched soon by some other process needs to be kept where it may be more quickly accessed than less needed data. While this can be addressed on the application design level, this has taken all sorts of forms above the application level on IBM mainframes over the decades.
Regarding keeping loads on processor cores balanced in a Unix environment, I suspect this is not relevant to MVS, which I think is more concerned with trying to minimize unnecessary CP context switching to optimize processor cache usage, even though this can result in a very skewed processor utilization under less than 100% load. Throwing more buffers at very active VSAM files, using LSR VSAM buffer pools in CICS regions to share in-memory data across many thousands of CICS transactions, and throwing large amounts of real memory at DB2 buffer pools to retain recently referenced DB2 table data pages in memory are all techniques that have been used on IBM mainframes under MVS to keep data that may be needed again soon in main memory without modifying application code; and the mainframe hardware support for virtual memory paging, processor cache, DASD controller cache and DASD device cache all serve a similar purpose at the hardware level. I am certain that one can obtain higher performance with fewer physical resources when an application design explicitly and correctly designates which output data will be quickly needed again by another process, like the mentioned approach of designing the application to use a system of internal, in-memory queues; but the approach that IBM seems to have taken in recent years of addressing performance problems at a higher level has the advantage that you may be able, with somewhat more resources, to adequately improve the performance of many applications without having to analyze and potentially redesign individual, imperfectly-designed applications. The rather dramatic reduction in the relative cost of hardware over the last decade or so has increased the attractiveness of letting the Operating System and hardware approximate which data is likely to benefit from being kept most accessible, especially when needed insight to do that explicitly may have been lacking or impractical during application design. -- Joel C. Ewing, Bentonville, AR [email protected] ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
