On Mon, 25 Aug 2008, Szakáts Viktor wrote: Hi Viktor,
> >Classic problem with memory fragmentation. Algorithms > >used by your C compiler to allocate memory from OS and > >then divide it for application do not work well with such > >code which is BTW killer for many memory managers. > >We can try to reduce this problem by adding code for > >array preallocation anyhow programmers should now about > >it and try to write more memory manager friendly code. > The problem with this is that such code (AAdd( a, v )) > is fairly common in regular Clipper code and even in > Harbour core .prg code. In tbrowse.prg for example, > there is no way to know the number of columns in advance. > teditor/dirscan are also affected. There is no problem with AADD() but with the _very_ large array and memory manager which is simply fatal. F.e. your code works perfectly on Linux with GLIBC. I added at the end: ? memory( HB_MEM_USED ) compiling it with FMSTAT and it reports: 4965585 It allocates less then 5MB for Harbour structures. It means that it should not need more then 20MB from OS if memory manager does not have some serious internal problems (in Linux total application memory is 17MB). I guess you are using BCC. Please try to compile Harbour with -DHB_FM_WIN32_ALLOC and check the results. It will be interesting if also default windows memory manager is effected. This example is very trivial and should not be a problem for good memory manager. > The end result is that it's difficult to really exploit > the "unlimited" size of arrays in Harbour since it > chokes much earlier because of this. IOW AADD() can > easily be considered as dangerous. If memory manager does not have some basic protection against memory fragmentation then any function which allocates memory can be dangerous in long working programs. Sooner or later we will have to implement our own memory manager which will be tuned for Harbour structures. > Maybe we should try to replicate the xhb preallocation > (and maybe lazy shrinking) methods to avoid this. But > you probably have some other and/or better ideas too. We will add preallocation but I would like to join this modification with our own memory manager. Please remember that some CRTLs have quite nice MM and works very well without user preallocation, f.e. the one from GLIBC in Linux. Adding item preallocation like in xHarbour arrays reduces only the problem. But if your example exploits it with C compiler you are using then I can easy create other example which will also exploit current xHarbour code. > [ I also wonder what other Harbour parts could suffer > from the same effect. ] Pure AADD() is not dangerous as long as total memory consumption is less then half of total memory available for process. To recreate the problem it's necessary to resize many times very large array allocating new memory blocks between resizing using some primitive or buggy memory manager. AFAIR we only have one place which may need very large arrays resized dynamically. It's HB_DirScan(). I even though about changing this code to make preallocation but I was too lazy ;-( I can update it quite fast. best regards, Przemek _______________________________________________ Harbour mailing list Harbour@harbour-project.org http://lists.harbour-project.org/mailman/listinfo/harbour