Bugs item #1185883, was opened at 2005-04-19 12:07 Message generated for change (Settings changed) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1185883&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Interpreter Core Group: Python 2.3 Status: Closed Resolution: Fixed Priority: 5 Submitted By: Kristján Valur (krisvale) >Assigned to: Georg Brandl (gbrandl) Summary: PyObject_Realloc bug in obmalloc.c Initial Comment: obmalloc.c:835 If the previous block was not handled by obmalloc, and the realloc is for growing the block, this memcpy may cross a page boundary and cause a segmentation fault. This scenario can happen if a previous allocation failed to successfully allocate from the obmalloc pools, due to memory starvation or other reasons, but was successfully allocated by the c runtime. The solution is to query the actual size of the allocated block, and copy only so much memory. Most modern platforms provide size query functions complementing the malloc()/free() calls. on Windows, this is the _msize () function. ---------------------------------------------------------------------- Comment By: Georg Brandl (birkenfeld) Date: 2005-07-11 06:03 Message: Logged In: YES user_id=1188172 Done. Misc/NEWS r1.1193.2.61, Objects/obmalloc.c r2.53.4.1. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2005-07-10 23:07 Message: Logged In: YES user_id=80475 Reinhold, would you like to do the backport? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2005-07-10 22:33 Message: Logged In: YES user_id=31435 Repaired, in Misc/NEWS 1.1312 Objects/obmalloc.c 2.54 Should be backported to 2.4 maint. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-05-27 09:35 Message: Logged In: YES user_id=6656 Ping! ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2005-04-19 15:34 Message: Logged In: YES user_id=31435 krisvale: Thank you for the very clear explanation. Even I understand this now <wink>. We won't use _msize here -- Python has to run under dozens of compilers and C libraries, and it's saner to give up on this "optimization" completely than to introduce a rat's nest of #ifdefs here. IOW, I expect the entire "if (nbytes <= SMALL_REQUEST_THRESHOLD)" block will go away, so that the platform realloc() gets called in every case obmalloc doesn't control the incoming block. BTW, note that there's no plan to do another release in the Python 2.3 line. ---------------------------------------------------------------------- Comment By: Kristján Valur (krisvale) Date: 2005-04-19 15:22 Message: Logged In: YES user_id=1262199 The platform is windows 2000/2003 server, single threaded C runtime. I have only had the chance to do postmortem debugging on this but it would appear to be as you describe: The following page is not mapped in. Windows doesn´t use the setbrk() method of heap management and doesn´t automatically move the break. Rather they (the multiple heaps) requests pages as required. A malloc may have succeeded from a different page and copying to much from the old block close to the boundary caused an exception _at_ the page boundary. Fyi, old block was 68 bytes at 0x6d85efb8. This block ends at -effc. The new size requested was 108 bytes. Reading 108 bytes from this address caused an exception at address 0x6d85f000. As you know, reading past a malloc block results in undefined behaviour and sometimes this can mean a crash. I have patched python locally to use MIN(nbytes, _msize(p)) in stead and we are about to run the modified version on our server cluster. Nodes were dying quite regularly because of this. I'll let you know if this changes anyting in that aspect. Btw, I work for ccp games, and we are running the MMORPG eve online (www.eveonline.com) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2005-04-19 15:00 Message: Logged In: YES user_id=31435 mwh: Umm ... I don't understand what the claim is. For example, what HW does Python run on where memcpy segfaults just because the address range crosses a page boundary? If that's what's happening, sounds more like a bug in the platform memcpy. I can memcpy blocks spanning thousands of pages on my box -- and so can you <wink>. krisvale: which OS and which C are you using? It is true that this code may try to access a bit of memory that wasn't allocated. If that's at the end of the address space, then I could see a segfault happening. If it is, I doubt there's any portable way to fix it short of PyObject_Realloc never trying to take over small blocks it didn't control to begin with. Then the platform realloc() will segfault instead <wink>. ---------------------------------------------------------------------- Comment By: Kristján Valur (krisvale) Date: 2005-04-19 14:39 Message: Logged In: YES user_id=1262199 I can only say that I´ve been seeing this happeing with our software. Admittedly it's because we are eating up all memory due to other reasons, but we would like to deal with that with a MemoryError rather than a crash. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-04-19 14:30 Message: Logged In: YES user_id=6656 Tim, what do you think? This is a pretty unlikely scenario, it seems to me. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1185883&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
