[issue1524938] PEP MemoryError with a lot of available memory gc not called
Itai added the comment: Sure, that's what i'll do for now. Its an ok workaround for me, I was just posting to support the notion that its a bug (lets call it usability bug) and something that people out there do run into. There's also a scenerio where you couldn't use this workaround - for example use a library precompiled in a pyd.. On Wed, Aug 18, 2010 at 6:13 PM, Ray.Allen wrote: > > Ray.Allen added the comment: > > How about calling gc.collect() explicitly in the loop? > > -- > nosy: +ysj.ray > > ___ > Python tracker > <http://bugs.python.org/issue1524938> > ___ > -- Added file: http://bugs.python.org/file18568/unnamed ___ Python tracker <http://bugs.python.org/issue1524938> ___Sure, that's what i'll do for now. Its an ok workaround for me, I was just posting to support the notion that its a bug (lets call it usability bug) and something that people out there do run into. There's also a scenerio where you couldn't use this workaround - for example use a library precompiled in a pyd.. On Wed, Aug 18, 2010 at 6:13 PM, Ray.Allen <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> wrote: Ray.Allen <mailto:ysj@gmail.com";>ysj@gmail.com> added the comment: How about calling gc.collect() explicitly in the loop? -- nosy: +ysj.ray ___ Python tracker <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> <http://bugs.python.org/issue1524938"; target="_blank">http://bugs.python.org/issue1524938> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1524938] PEP MemoryError with a lot of available memory gc not called
Itai added the comment: You are right, ofcourse... I haven't got the time for doing the right thing, But I've found another workaround that helped me though and might be helpful to others. (not sure its for this thread though but...) Windows on default limits the amount of memory for 32 bit processes to 2GB. There's a bit in the PE image which tells 64 bit windows to give it 4GB (on 32 bit windows PAE needs to be enabled too) which is called IMAGE_FILE_LARGE_ADDRESS_AWARE. There's a post-build way to enable it with the editbin.exe utility which comes with visual studio like this: editbin.exe /LARGEADDRESSAWARE python.exe It works for me since it gives me x2 memory on my 64 bit os. I have to say it could be dangerous since it essentially says no where in python code pointers are treated as negative numbers. I figured this should be right since there's a 64 bit version of python... On Wed, Aug 18, 2010 at 9:27 PM, Martin v. Löwis wrote: > > Martin v. Löwis added the comment: > > Anybody *really* interested in this issue: somebody will need to write a > PEP, get it accepted, and provide an implementations. Open source is about > scratching your own itches: the ones affected by a problems are the ones > which are also expected to provide solutions. > > -- > > ___ > Python tracker > <http://bugs.python.org/issue1524938> > ___ > -- Added file: http://bugs.python.org/file18583/unnamed ___ Python tracker <http://bugs.python.org/issue1524938> ___You are right, ofcourse... I haven't got the time for doing the right thing,But I've found another workaround that helped me though and might be helpful to others.(not sure its for this thread though but...) Windows on default limits the amount of memory for 32 bit processes to 2GB. There's a bit in the PE image which tells 64 bit windowsto give it 4GB (on 32 bit windows PAE needs to be enabled too) which is called IMAGE_FILE_LARGE_ADDRESS_AWARE. There's a post-build way to enable it with the editbin.exe utility which comes with visual studio like this:editbin.exe /LARGEADDRESSAWARE python.exeIt works for me since it gives me x2 memory on my 64 bit os. I have to say it could be dangerous since it essentially says no where in python code pointers are treated as negative numbers. I figured this should be right since there's a 64 bit version of python...On Wed, Aug 18, 2010 at 9:27 PM, Martin v. Löwis <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> wrote: Martin v. Löwis <mailto:mar...@v.loewis.de";>mar...@v.loewis.de> added the comment: Anybody *really* interested in this issue: somebody will need to write a PEP, get it accepted, and provide an implementations. Open source is about scratching your own itches: the ones affected by a problems are the ones which are also expected to provide solutions. -- ___ Python tracker <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> <http://bugs.python.org/issue1524938"; target="_blank">http://bugs.python.org/issue1524938> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1524938] PEP MemoryError with a lot of available memory gc not called
Itai i added the comment: Hi all, I'm joining Mark's assertion - this is a real issue for me too. I've stumbled into this problem too. I have a numpy/scipy kind of application (about 6000+ lines so far) which needs to allocate alot of memory for statistics derived from "real life data" which is then transformed a few times by different algorithms (which means allocating more memory, but dumping the previous objects). Currently I'm getting MemoryError when I try to use the entire dataset, both on linux and on windows, python 2.5 on 64bit 4gb mem machine. (The windows python is a 32bit version though cause it needs to be compatible with some dlls. This is the same reason I use python 2.5) -- nosy: +Itai.i versions: -Python 2.7, Python 3.1 ___ Python tracker <http://bugs.python.org/issue1524938> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46785] On Windows, os.stat() can fail if called while another process is creating or deleting the file
Itai Steinherz added the comment: I'd like to work on this, however I'm not sure how this could be unit-tested. Any ideas? -- nosy: +itaisteinherz ___ Python tracker <https://bugs.python.org/issue46785> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46785] On Windows, os.stat() can fail if called while another process is creating or deleting the file
Itai Steinherz added the comment: Thanks for the comprehensive reply, Eryk! I have a few questions regarding your suggestion: 1. Why catch ERROR_NOT_READY and ERROR_BAD_NET_NAME as well? How do you know that FindFirstFileW() may return them? 2. Why can't the filename of the "foo"-like file in the test be simply os_helper.TESTFN, as done in some other tests? 3. I noticed some code snippets used in tests are wrapped in a seemingly-redundant `if 1: ...` statement. Why is that? -- ___ Python tracker <https://bugs.python.org/issue46785> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46785] On Windows, os.stat() can fail if called while another process is creating or deleting the file
Change by Itai Steinherz : -- keywords: +patch pull_requests: +29956 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/31858 ___ Python tracker <https://bugs.python.org/issue46785> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46785] On Windows, os.stat() can fail if called while another process is creating or deleting the file
Itai Steinherz added the comment: Interesting, thanks again :) -- ___ Python tracker <https://bugs.python.org/issue46785> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21073] Py_ReprEnter potentially misbehaves during malformed thread states
New submission from Itai Bar-Natan: While browsing the Python source code, I found this suspicious snippet in Py_ReprEnter: dict = PyThreadState_GetDict(); if (dict == NULL) return 0; It seems to me like the last line should be "return -1;". The way the program currently behaves, if PyThreadState_GetDict() fails and returns NULL, Py_ReprEnter will fail silently and always report that the input isn't in a recursive loop. The correct behavior is to report an error. It would be difficult to explicitly exhibit this error since it relies on another component of Python failing first. One possible way would be to call PyObject_Repr on a recursive structure before fully initializing Python. I haven't tested this. Alternately, it's possible that this behavior is intentional because we want PyObject_Repr to work for non-self-referential structures even before Python is fully initialized (perhaps it could be called during initialization), in exchange for a small risk of failure if it is called with a self-referential structure before initialization. In that case I suggest that this should be pointed out explicitly in the comments to this function. -- components: Interpreter Core messages: 214920 nosy: itaibn priority: normal severity: normal status: open title: Py_ReprEnter potentially misbehaves during malformed thread states type: behavior versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker <http://bugs.python.org/issue21073> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25631] Segmentation fault with invalid Unicode command-line arguments in embedded Python
New submission from Itai Bar-Natan:
The following embedded application, which calls Py_Main with a "-W X" argument
where X is not a valid Unicode string, returns a segmentation fault:
```
#include "Python.h"
main() {
wchar_t *invalid_str;
invalid_str = malloc(2*sizeof(wchar_t));
invalid_str[0] = 0x11;
invalid_str[1] = 0;
wchar_t *argv[4] = {L"embedded-python", L"-W", invalid_str, NULL};
Py_Main(3, argv);
}
```
This segmentation fault is present in Python 3.4, 3.5, and the latest
development branch I downloaded, but is not present in Python 3.2. This program
is obviously invalid and it may be reasonable to emit a fatal error in this
situation, but it should not give a segmentation fault.
I believe the issue is that this codes leads to exception being thrown before
exceptions are initialized, and more specifically, a call to
PyExceptionClass_Check() within PyErr_Object() reads a NULL pointer. I haven't
tested this but I expect that this problem would not appear when calling Python
directly since Python sanitizes the command line arguments from main().
Nonetheless even here the possibility of other exceptions being raised early in
the initialization sequence remains a potential problem.
--
components: Interpreter Core
messages: 254703
nosy: itaibn
priority: normal
severity: normal
status: open
title: Segmentation fault with invalid Unicode command-line arguments in
embedded Python
type: crash
versions: Python 3.4, Python 3.5, Python 3.6
___
Python tracker
<http://bugs.python.org/issue25631>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
