Re: [Python-Dev] bytearray and array.array are not thread-safe
Le dimanche 06 juillet 2008 à 00:28 +0200, "Martin v. Löwis" a écrit : > I propose that new codes s*, t*, w* are added, and that s#,t#,w# refuses > objects which implement a releasebuffer procedure (alternatively, s# etc > might be removed altogether right away). Users of s* then need to pass > in a Py_Buffer view pointer that gets filled, and need to explicitly > release the buffer. For convenience, it might help if the Py_buffer > structure includes a borrowed PyObject* to the underlying object, along > with a PyBuffer_Release procedure/macro. Why a borrowed reference rather than a new one? It could be decref'ed as part as the proposed PyBuffer_Release procedure. Overall it sounds like a clean resolution of the problem. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On 01:02 am, [EMAIL PROTECTED] wrote: To bring my $0.02 to this discussion: the Pybots 'community buildbots' turned out largely to be a failure. Let's not say it's a failure. Let's instead say that it hasn't yet become a success :-). I still haven't given up, and I hope this thread will spur project leaders into donating time, or resources, to the Pybots project. It has been my bitter observation about the Open Source world that people just LOVE to get stuff for free. As soon as you mention more involvement from them in the form of time, money, hardware resources, etc., the same brave proponents of cool things to be done are nowhere to be found. I think this list is the wrong place to go to reach the people who need to be reached. It's python core developers and other people already involved in and aware of core development. That said I'm not sure what the *right* place is; I think your blog is syndicated on the unofficial planet python, so maybe that's a good place to start. Sadly, the right thing to do in terms of drumming up support is to get someone interested in PR and have them go to each project individually, but that might be more effort than setting up the buildbots themselves, at least initially... However, let's say that this were tremendously successful, and lots of people start paying attention. I think pybots.org needs to be updated to say exactly what a participant interested in python testing needs to do, beyond "here's how you set up a buildbot" (a page that is actually a daunting-looking blog post which admits it may be somewhat outdated), because setting up a buildbot might not be the only thing that the project needs. It's one thing to tell people that they need to be helping out (and I'm sure you're right) but it's much more useful to get the message out that "we really need people to do X, Y, and Z". One thing I will definitely commit to is that if you make a "cry for help" page, I'll blog about it to drive attention to it, and I'll encourage the other, perhaps better-read Python bloggers I know to do so as well. My personal interest at the moment is to get all of the irrelevant red off of the community builders page. Whether or not you believe in an XP "green bar" philosophy, the large number of spurious failures is distracting. Who is it that is capable of making appropriate changes? Is there something I could do to help with that? Note that I'm committing to say that I can do *that*, but, at least you could shut me up by making it my fault ;-). (I'd also like to improve the labels of the build slaves. What exactly is "x86 Red Hat 9 trunk" testing? Trunk of what? What project?) It would be good to remove the perception that it's somebody else's problem as much as possible. Right now, all these dead buildbots suggest to the various communities, "oh, I guess that guy who runs that buildbot needs to fix it". The dead bots should just be killed off, and their projects removed from the list, so that if someone wants to get involved and set up a bot for lxml, they're not put off of it by the fact that it might be rude to the guy who is currently (allegedly) running it. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytearray and array.array are not thread-safe
> Why a borrowed reference rather than a new one? It could be decref'ed as > part as the proposed PyBuffer_Release procedure. The question is a) whether a Py_Buffer remains valid even if the object goes away. That seems not to be the case, i.e. the caller of getbuffer needs to hold onto the object, anyway. b) whether it would still be correct to call releasebuffer explicitly. Of course, as getbuffer would have to fill the object into the view, releasebuffer could also DECREF the included object. Alternatively, there could be a pair of functions PyBuffer_Get and PyBuffer_Release, which would fill the object into the view itself. So I withdraw issue b; the real question remains whether it is desired that a buffer will remain alive as long as there is a view to it. That is a question for the buffer experts to answer; it may also have impacts on cyclic garbage collection (as inclusion of a view into an object will mean that the tp_traverse function must also Py_VISIT the embedded object). > Overall it sounds like a clean resolution of the problem. Unfortunately, it's also a significant change at this point. I personally won't have time to provide a patch, but I think a patch is needed before the last beta. IOW, the issue should become a release blocker. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
[EMAIL PROTECTED] writes: > On 01:02 am, [EMAIL PROTECTED] wrote: > >To bring my $0.02 to this discussion: the Pybots 'community buildbots' > >turned out largely to be a failure. > > Let's not say it's a failure. Let's instead say that it hasn't yet > become a success :-). +1 > >I still haven't given up, and I hope this thread will spur project > >leaders into donating time, or resources, to the Pybots project. > I think this list is the wrong place to go to reach the people who need > to be reached. It's python core developers and other people already > involved in and aware of core development. That said I'm not sure what > the *right* place is; I think your blog is syndicated on the unofficial > planet python, so maybe that's a good place to start. Sadly, the right > thing to do in terms of drumming up support is to get someone interested > in PR and have them go to each project individually, but that might be > more effort than setting up the buildbots themselves, at least > initially... Exactly, and that's why nobody should be "bitter" about it. The problem is that the while overall the effort and rewards look to be balanced in favor of the rewards, the startup costs for individuals are quite high. I think this *is* the place to start, though. The project leaders "should" be, and probably generally are, "here". They have the strongest interest in any individual 'bot, while Guido is quite correct in saying python-dev can't afford to have strong interest in all the bots. > However, let's say that this were tremendously successful, and lots of > people start paying attention. I think pybots.org needs to be updated > to say exactly what a participant interested in python testing needs to > do, beyond "here's how you set up a buildbot" (a page that is actually a > daunting-looking blog post which admits it may be somewhat outdated), > because setting up a buildbot might not be the only thing that the > project needs. It's one thing to tell people that they need to be > helping out (and I'm sure you're right) but it's much more useful to get > the message out that "we really need people to do X, Y, and Z". One > thing I will definitely commit to is that if you make a "cry for help" > page, I'll blog about it to drive attention to it, and I'll encourage > the other, perhaps better-read Python bloggers I know to do so as > well. Two suggestions in this vein: First, I think it's established that some but not all "red community bots" *are* of interest to Python core development. While I'm not aware of the technical details, I estimate that triaging the community 'bot failures is probably similar to reviewing bugs in the Python tracker. Perhaps Martin van Loewis and others who have offered the 5-for-1 review deal would be willing to extend the definition of "review" to include initial bug reports based on a red community bot (ie, you review the community 'bot failure and decide it is something that should concern Python core development). Perhaps that's not appropriate, but a similar system could be set up. Second, something intermediate between the occasional half-hour of triaging bugs and a full-blown PR campaign at the projects would be documenting the criteria for reporting a failure on a community 'bot to the Python tracker as a bug, etc. This would also serve as a basis for talking to project lurker who might have the odd half-hour to do some "red bot" triaging. (By criteria I mean the kinds of things that Python core considers necessary breakage in new versions that downstream must address in their own code, vs. regressions in a x.y.z patchlevel, etc. The kind of thing that glyph and Guido were discussing earlier.) > It would be good to remove the perception that it's somebody else's > problem as much as possible. To the extent that a 'bot is running prerelease project against prerelease Python, this is probably not very doable. If Python is stable and the project version is prerelease, it's the project's bug until proven otherwise, and vice versa. If both are stable, again some expertise is probably needed for triage. I guess that means that one important task is to classfy the bots in a two-by-two matrix according to stability of project and Python respectively. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> (I'd also like to improve the labels of the build slaves. What exactly > is "x86 Red Hat 9 trunk" testing? Trunk of what? What project?) It seems like you would like to edit the master configuration file. That can be arranged fairly easily. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packing and unpacking integers
> I believe there is still a need for packing integers as strings and > unpacking strings as integers, more specifically, offering to Python > an interface to _PyLong_FromByteArray() and _PyLong_AsByteArray(). I > would be happy to write the functionality and unittests this coming > week for 2.6 and 3.0 if I get the ok. If not, I can write it for 2.7 > and 3.1 . I think it needs to be deferred to the next releases, given that the beta release already happened. If you have any spare time, please look into some of the real serious, release-blocking bug reports. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
On Sun, Jul 6, 2008 at 8:46 AM, <[EMAIL PROTECTED]> wrote: > > However, let's say that this were tremendously successful, and lots of > people start paying attention. I think pybots.org needs to be updated to > say exactly what a participant interested in python testing needs to do, > beyond "here's how you set up a buildbot" (a page that is actually a > daunting-looking blog post which admits it may be somewhat outdated), > because setting up a buildbot might not be the only thing that the project > needs. It's one thing to tell people that they need to be helping out (and > I'm sure you're right) but it's much more useful to get the message out that > "we really need people to do X, Y, and Z". One thing I will definitely > commit to is that if you make a "cry for help" page, I'll blog about it to > drive attention to it, and I'll encourage the other, perhaps better-read > Python bloggers I know to do so as well. I have posted 'cries for help' repeatedly on my blog, with generally little success. See http://agiletesting.blogspot.com/search?q=pybots . But I will post again. If you can include here a paragraph of what your vision of the 'X, Y and Z' above is, that'd help too. I think I've been pretty clear about the benefits that the Pybots farm can bring to a given project, so all project leaders on this list should be aware of them IMO. If not, I'd be happy to rehash them. But the home page of pybots.org is pretty self-explanatory I think. > > My personal interest at the moment is to get all of the irrelevant red off > of the community builders page. Whether or not you believe in an XP "green > bar" philosophy, the large number of spurious failures is distracting. Who > is it that is capable of making appropriate changes? Is there something I > could do to help with that? Note that I'm committing to say that I can do > *that*, but, at least you could shut me up by making it my fault ;-). > I'll send a message to the pybots mailing list asking people whose buildbots are turned off if they're still interested in running them. Negative or no answers will mean we can remove them from the farm. > (I'd also like to improve the labels of the build slaves. What exactly is > "x86 Red Hat 9 trunk" testing? Trunk of what? What project?) > It's not only a question of changing a static label in this case. A given buildslave can run the tests for multiple projects, in which case it becomes tricky to change the label on the fly accordingly. As an aside, the slave you mention was running on my machine, and I used it to run the Twisted tests, but I shut it down a while ago because the buildbot process was taking too many resources. If the Twisted project can donate a machine, I'd be happy to include it in the Pybots farm ASAP. > It would be good to remove the perception that it's somebody else's problem > as much as possible. Right now, all these dead buildbots suggest to the > various communities, "oh, I guess that guy who runs that buildbot needs to > fix it". The dead bots should just be killed off, and their projects > removed from the list, so that if someone wants to get involved and set up a > bot for lxml, they're not put off of it by the fact that it might be rude to > the guy who is currently (allegedly) running it. As I said, I'll see what the current owners have to say, and then I'll report back to this list. Thanks for offering your help! Grig ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Play with fuzzing
Hi, I wrote a fuzzing "framework" called Fusil and this week I wrote a fuzzer for Python. The idea is quite simple: for a module, - list all functions, classes and class methods - call a function with random arguments (of random types) - instanciate a class with random arguments - if the class is created correctly, call methods with random arguments Example: - 8< --- print "Call 39/40: linuxaudiodev.open()" try: linuxaudiodev.open( # argument 1/2 u"\u62C0\uFBD7\uB46A\u55E0\uFB7B\uD392\u7CEE", # argument 2/2 52.682, ) except Exception, err: print >>stderr, "ERROR: %s" % err - 8< --- I tried it on CPython 2.5 and then on CPython trunk (future 2.6). I found some bugs, see last bug entries in Python bugtracker. Just an example: http://bugs.python.org/issue3304 -> invalid call to PyMem_Free() in fileio_init() Most bugs crash with a segmentation fault, abort or a denial of service. If you would like to try my fuzzer, use: (1) svn co http://fusil.hachoir.org/svn/trunk fusil (2) cd fusil (3) ./run_fusil.sh -p projects/python.py --fast --remove ALL The option --fast goes faster, --remove does remove session directory even if Python generated some files, and "ALL" test all modules. FUSIL IS NOT SAFE! So run it under a different user using to avoid dangerous call to os.unlink(). The module list is hardcoded: it's the list of CPython modules written in C. More informations about Fusil: http://fusil.hachoir.org/trac -- Victor Stinner aka haypo http://www.haypocalc.com/blog/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Play with fuzzing
On Sun, Jul 6, 2008 at 4:11 PM, Victor Stinner <[EMAIL PROTECTED]> wrote: > Hi, > > I wrote a fuzzing "framework" called Fusil and this week I wrote a fuzzer for > Python. The idea is quite simple: for a module, > - list all functions, classes and class methods > - call a function with random arguments (of random types) > - instanciate a class with random arguments > - if the class is created correctly, call methods with random arguments > > Example: > - 8< --- > print "Call 39/40: linuxaudiodev.open()" > try: >linuxaudiodev.open( ># argument 1/2 >u"\u62C0\uFBD7\uB46A\u55E0\uFB7B\uD392\u7CEE", ># argument 2/2 >52.682, >) > except Exception, err: >print >>stderr, "ERROR: %s" % err > - 8< --- > > I tried it on CPython 2.5 and then on CPython trunk (future 2.6). I found some > bugs, see last bug entries in Python bugtracker. Just an example: > > http://bugs.python.org/issue3304 > -> invalid call to PyMem_Free() in fileio_init() > You can use http://bugs.python.org/issue?%40search_text=&title=&%40columns=title&id=&%40columns=id&creation=&creator=haypo&activity=2008-07-06&%40columns=activity&%40sort=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&%40group=priority&status=1&%40columns=status&resolution=&%40pagesize=50&%40startwith=0&%40queryname=&%40old-queryname=&%40action=search to see all of the bugs Victor has filed today (looks like eight). -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots and Python release quality metrics
> It's not only a question of changing a static label in this case. A > given buildslave can run the tests for multiple projects, in which > case it becomes tricky to change the label on the fly accordingly. I think you could set up different builders for a single slave in that case (use a slave lock to make them run sequentially). > As > an aside, the slave you mention was running on my machine, and I used > it to run the Twisted tests, but I shut it down a while ago because > the buildbot process was taking too many resources. If the Twisted > project can donate a machine, I'd be happy to include it in the Pybots > farm ASAP. Please talk to Trent Nelson. He has a Windows machine that he donated precisely for that kind of activity. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Play with fuzzing
> I wrote a fuzzing "framework" called Fusil and this week I wrote a fuzzer for > Python. The idea is quite simple: for a module, > - list all functions, classes and class methods > - call a function with random arguments (of random types) > - instanciate a class with random arguments > - if the class is created correctly, call methods with random arguments I was already wondering how you found out all these things. It's quite amazing! Thanks, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
