Re: [Python-Dev] status of development documentation
Neal Norwitz wrote: > On 12/23/05, Tim Peters <[EMAIL PROTECTED]> wrote: > >>>http://docs.python.org/dev/results/ >> >>Wow! You get no test failures! I guess nobody tests on Windows >>anymore. I've been getting test failures for months, and just > > > Hmmm, I thought others were running the tests on Windows too. There > was one report on Nov 22 about running Purify on Windows 2k (subject: > ast status, memory leaks, etc). He had problems with a stack overflow > in test_compile. He was going to disable the test and re-run. I > never heard back though. Based on that info, I would guess that > test_builtin was working on Win 2k on Nov 22. > > >>_assumed_ this was known damage everywhere so was waiting for someone >>else to fix it ;-) (A parenthentical question: is there a reason you >>don't pass -uall to regrtest.py?) > [...] I suppose this might be evidence in support of the arguments for trying to make Python compile on Windows under an open-source toolset. The reason *I* don't test under Windows is because I can't build under Windows, so I only run Python installed from packaged installers. Alternatively, is there any mileage in trying to either get Sourceforge to provide Windows machines in the compile farm, or get Microsoft to provide more software fee to Windows testers? regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC www.holdenweb.com PyCon TX 2006 www.python.org/pycon/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
Steve Holden wrote: [...] > Alternatively, is there any mileage in trying to either get Sourceforge > to provide Windows machines in the compile farm, or get Microsoft to > provide more software fee to Windows testers? ^fee^free^ -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC www.holdenweb.com PyCon TX 2006 www.python.org/pycon/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
Neal> I guess you might have to binary search by date to try and find Neal> the problem. Probably needs to be a binary search by revision. I believe Martin indicated a side effect of the conversion to subversion was that date-based updates don't work. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
Steve> Alternatively, is there any mileage in trying to either get Steve> Sourceforge to provide Windows machines in the compile farm, or Steve> get Microsoft to provide more software fee to Windows testers? How about seeing if Microsoft has or will create a compile farm? Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
[EMAIL PROTECTED] wrote: > Steve> Alternatively, is there any mileage in trying to either get > Steve> Sourceforge to provide Windows machines in the compile farm, or > Steve> get Microsoft to provide more software fee to Windows testers? > > How about seeing if Microsoft has or will create a compile farm? > I'll see what I can do ... regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC www.holdenweb.com PyCon TX 2006 www.python.org/pycon/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
[Neal] > Hmmm, I thought others were running the tests on Windows too. There > was one report on Nov 22 about running Purify on Windows 2k (subject: > ast status, memory leaks, etc). He had problems with a stack overflow > in test_compile. He was going to disable the test and re-run. I > never heard back though. Based on that info, I would guess that > test_builtin was working on Win 2k on Nov 22. I wouldn't assume that. My "nobody" was wrt the universe of Python developers, not users; folks like Trent and MarkH and you and me. Without "normal" baseline test experience, users don't understand what they're seeing, and so their reports can be highly misleading. You can trust that while I don't understand what I'm seeing here either, at least I told the absolute truth about it and didn't hold back critical details ;-) That said, I was hoping to do some Python work over Thanksgiving week, but was mortally discouraged on the Nov 19-20 weekend by all the test failures I saw. So I'm pretty sure (but not certain) that test_builtin was failing then. >> (A parenthentical question: is there a reason you don't pass -uall to >> regrtest.py?) > It's calling make test. I thought about calling regrtest.py instead > and doing as you suggest. Is there a benefit to running make test? You're asking a Windows guy about make: bad career move ;-) > I know it runs with and without -O. I guess it's only machine time, I > could run make test and regrtest.py -uall. -uall is helpful in finding bugs. One thing in particular here is that test_compiler runs only a tiny subset of its full test unless an appropriate -u flag is given. >> On WinXP Pro SP2 today, passing -uall, and after fixing all the MS >> compiler warnings that have crept in: >> >> 251 tests OK. >> 12 tests failed: >> test_builtin test_coding test_compiler test_pep263 >> test_univnewlines test_urllib test_urllib2 test_urllibnet >> test_userlist test_wave test_whichdb test_zipfile >> 1 skip unexpected on win32: >> test_xml_etree_c > Ouch! I'm might be to blame for at least some of those. :-( I'm sure it's not as bad as it looks. For example, test_coding and (the -uall) test_compiler fail for exactly the same reason. For another, when a test fails on Windows, it sometimes leaves a "@test" file or directory behind, which causes a cascade of bogus failures later. For example, test_userlist, test_wave, test_whichdb, and test_zipfile all pass when run alone here. Others probably do too. ... > > Do you know if the tests were broken before the AST merge (about Oct > 22 I think)? I don't know. I'm getting more evidence that most (if not all) of the failures are due to compile-time parsing screwups, so the AST merge is a prime suspect. Is it possible that generated parse tables (whatever) are out-of-date on a Windows box? There are no makefiles here, and the Windows build has always relied on Unix-heads to check in correct generated parser files. >> The code up to the first failure is short: >> >> bom = '\xef\xbb\xbf' >> compile(bom + 'print 1\n', '', 'exec') >> >> Curiously, that sequence doesn't blow up under the released Windows >> Python 2.4.2, so somebody broke something here since then ... > There were a bunch of changes to Parser/tokenizer.c to handle error > conditions. Those go back to Oct 1. I don't *think* those would > cause these, but I'm not sure. > > Sorry, I don't know any more. I guess you might have to binary search > by date to try and find the problem. That's just silly ;-) What I need is someone who understands what this code is _supposed_ to do, so we can fix it; finding the checkin that caused it isn't nearly as interesting. Windows has an excellent debug-build debugger and I can easily step through the code. But I have no idea why compiling a string starting with '\xef\xbb\xbf' should _not_ be a syntax error -- it looks like a syntax error to me. And when I step thru the code, it looks like a syntax error to the parser too. It peels off the first character (\xef), and says "syntax error" at that point: Py_CompileStringFlags -> PyParser_ASTFromString -> PyParser_ParseStringFlagsFilename -> parsetok -> PyTokenizer_Get That sets `a` to point at the start of the string, `b` to point at the second character, and returns type==51. Then `len` is set to 1, `str` is malloc'ed to hold 2 bytes, and `str` is filled in with "\xef\x00" (the first byte of the input, as a NUL-terminated C string). PyParser_AddToken then calls classify(), which falls off the end of its last loop and returns -1: syntax error. So how it gets there is really quite straightfoward. The problem for me is that I have no idea why it _should_ do something other than that, let alone what that may be. This needs someone who knows something :-) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.pyth
Re: [Python-Dev] status of development documentation
Tim Peters wrote: > [Neal] > >>Hmmm, I thought others were running the tests on Windows too. There >>was one report on Nov 22 about running Purify on Windows 2k (subject: >>ast status, memory leaks, etc). He had problems with a stack overflow >>in test_compile. He was going to disable the test and re-run. I >>never heard back though. Based on that info, I would guess that >>test_builtin was working on Win 2k on Nov 22. > > > I wouldn't assume that. My "nobody" was wrt the universe of Python > developers, not users; folks like Trent and MarkH and you and me. > Without "normal" baseline test experience, users don't understand what > they're seeing, and so their reports can be highly misleading. You > can trust that while I don't understand what I'm seeing here either, > at least I told the absolute truth about it and didn't hold back > critical details ;-) > > That said, I was hoping to do some Python work over Thanksgiving week, > but was mortally discouraged on the Nov 19-20 weekend by all the test > failures I saw. So I'm pretty sure (but not certain) that > test_builtin was failing then. > > >>>(A parenthentical question: is there a reason you don't pass -uall to >>>regrtest.py?) > > >>It's calling make test. I thought about calling regrtest.py instead >>and doing as you suggest. Is there a benefit to running make test? > > > You're asking a Windows guy about make: bad career move ;-) > > >>I know it runs with and without -O. I guess it's only machine time, I >>could run make test and regrtest.py -uall. > > > -uall is helpful in finding bugs. One thing in particular here is > that test_compiler runs only a tiny subset of its full test unless an > appropriate -u flag is given. > > >>>On WinXP Pro SP2 today, passing -uall, and after fixing all the MS >>>compiler warnings that have crept in: >>> >>>251 tests OK. >>>12 tests failed: >>>test_builtin test_coding test_compiler test_pep263 >>>test_univnewlines test_urllib test_urllib2 test_urllibnet >>>test_userlist test_wave test_whichdb test_zipfile >>>1 skip unexpected on win32: >>>test_xml_etree_c > > >>Ouch! I'm might be to blame for at least some of those. :-( > > > I'm sure it's not as bad as it looks. For example, test_coding and > (the -uall) test_compiler fail for exactly the same reason. For > another, when a test fails on Windows, it sometimes leaves a "@test" > file or directory behind, which causes a cascade of bogus failures > later. For example, test_userlist, test_wave, test_whichdb, and > test_zipfile all pass when run alone here. Others probably do too. > > ... > >>Do you know if the tests were broken before the AST merge (about Oct >>22 I think)? > > > I don't know. I'm getting more evidence that most (if not all) of the > failures are due to compile-time parsing screwups, so the AST merge is > a prime suspect. > > Is it possible that generated parse tables (whatever) are out-of-date > on a Windows box? There are no makefiles here, and the Windows build > has always relied on Unix-heads to check in correct generated parser > files. > > >>>The code up to the first failure is short: >>> >>>bom = '\xef\xbb\xbf' >>>compile(bom + 'print 1\n', '', 'exec') >>> >>>Curiously, that sequence doesn't blow up under the released Windows >>>Python 2.4.2, so somebody broke something here since then ... > > >>There were a bunch of changes to Parser/tokenizer.c to handle error >>conditions. Those go back to Oct 1. I don't *think* those would >>cause these, but I'm not sure. >> >>Sorry, I don't know any more. I guess you might have to binary search >>by date to try and find the problem. > > > That's just silly ;-) What I need is someone who understands what > this code is _supposed_ to do, so we can fix it; finding the checkin > that caused it isn't nearly as interesting. Windows has an excellent > debug-build debugger and I can easily step through the code. But I > have no idea why compiling a string starting with '\xef\xbb\xbf' > should _not_ be a syntax error -- it looks like a syntax error to me. > > And when I step thru the code, it looks like a syntax error to the > parser too. It peels off the first character (\xef), and says "syntax > error" at that point: > > Py_CompileStringFlags -> > PyParser_ASTFromString -> > PyParser_ParseStringFlagsFilename -> > parsetok -> > PyTokenizer_Get > > That sets `a` to point at the start of the string, `b` to point at the > second character, and returns type==51. Then `len` is set to 1, > `str` is malloc'ed to hold 2 bytes, and `str` is filled in with > "\xef\x00" (the first byte of the input, as a NUL-terminated C > string). > > PyParser_AddToken then calls classify(), which falls off the end of > its last loop and returns -1: syntax error. > > So how it gets there is really quite straightfoward. The problem for > me is that I have no idea why it _should_ do something other than > that, let alone what that may be. PEP
Re: [Python-Dev] documentation comments
2005/12/22, Fred L. Drake, Jr. <[EMAIL PROTECTED]>: > In general, my worry is less with dealing with spam than with ensuring > integration of content enhancements before release candidates go out. Well, I think that the most important part of annotable documentation is just cuantitative feedback, not cualitative. I mean, I don't care if a doc page has a comment and it doesn't get included in the next release (if you want it included, submit it in a more formal way). But, if there's a page with 25 notes, that implies that there's a problem with that doc page, and something (integrating some of the notes or not) must be done with that text before the next release. .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
[Samuele Pedroni] > PEP263: > > """ > To aid with platforms such as Windows, which add Unicode BOM marks > to the beginning of Unicode files, the UTF-8 signature > '\xef\xbb\xbf' will be interpreted as 'utf-8' encoding as well > (even if no magic encoding comment is given). > """ So test_pep263 is almost certainly failing for a related reason (and I'll skip noting the irony that this was for the benefit of Windows ;-)). I didn't step through any code under the debugger that looked like it was _trying_ to special-case '\xef\xbb\xbf', so this still needs someone who knows which path(s) the code is supposed to take. I'll help, but I don't have time to drive this. test_pep263 test test_pep263 failed -- Traceback (most recent call last): File "C:\Code\python\lib\test\test_pep263.py", line 12, in test_pep263 '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd' AssertionError: '\xc3\xb0\xc3\x89\xc3\x94\xc3\x8f\xc3\x8e' != '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd' ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
On 12/24/05, Tim Peters <[EMAIL PROTECTED]> wrote: > You're asking a Windows guy about make: bad career move ;-) :-) > -uall is helpful in finding bugs. One thing in particular here is > that test_compiler runs only a tiny subset of its full test unless an > appropriate -u flag is given. Not to mention networking tests. > >> The code up to the first failure is short: > >> > >> bom = '\xef\xbb\xbf' > >> compile(bom + 'print 1\n', '', 'exec') > That sets `a` to point at the start of the string, `b` to point at the > second character, and returns type==51. Then `len` is set to 1, > `str` is malloc'ed to hold 2 bytes, and `str` is filled in with > "\xef\x00" (the first byte of the input, as a NUL-terminated C > string). This gives me an idea (ie, wild ass guess). r39680 checked in on 2005-10-06 to speed up unicode charmap decoding. Dunno if it's likely or not. Gotta run, I'm headed east. Good luck. Merry Christmas and Happy New Year everyone! n ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
Neal Norwitz wrote: > On 12/24/05, Tim Peters <[EMAIL PROTECTED]> wrote: > > [...] >> >> The code up to the first failure is short: >> >> >> >> bom = '\xef\xbb\xbf' >> >> compile(bom + 'print 1\n', '', 'exec') > >> That sets `a` to point at the start of the string, `b` to point at the >> second character, and returns type==51. Then `len` >> is set to 1, `str` is malloc'ed to hold 2 bytes, and `str` is filled in with >> "\xef\x00" (the first byte of the input, as a NUL-terminated C >> string). > > This gives me an idea (ie, wild ass guess). r39680 checked in on > 2005-10-06 to speed up unicode charmap decoding. Dunno if it's likely or > not. Gotta run, I'm headed east. Good luck. Other candidates might be the patches to Parser/tokenizer.c. (Unfortunately I don't have a machine to test this right now). Bye, Walter Dörwald ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] NotImplemented reaching top-level
Folks,
There's a bug about number coercion about Decimal
(http://www.python.org/sf/1355842).
The bug appeared after some changes Raymond and I did a few months
ago, solving something else (started to return NotImplemented instead
of raising TypeError, to better work with custom objects that
implements type coercion from Decimal).
The point is that I'm really astonished about the following behaviour,
and don't know where to start searching:
>>> d
Decimal("1") # using decimal.py rev. 39328 from svn
>>> d + 1.2
Traceback (most recent call last):
File "", line 1, in ?
TypeError: unsupported operand type(s) for +:
'Decimal' and
'float'
>>> d += 1.2
>>> d
NotImplemented
>>>
Thanks for any tip.
.Facundo
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
[Neal Norwitz]
> This gives me an idea (ie, wild ass guess). r39680 checked in on
> 2005-10-06 to speed up unicode charmap decoding. Dunno if it's likely
> or not. Gotta run, I'm headed east. Good luck.
Nope, it's not calling any decoding functions at all on Windows, let
alone optimized ones ;-)
Who knows what the code is supposed to do? (Sorry, but I don't want
to chase random patch guesses -- it's a wasteful approach to this
one.)
I'm getting a strong suspicion that I'm the only developer to _try_
building the trunk on WinXP since the AST merge was done, and that
something obscure is fundamentally broken with it on this box. For
example, in tokenizer.c, these functions don't even exist on Windows
today (because an enclosing #ifdef says not to compile them):
error_ret
new_string
get_normal_name
get_coding_spec
check_coding_spec
check_bom
fp_readl
fp_setreadl
fp_getc
fp_ungetc
decoding_fgets
decoding_feof
buf_getc
buf_ungetc
buf_setreadl
translate_into_utf8
decode_str
OK, that's not quite true. "Degenerate" forms of three of those
functions exist on Windows:
static char *
decoding_fgets(char *s, int size, struct tok_state *tok)
{
return fgets(s, size, tok->fp);
}
static int
decoding_feof(struct tok_state *tok)
{
return feof(tok->fp);
}
static const char *
decode_str(const char *str, struct tok_state *tok)
{
return str;
}
In the simple failing test, that degenerate decode_str() is getting
called. If the "fancy" decode_str() were being used instead, that one
_does_ call check_bom(). Why do we have two versions of these
functions? Which set is supposed to be in use now? What's the
meaning of "#ifdef PGEN" today? Should it be true or false?
> Merry Christmas and Happy New Year everyone!
Same to you, Neal!
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
FWIW, test_builtin and test_pep263 both passed on WinXP in rev 39757. That's the last revision before the AST branch was merged. I can't build rev 39758 on WinXP (VC complains that pythoncore.vcproj can't be loaded -- looks like it got checked in with unresolved SVN conflict markers -- which isn't easy to do under SVN ;-( ), so don't know about that. The first revision at which Python built again was 39791 (23 Oct), and test_builtin and test_pep263 both fail under that the same way they fail today. I'm darned near certain that we're not using the _intended_ parsing code on Windows now -- PGEN is still #define'd when the "final" parsing code is compiled into python25.dll. Don't know how to fix that (I don't understand it). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
On 12/24/05, Tim Peters <[EMAIL PROTECTED]> wrote: > FWIW, test_builtin and test_pep263 both passed on WinXP in rev 39757. > That's the last revision before the AST branch was merged. > > I can't build rev 39758 on WinXP (VC complains that pythoncore.vcproj > can't be loaded -- looks like it got checked in with unresolved SVN > conflict markers -- which isn't easy to do under SVN ;-( ), so don't > know about that. > > The first revision at which Python built again was 39791 (23 Oct), and > test_builtin and test_pep263 both fail under that the same way they > fail today. > Both syntax errors, right? My mind is partially gone thanks to being on vacation so following this thread has been abnormally hard. =) Since it is a syntax error there won't be any bytecode to compare against. > I'm darned near certain that we're not using the _intended_ parsing > code on Windows now -- PGEN is still #define'd when the "final" > parsing code is compiled into python25.dll. Don't know how to fix > that (I don't understand it). But the AST branch didn't touch the parser (unless you are grouping ast.c and compile.c under the "parser" umbrella just to throw me off =). What can I do to help? Do you need me to step through something? Do you need to know how gcc is preprocessing some file? -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] status of development documentation
[Tim]
>> FWIW, test_builtin and test_pep263 both passed on WinXP in rev 39757.
>> That's the last revision before the AST branch was merged.
>>
>> I can't build rev 39758 on WinXP (VC complains that pythoncore.vcproj
>> can't be loaded -- looks like it got checked in with unresolved SVN
>> conflict markers -- which isn't easy to do under SVN ;-( ), so don't
>> know about that.
>>
>> The first revision at which Python built again was 39791 (23 Oct), and
>> test_builtin and test_pep263 both fail under that the same way they
>> fail today.
[Brett]
> Both syntax errors, right?
In test_builtin, yes, two syntax errors. test_pep263 is different:
test test_pep263 failed -- Traceback (most recent call last):
File "C:\Code\python\lib\test\test_pep263.py", line 12, in test_pep263
'\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd'
AssertionError:
'\xc3\xb0\xc3\x89\xc3\x94\xc3\x8f\xc3\x8e' !=
'\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd'
That's not a syntax error, it's a wrong result. There are other
parsing-related test failures, but those are the only two I've written
up so far (partly because I expect they all have the same underlying
cause, and partly because nobody so far seems to understand the code
well enough to explain why the first one works on any platform ;-)).
> My mind is partially gone thanks to being on vacation so following this thread
> has been abnormally hard. =)
>
> Since it is a syntax error there won't be any bytecode to compare against.
Shouldn't be needed. The snippet:
bom = '\xef\xbb\xbf'
compile(bom + 'print 1\n', '', 'exec')
treats the `bom` prefix like any other sequence of illegal characters.
That's why it raises SyntaxError:
It peels off the first character (\xef), and says "syntax
error" at that point:
Py_CompileStringFlags ->
PyParser_ASTFromString ->
PyParser_ParseStringFlagsFilename ->
parsetok ->
PyTokenizer_Get
That sets `a` to point at the start of the string, `b` to point at the
second character, and returns type==51. Then `len` is set to 1,
`str` is malloc'ed to hold 2 bytes, and `str` is filled in with
"\xef\x00" (the first byte of the input, as a NUL-terminated C
string).
PyParser_AddToken then calls classify(), which falls off the end of
its last loop and returns -1: syntax error.
and later:
I'm getting a strong suspicion that I'm the only developer to _try_
building the trunk on WinXP since the AST merge was done, and that
something obscure is fundamentally broken with it on this box. For
example, in tokenizer.c, these functions don't even exist on Windows
today (because an enclosing #ifdef says not to compile them):
error_ret
new_string
get_normal_name
get_coding_spec
check_coding_spec
check_bom
fp_readl
fp_setreadl
fp_getc
fp_ungetc
decoding_fgets
decoding_feof
buf_getc
buf_ungetc
buf_setreadl
translate_into_utf8
decode_str
OK, that's not quite true. "Degenerate" forms of three of those
functions exist on Windows:
static char *
decoding_fgets(char *s, int size, struct tok_state *tok)
{
return fgets(s, size, tok->fp);
}
static int
decoding_feof(struct tok_state *tok)
{
return feof(tok->fp);
}
static const char *
decode_str(const char *str, struct tok_state *tok)
{
return str;
}
In the simple failing test, that degenerate decode_str() is getting
called. If the "fancy" decode_str() were being used instead, that one
_does_ call check_bom(). Why do we have two versions of these
functions? Which set is supposed to be in use now? What's the
meaning of "#ifdef PGEN" today? Should it be true or false?
>> I'm darned near certain that we're not using the _intended_ parsing
>> code on Windows now -- PGEN is still #define'd when the "final"
>> parsing code is compiled into python25.dll. Don't know how to fix
>> that (I don't understand it).
> But the AST branch didn't touch the parser (unless you are grouping
> ast.c and compile.c under the "parser" umbrella just to throw me off
> =).
Possibly. See above for unanswered questions about tokenizer.c, which
appears to be the whole problem wrt test_builtin. Python couldn't be
built under VC7.1 on Windows after the AST merge. However that got
repaired left parsing/tokenizing broken on Windows wrt (at least) some
encoding gimmicks. Since the tests passed immediately before the AST
merge, and failed the first time Python could be built again after
that merge, it's the only natural candidate for finger-wagging.
> What can I do to help?
I don't know. Enjoying Christmas couldn't hurt :-) What this needs
is someone who understands how
bom = '\xef\xbb\xbf'
compile(bom + 'print 1\n', '', 'exec')
is supposed to work at the front-end level.
> Do you need me to step through something?
Why doesn't the little code snippet above fail anywhere else?
Re: [Python-Dev] status of development documentation
On 12/24/05, Tim Peters <[EMAIL PROTECTED]> wrote:
> [Tim]
> >> FWIW, test_builtin and test_pep263 both passed on WinXP in rev 39757.
> >> That's the last revision before the AST branch was merged.
> >>
> >> I can't build rev 39758 on WinXP (VC complains that pythoncore.vcproj
> >> can't be loaded -- looks like it got checked in with unresolved SVN
> >> conflict markers -- which isn't easy to do under SVN ;-( ), so don't
> >> know about that.
> >>
> >> The first revision at which Python built again was 39791 (23 Oct), and
> >> test_builtin and test_pep263 both fail under that the same way they
> >> fail today.
>
> [Brett]
> > Both syntax errors, right?
>
> In test_builtin, yes, two syntax errors. test_pep263 is different:
>
> test test_pep263 failed -- Traceback (most recent call last):
> File "C:\Code\python\lib\test\test_pep263.py", line 12, in test_pep263
> '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd'
> AssertionError:
> '\xc3\xb0\xc3\x89\xc3\x94\xc3\x8f\xc3\x8e' !=
> '\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd'
>
> That's not a syntax error, it's a wrong result. There are other
> parsing-related test failures, but those are the only two I've written
> up so far (partly because I expect they all have the same underlying
> cause, and partly because nobody so far seems to understand the code
> well enough to explain why the first one works on any platform ;-)).
>
> > My mind is partially gone thanks to being on vacation so following this
> > thread
> > has been abnormally hard. =)
> >
> > Since it is a syntax error there won't be any bytecode to compare against.
>
> Shouldn't be needed. The snippet:
>
> bom = '\xef\xbb\xbf'
> compile(bom + 'print 1\n', '', 'exec')
>
> treats the `bom` prefix like any other sequence of illegal characters.
> That's why it raises SyntaxError:
>
> It peels off the first character (\xef), and says "syntax
> error" at that point:
>
> Py_CompileStringFlags ->
> PyParser_ASTFromString ->
> PyParser_ParseStringFlagsFilename ->
> parsetok ->
> PyTokenizer_Get
>
> That sets `a` to point at the start of the string, `b` to point at the
> second character, and returns type==51. Then `len` is set to 1,
> `str` is malloc'ed to hold 2 bytes, and `str` is filled in with
> "\xef\x00" (the first byte of the input, as a NUL-terminated C
> string).
>
> PyParser_AddToken then calls classify(), which falls off the end of
> its last loop and returns -1: syntax error.
>
> and later:
>
> I'm getting a strong suspicion that I'm the only developer to _try_
> building the trunk on WinXP since the AST merge was done, and that
> something obscure is fundamentally broken with it on this box. For
> example, in tokenizer.c, these functions don't even exist on Windows
> today (because an enclosing #ifdef says not to compile them):
>
> error_ret
> new_string
> get_normal_name
> get_coding_spec
> check_coding_spec
> check_bom
> fp_readl
> fp_setreadl
> fp_getc
> fp_ungetc
> decoding_fgets
> decoding_feof
> buf_getc
> buf_ungetc
> buf_setreadl
> translate_into_utf8
> decode_str
>
> OK, that's not quite true. "Degenerate" forms of three of those
> functions exist on Windows:
>
> static char *
> decoding_fgets(char *s, int size, struct tok_state *tok)
> {
>return fgets(s, size, tok->fp);
> }
>
> static int
> decoding_feof(struct tok_state *tok)
> {
>return feof(tok->fp);
> }
>
> static const char *
> decode_str(const char *str, struct tok_state *tok)
> {
> return str;
> }
>
> In the simple failing test, that degenerate decode_str() is getting
> called. If the "fancy" decode_str() were being used instead, that one
> _does_ call check_bom(). Why do we have two versions of these
> functions? Which set is supposed to be in use now? What's the
> meaning of "#ifdef PGEN" today? Should it be true or false?
>
Looking at the logs for tokenizer.c, tokenizer.h, and
tokenizer_pgen.c, it looks like this stuff has not been heavily
touched since Martin did stuff for PEP 263.
> >> I'm darned near certain that we're not using the _intended_ parsing
> >> code on Windows now -- PGEN is still #define'd when the "final"
> >> parsing code is compiled into python25.dll. Don't know how to fix
> >> that (I don't understand it).
>
> > But the AST branch didn't touch the parser (unless you are grouping
> > ast.c and compile.c under the "parser" umbrella just to throw me off
> > =).
>
> Possibly. See above for unanswered questions about tokenizer.c, which
> appears to be the whole problem wrt test_builtin. Python couldn't be
> built under VC7.1 on Windows after the AST merge. However that got
> repaired left parsing/tokenizing broken on Windows wrt (at least) some
> encoding gimmicks. Since the tests passed immediately before the AST
> merge, and failed the f
