Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-22 Thread Nikita Popov
So, is there any consensus about this? Nikita On Fri, Sep 16, 2011 at 10:07 AM, Ferenc Kovacs wrote: >> >> Wait wait wait. Thats the point here? >> __COMPILER_HALT_OFFSET__ already tells you where the data starts. >> >> -Hannes >> > > I didn't sent this message first, but after reading the mail

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-16 Thread Ferenc Kovacs
> > Wait wait wait. Thats the point here? > __COMPILER_HALT_OFFSET__ already tells you where the data starts. > > -Hannes > I didn't sent this message first, but after reading the mail from Chris, I think maybe it would clear the confusion: It is about tokenizing a file which has __halt_compiler(

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Nicolas Grekas
> What I am really hearing in this thread is some people have interest in a > way to easily access the __COMPILER_HALT_OFFSET__ Well, not exactly for me : I'm not interested strictly on getting the offset. I'm more interested in doing generic static code analysis, and part of that is getting the

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Chris Stockton
Hello, On Thu, Sep 15, 2011 at 8:13 AM, Nikita Popov wrote: > On Thu, Sep 15, 2011 at 5:05 PM, Hannes Magnusson > wrote: >> Wait wait wait. Thats the point here? >> __COMPILER_HALT_OFFSET__ already tells you where the data starts. > __COMPILER_HALT_OFFSET__ tells you the offset if you are runnin

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Nikita Popov
On Thu, Sep 15, 2011 at 5:05 PM, Hannes Magnusson wrote: > Wait wait wait. Thats the point here? > __COMPILER_HALT_OFFSET__ already tells you where the data starts. __COMPILER_HALT_OFFSET__ tells you the offset if you are running the file. It is only available in the file that does the actual __ha

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Hannes Magnusson
On Thu, Sep 15, 2011 at 16:51, Nikita Popov wrote: > Hi Hannes, thanks for your response! > > On Thu, Sep 15, 2011 at 9:22 AM, Hannes Magnusson > wrote: >> the change to ext/tokenizer/tests/token_get_all_variation16.phpt seems >> really weird? > I changed the test to not include a __halt_compiler

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Nikita Popov
Hi Hannes, thanks for your response! On Thu, Sep 15, 2011 at 9:22 AM, Hannes Magnusson wrote: > the change to ext/tokenizer/tests/token_get_all_variation16.phpt seems > really weird? I changed the test to not include a __halt_compiler statement anymore, as it didn't make sense in this context, an

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Hannes Magnusson
On Tue, Sep 13, 2011 at 22:08, Ferenc Kovacs wrote: > On Tue, Sep 13, 2011 at 9:56 PM, Christopher Jones > wrote: >> >> >> On 09/13/2011 12:25 PM, Nikita Popov wrote: >>> >>> I changed my previous patch to an SVN patch, so it is easier to apply >>> and added another patch (the one called "tokeniz

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Ferenc Kovacs
On Tue, Sep 13, 2011 at 9:56 PM, Christopher Jones wrote: > > > On 09/13/2011 12:25 PM, Nikita Popov wrote: >> >> I changed my previous patch to an SVN patch, so it is easier to apply >> and added another patch (the one called "tokenizer_patch_full.txt"), >> which additionally fetches the rest int

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Christopher Jones
On 09/13/2011 12:25 PM, Nikita Popov wrote: I changed my previous patch to an SVN patch, so it is easier to apply and added another patch (the one called "tokenizer_patch_full.txt"), which additionally fetches the rest into a T_INLINE_HTML. (The "_full" patch thus contains both changes. I didn'

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Nikita Popov
I changed my previous patch to an SVN patch, so it is easier to apply and added another patch (the one called "tokenizer_patch_full.txt"), which additionally fetches the rest into a T_INLINE_HTML. (The "_full" patch thus contains both changes. I didn't know how I could separate them.) Could somebo

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Nicolas Grekas
> I just set up an PHP environment and wrote a proper patch (including test > changes) to make it collect the next three tokens. It's a git patch and I'm > not sure whether it's compatible with SVN patches. I would love it if this > would go into 5.4 before beta. I didn't know how one could fetch t

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Nikita Popov
I just set up an PHP environment and wrote a proper patch (including test changes) to make it collect the next three tokens. It's a git patch and I'm not sure whether it's compatible with SVN patches. I would love it if this would go into 5.4 before beta. I didn't know how one could fetch the rest

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-09 Thread Nikita Popov
On Fri, Sep 9, 2011 at 10:46 AM, Nicolas Grekas wrote: > You can also just count the number of semantic token after T_HALT_COMPILER > (ie excluding whitespace and comments) and once you hit 3, halt. > [...] > Maybe the binary data itself, as a big T_INLINE_HTML for example ? In favor of both propo

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-09 Thread Nicolas Grekas
Thank you Nikita for take this subject here! On Fri, Sep 9, 2011 at 10:01, Ferenc Kovacs wrote: > don't break there but for the next ';'. You can also just count the number of semantic token after T_HALT_COMPILER (ie excluding whitespace and comments) and once you hit 3, halt. less confusing

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-09 Thread Ferenc Kovacs
On Fri, Sep 9, 2011 at 9:15 AM, Nikita Popov wrote: > In Bug #54089 [1] a patch was applied that cuts of token_get_all() > output after a T_HALT_COMPILER token. This was done because otherwise > PHP would keep on lexing after that and would generate errors because > of binary data (which is not va

[PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-09 Thread Nikita Popov
In Bug #54089 [1] a patch was applied that cuts of token_get_all() output after a T_HALT_COMPILER token. This was done because otherwise PHP would keep on lexing after that and would generate errors because of binary data (which is not valid PHP, mostly.) The problem with the patch is, that there