BTW, the loop always happens at the VERY end of the comment, so when you
get to the end the(the last call of next() that should return null);
tok = tokenizer.next();
never returns, it gets stuck trying to exit.
Mike
On Thu, Jun 4, 2015 at 2:34 PM, Michael Schmalle <teotigraphix...@gmail.com>
wrote:
> I posted about this a couple weeks ago and I tried recompiling with JFlex
> 1.5 I think, the older version and still had the problem.
>
> Maybe I messed up something but I tried with my same asdoc code when I
> fixed the build for the FlexJS asdocs. I wanted to see it work with my
> version of a documentor.
>
> I think IIRC, I actually tried a simple test case and it would work.
>
> I have code that uses the tokenizer;
>
>
> public void compile()
> {
> if (token == null)
> return;
>
> String data = token.getText();
> ASDocTokenizer tokenizer = new ASDocTokenizer(false);
> tokenizer.setReader(new StringReader(data));
> ASDocToken tok = tokenizer.next();
> boolean foundDescription = false;
> DocTag pendingTag = null;
>
> try
> {
> while (tok != null)
> {
> if (!foundDescription
> && tok.getType() == ASTokenTypes.TOKEN_ASDOC_TEXT)
> {
> description = tok.getText();
> }
> else
> {
> // do tags
> if (tok.getType() == ASTokenTypes.TOKEN_ASDOC_TAG)
> {
> if (pendingTag != null)
> {
> addTag(pendingTag);
> pendingTag = null;
> }
> pendingTag = new
> DocTag(tok.getText().substring(1));
> }
> else if (tok.getType() ==
> ASTokenTypes.TOKEN_ASDOC_TEXT)
> {
> pendingTag.setDescription(tok.getText());
> addTag(pendingTag);
> pendingTag = null;
> }
> }
>
> foundDescription = true;
>
> tok = tokenizer.next();
> }
> }
> catch (Exception e)
> {
> e.printStackTrace();
> }
> }
>
> Mike
>
>
> On Thu, Jun 4, 2015 at 2:30 PM, Alex Harui <aha...@adobe.com> wrote:
>
>>
>>
>> On 6/4/15, 11:23 AM, "Michael Schmalle" <teotigraphix...@gmail.com>
>> wrote:
>> >>Hmm. Maybe I should spend some time looking into fixing ASDocTokenizer?
>> >> Was the problem that it didn’t work on every AS file we current have?
>> >>
>> >
>> >
>> >It doesn't work on anything, there is an infinite loop in the scanner
>> that
>> >is created by JFlex, the RawASDocTokenizer is broken.
>> >
>> >What is weird is I was using the SAME code base when I wrote the asdoc
>> >documenter I have 2 years ago and it worked fine.
>>
>> We upgraded the version of JFlex, IIRC. I’ll take a look. What setup did
>> you have for trying it? Did you run it on the Flex SDK or FlexJS SDK or
>> did it even loop on a simple test case?
>>
>> -Alex
>>
>>
>
