On Sat, 08 Jan 2005 18:22:53 +1000, Nick Coghlan <[EMAIL PROTECTED]> wrote:
>Bengt Richter wrote: >> IOW, I think there is a fix: keep tokenizing greedily and tokenize floating >> point as >> a sequence of integers and operators, and let <integer><dot><integer> be >> translated by >> the compiler to floating point, and <integer><dotdot><integer> be translated >> to the >> appropriate generator expression implementation. > >That would be: > ><int-literal><dot><int-literal> -> float(<int-literal> + "." + <int-literal>) ><int-literal><dot><identifier> -> getattr(int(<int-literal>), <identifier>) ><int-literal><dot><dot><int-literal> -> xrange(<int-literal>, <int-literal>) > >However, the problem comes when you realise that 1e3 is also a floating point >literal, as is 1.1e3. > Ok, that requires a little more imagination ;-) I think it can be solved, but I haven't explored it all the way through ;-) The key seems to be to be to condition the recognition of tokens as if recognizing an old potentially floating point number, but emitting number-relevant separate tokens so long as there is no embedded spaces. When a number ends, emitting an end marker should permit the compiler to deal with the various compositions. We still aren't looking ahead more than one, but we are carrying context, just as we do to accumulate digits of an integer or characters of a name, but the context may continue and determine what further tokens are emitted. E.g. the 'e' in the embedded numeric context, becomes <fexp> rather than a name. In the following, <eon> :== end of number token 1.1 -> <1><dot><1><eon> 1 .1 -> <1><eon><dot><1><eon> 1.e1 -> <1><dot><fexp><1><eon> 1 .e1 -> <1><eon><dot><e1> 1.2e3 -> <1><dot><2><fexp><3><eon> 1..2 -> <1><eon><doubledot><1><eon> 1. .2 -> <1><dot><eon><dot><2><eon> (syntax error) 1 ..2 -> <1><eon><doubledot><1><eon> I just have the feeling that there is a solution, whatever the hiccups ;-) Regards, Bengt Richter -- http://mail.python.org/mailman/listinfo/python-list
