Thomas Lotze wrote: > Does anybody here have a third way of dealing with this?
Sleeping a night sometimes is an insightful exercise *g* I realized that there is a reason why fiddling with the pointer from outside the generator defeats much of the purpose of using one. The implementation using a simple method call instead of a generator needs to store some internal state variables on an object to save them for the next call, among them the pointer and a tokenization mode. I could make the thing a generator by turning the single return statement into a yield statement and adding a loop, leaving all the importing and exporting of the pointer intact - after all, someone might reset the pointer between next() calls. This is, however, hardly using all the possibilities a generator allows. I'd rather like to get rid of the mode switches by doing special things where I detect the need for them, yielding the result, and proceeding as before. But as soon as I move information from explicit (state variables that can be reset along with the pointer) to implicit (the point where the generator is suspended after yielding a token), resetting the pointer will lead to inconsistencies. So, it seems to me that if I do want to use generators for any practical reason instead of just because generators are way cool, they need to be instantiated anew each time the pointer is reset, for simple consistency reasons. Now a very simple idea struck me: If one is worried about throwing away a generator as a side-effect of resetting the tokenization pointer, why not define the whole tokenizer as not being resettable? Then the thing needs to be re-instantiated very explicitly every time it is pointed somewhere. While still feeling slightly awkward, it has lost the threat of doing unexpected things. Does this sound reasonable? -- Thomas -- http://mail.python.org/mailman/listinfo/python-list
