Tokenizer queston: how can I force ? and ! to be separate tokens?

[OBender]

Hi All,

 

I need to make ? and ! characters to be a separate token e.g. to split [how
are you?] in to 4 tokens [how], [are], [you] and [?] what would be the best
way to do this?

 

Thanks