On 04/19/2015 05:42 PM, BartC wrote:
So I'm aware of some of the things that are involved. (BTW that project worked reasonably well, but I decided to go in a different direction: turning "J" from a mere syntax into an actual language of its own.)
Something you might try with your new language is to have an editor that can parse the source text to a tokenised "text" data file. It should be a fairly direct text to text translation, so putting it into the editor should be doable.
Once you get that far, you can add pluggable token parsers to the editor that can unparse the token files to a specific user format for editing, and reparse it back to the standardised token file for storage. Now the users can choose a customised dialect of your language. And the compiler will work on a single token file type.
This will also remove the need for pretty printers and formatters as they will be built into the editors pluggable token parser. Just have the editor tokenise and then immediately untokenise and you just checked for syntax errors and reformatted the program. The highlighter could also work with the token parser as well. The only catch is how to handle compile and run time errors. The editor will need to read an error file and translate it back to the source. I think it's doable but it will be tricky to get it to work well.
The reason to make the split at tokens is, it's at a stage that most of what is needed is already in a text editor.
The token file format becomes the bridge that spans the gap between the user/editor and the compiler/interpreter.
Then your compiler/interpreter is all implementation details that work on a single standardised token file format rather than unformatted text files.
Ron -- https://mail.python.org/mailman/listinfo/python-list
