Edward K. Ream <edream...@gmail.com> 于2024年10月11日周五 19:01写道： > > The file tbo-in-rust.leo (in my ekr-tbo-in-rust repo) now contains the > sources for the Rust and Python tokenizers, as shown by Leo's import > commands. This work concludes (for now) my work with Rust. > > > As you may recall, Python's tokenizer is much faster than Rust's. The sources > show why. Python's tokenizer is highly optimized C code, complete with > critical sections! > > > Code overview > > > - Most of Lib/tokenize.py pertains only to the (non-critical) untokenize > method. > > - Tokenize.py delegates the tokenize method to the > _generate_tokens_from_c_tokenizer in the _tokenize module. > > - It took several hours to find the _tokenize module! It's in the file > cpython/Python/Python-tokenize.c (!!) > > > The magic happens in the function: > > PyMODINIT_FUNC PyInit__tokenize(void); > > which is part of Python's C-language interface. > > > Summary > > > tbo-in-rust.leo now contains the sources for the Python and Rust tokenizers. > The Python tokenizer is highly optimized C code. > > > Today's work completes my study of Rust for now. > > > Edward > > -- > You received this message because you are subscribed to the Google Groups > "leo-editor" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to leo-editor+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/leo-editor/f590dcd4-6540-4fc6-b10e-45e2f40b3764n%40googlegroups.com.
That's why it is said: Python is just a friendly DSL of C language And Rust's goal is just a safe C++, but C++'s performance has never been as good as C; So, Leo's use of Rust may not bring about a qualitative improvement, but learning and using Rust is a good experience. -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAAFijRe0q1gHAeRhVquWjj7FqXU9rR1v-a6FAqk_aD85mHTz2w%40mail.gmail.com.
