ilya-biryukov added inline comments.
================ Comment at: clang/include/clang/Tooling/Syntax/Corpus.h:38 + std::pair<FileID, llvm::ArrayRef<syntax::Token>> + tokenizeBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer); + ---------------- sammccall wrote: > ilya-biryukov wrote: > > sammccall wrote: > > > Are you planning to have a way to add tokens directly? Having to turn > > > them into text and re-lex them seems like it might be inconvenient. > > The tokens have source locations and refer to a text in some buffer. > > `tokenizeBuffer` makes is easier, not harder, to mock tokens. > Fair enough. > > `lexBuffer` might be a slightly clearer name? > > Who are the intended non-test users of this function? Are they better served > by being able (and responsible) for constructing a MemoryBuffer with e.g. a > sensible name and ownership, or would it be better to pass a StringRef and > have the Arena come up with a sensible "anonymous" name? The only use-case in my prototype so far is creating token nodes for punctuation nodes, e.g. say you want to create an expr of the form `<a>+<b>`, where both `a` and `b` are existing expressions and you need to synthesize a leaf node for `+`. We use this function to synthesize a buffer with the corresponding token. All the use-cases I can imagine are around synthesizing syntax trees (as opposed to constructing them from the AST). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D61637/new/ https://reviews.llvm.org/D61637 _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
