llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT--> @llvm/pr-subscribers-clang Author: Aaron Ballman (AaronBallman) <details> <summary>Changes</summary> The functionality is incomplete and the authors have since shifted gears to other work, so this is effectively unmaintained. The original design document for clang-pseudo can be found at: https://docs.google.com/document/d/1eGkTOsFja63wsv8v0vd5JdoTonj-NlN3ujGF0T7xDbM/edit in case anyone wishes to pick this project back up again in the future. Original RFC: https://discourse.llvm.org/t/removing-pseudo-parser/71131/ --- Patch is 392.50 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/109154.diff 102 Files Affected: - (modified) clang-tools-extra/CMakeLists.txt (-1) - (modified) clang-tools-extra/clangd/CMakeLists.txt (-1) - (modified) clang-tools-extra/clangd/SemanticSelection.cpp (+16-16) - (renamed) clang-tools-extra/clangd/support/Bracket.cpp (+4-4) - (renamed) clang-tools-extra/clangd/support/Bracket.h (+6-6) - (modified) clang-tools-extra/clangd/support/CMakeLists.txt (+4) - (renamed) clang-tools-extra/clangd/support/DirectiveTree.cpp (+3-3) - (renamed) clang-tools-extra/clangd/support/DirectiveTree.h (+6-6) - (renamed) clang-tools-extra/clangd/support/Lex.cpp (+3-3) - (renamed) clang-tools-extra/clangd/support/Token.cpp (+3-3) - (renamed) clang-tools-extra/clangd/support/Token.h (+5-5) - (removed) clang-tools-extra/pseudo/CMakeLists.txt (-12) - (removed) clang-tools-extra/pseudo/DesignNotes.md (-123) - (removed) clang-tools-extra/pseudo/Disambiguation.md (-367) - (removed) clang-tools-extra/pseudo/README.md (-37) - (removed) clang-tools-extra/pseudo/benchmarks/Benchmark.cpp (-156) - (removed) clang-tools-extra/pseudo/benchmarks/CMakeLists.txt (-9) - (removed) clang-tools-extra/pseudo/fuzzer/CMakeLists.txt (-16) - (removed) clang-tools-extra/pseudo/fuzzer/Fuzzer.cpp (-82) - (removed) clang-tools-extra/pseudo/fuzzer/Main.cpp (-16) - (removed) clang-tools-extra/pseudo/gen/CMakeLists.txt (-11) - (removed) clang-tools-extra/pseudo/gen/Main.cpp (-172) - (removed) clang-tools-extra/pseudo/include/CMakeLists.txt (-32) - (removed) clang-tools-extra/pseudo/include/clang-pseudo/Disambiguate.h (-64) - (removed) clang-tools-extra/pseudo/include/clang-pseudo/Forest.h (-236) - (removed) clang-tools-extra/pseudo/include/clang-pseudo/GLR.h (-170) - (removed) clang-tools-extra/pseudo/include/clang-pseudo/Language.h (-64) - (removed) clang-tools-extra/pseudo/include/clang-pseudo/cli/CLI.h (-35) - (removed) clang-tools-extra/pseudo/include/clang-pseudo/cxx/CXX.h (-91) - (removed) clang-tools-extra/pseudo/include/clang-pseudo/grammar/Grammar.h (-230) - (removed) clang-tools-extra/pseudo/include/clang-pseudo/grammar/LRGraph.h (-196) - (removed) clang-tools-extra/pseudo/include/clang-pseudo/grammar/LRTable.h (-278) - (removed) clang-tools-extra/pseudo/lib/CMakeLists.txt (-31) - (removed) clang-tools-extra/pseudo/lib/Disambiguate.cpp (-48) - (removed) clang-tools-extra/pseudo/lib/Forest.cpp (-199) - (removed) clang-tools-extra/pseudo/lib/GLR.cpp (-772) - (removed) clang-tools-extra/pseudo/lib/cli/CLI.cpp (-54) - (removed) clang-tools-extra/pseudo/lib/cli/CMakeLists.txt (-15) - (removed) clang-tools-extra/pseudo/lib/cxx/CMakeLists.txt (-19) - (removed) clang-tools-extra/pseudo/lib/cxx/CXX.cpp (-452) - (removed) clang-tools-extra/pseudo/lib/cxx/cxx.bnf (-775) - (removed) clang-tools-extra/pseudo/lib/grammar/CMakeLists.txt (-10) - (removed) clang-tools-extra/pseudo/lib/grammar/Grammar.cpp (-190) - (removed) clang-tools-extra/pseudo/lib/grammar/GrammarBNF.cpp (-362) - (removed) clang-tools-extra/pseudo/lib/grammar/LRGraph.cpp (-265) - (removed) clang-tools-extra/pseudo/lib/grammar/LRTable.cpp (-79) - (removed) clang-tools-extra/pseudo/lib/grammar/LRTableBuild.cpp (-121) - (removed) clang-tools-extra/pseudo/test/.clang-format (-1) - (removed) clang-tools-extra/pseudo/test/CMakeLists.txt (-29) - (removed) clang-tools-extra/pseudo/test/Unit/lit.cfg.py (-25) - (removed) clang-tools-extra/pseudo/test/Unit/lit.site.cfg.py.in (-11) - (removed) clang-tools-extra/pseudo/test/check-cxx-bnf.test (-2) - (removed) clang-tools-extra/pseudo/test/crash/backslashes.c (-4) - (removed) clang-tools-extra/pseudo/test/cxx/capture-list.cpp (-23) - (removed) clang-tools-extra/pseudo/test/cxx/contextual-keywords.cpp (-9) - (removed) clang-tools-extra/pseudo/test/cxx/dangling-else.cpp (-22) - (removed) clang-tools-extra/pseudo/test/cxx/decl-specfier-seq.cpp (-27) - (removed) clang-tools-extra/pseudo/test/cxx/declarator-function.cpp (-9) - (removed) clang-tools-extra/pseudo/test/cxx/declarator-var.cpp (-9) - (removed) clang-tools-extra/pseudo/test/cxx/declator-member-function.cpp (-9) - (removed) clang-tools-extra/pseudo/test/cxx/empty-member-declaration.cpp (-7) - (removed) clang-tools-extra/pseudo/test/cxx/empty-member-spec.cpp (-13) - (removed) clang-tools-extra/pseudo/test/cxx/keyword.cpp (-12) - (removed) clang-tools-extra/pseudo/test/cxx/literals.cpp (-43) - (removed) clang-tools-extra/pseudo/test/cxx/mixed-designator.cpp (-27) - (removed) clang-tools-extra/pseudo/test/cxx/nested-name-specifier.cpp (-28) - (removed) clang-tools-extra/pseudo/test/cxx/parameter-decl-clause.cpp (-14) - (removed) clang-tools-extra/pseudo/test/cxx/predefined-identifier.cpp (-5) - (removed) clang-tools-extra/pseudo/test/cxx/recovery-func-parameters.cpp (-13) - (removed) clang-tools-extra/pseudo/test/cxx/recovery-init-list.cpp (-13) - (removed) clang-tools-extra/pseudo/test/cxx/structured-binding.cpp (-6) - (removed) clang-tools-extra/pseudo/test/cxx/template-empty-type-parameter.cpp (-3) - (removed) clang-tools-extra/pseudo/test/cxx/unsized-array.cpp (-7) - (removed) clang-tools-extra/pseudo/test/fuzzer.cpp (-4) - (removed) clang-tools-extra/pseudo/test/glr-variant-start.cpp (-9) - (removed) clang-tools-extra/pseudo/test/glr.cpp (-30) - (removed) clang-tools-extra/pseudo/test/html-forest.c (-8) - (removed) clang-tools-extra/pseudo/test/lex.c (-42) - (removed) clang-tools-extra/pseudo/test/lit.cfg.py (-20) - (removed) clang-tools-extra/pseudo/test/lit.local.cfg (-2) - (removed) clang-tools-extra/pseudo/test/lit.site.cfg.py.in (-14) - (removed) clang-tools-extra/pseudo/test/lr-build-basic.test (-32) - (removed) clang-tools-extra/pseudo/test/lr-build-conflicts.test (-49) - (removed) clang-tools-extra/pseudo/test/strip-directives.c (-49) - (removed) clang-tools-extra/pseudo/tool/CMakeLists.txt (-30) - (removed) clang-tools-extra/pseudo/tool/ClangPseudo.cpp (-243) - (removed) clang-tools-extra/pseudo/tool/HTMLForest.cpp (-192) - (removed) clang-tools-extra/pseudo/tool/HTMLForest.css (-93) - (removed) clang-tools-extra/pseudo/tool/HTMLForest.html (-15) - (removed) clang-tools-extra/pseudo/tool/HTMLForest.js (-290) - (removed) clang-tools-extra/pseudo/unittests/BracketTest.cpp (-117) - (removed) clang-tools-extra/pseudo/unittests/CMakeLists.txt (-32) - (removed) clang-tools-extra/pseudo/unittests/CXXTest.cpp (-30) - (removed) clang-tools-extra/pseudo/unittests/DirectiveTreeTest.cpp (-357) - (removed) clang-tools-extra/pseudo/unittests/DisambiguateTest.cpp (-111) - (removed) clang-tools-extra/pseudo/unittests/ForestTest.cpp (-180) - (removed) clang-tools-extra/pseudo/unittests/GLRTest.cpp (-789) - (removed) clang-tools-extra/pseudo/unittests/GrammarTest.cpp (-213) - (removed) clang-tools-extra/pseudo/unittests/LRTableTest.cpp (-76) - (removed) clang-tools-extra/pseudo/unittests/TokenTest.cpp (-224) - (modified) clang/docs/ClangFormattedStatus.rst (-5) - (modified) clang/docs/tools/clang-formatted-files.txt (-1) ``````````diff diff --git a/clang-tools-extra/CMakeLists.txt b/clang-tools-extra/CMakeLists.txt index f6a6b57b5ef0bc..6b6f2b1ca22765 100644 --- a/clang-tools-extra/CMakeLists.txt +++ b/clang-tools-extra/CMakeLists.txt @@ -27,7 +27,6 @@ add_subdirectory(clang-move) add_subdirectory(clang-query) add_subdirectory(include-cleaner) add_subdirectory(pp-trace) -add_subdirectory(pseudo) add_subdirectory(tool-template) option(CLANG_TOOLS_EXTRA_INCLUDE_DOCS "Generate build targets for the Clang Extra Tools docs." diff --git a/clang-tools-extra/clangd/CMakeLists.txt b/clang-tools-extra/clangd/CMakeLists.txt index c21d277d2ffcbd..8dcbf5f47e056a 100644 --- a/clang-tools-extra/clangd/CMakeLists.txt +++ b/clang-tools-extra/clangd/CMakeLists.txt @@ -183,7 +183,6 @@ target_link_libraries(clangDaemon ${LLVM_PTHREAD_LIB} clangIncludeCleaner - clangPseudo clangTidy clangTidyUtils diff --git a/clang-tools-extra/clangd/SemanticSelection.cpp b/clang-tools-extra/clangd/SemanticSelection.cpp index 3d687173b2be99..dd7116e619e6d0 100644 --- a/clang-tools-extra/clangd/SemanticSelection.cpp +++ b/clang-tools-extra/clangd/SemanticSelection.cpp @@ -11,9 +11,6 @@ #include "Protocol.h" #include "Selection.h" #include "SourceCode.h" -#include "clang-pseudo/Bracket.h" -#include "clang-pseudo/DirectiveTree.h" -#include "clang-pseudo/Token.h" #include "clang/AST/DeclBase.h" #include "clang/Basic/SourceLocation.h" #include "clang/Basic/SourceManager.h" @@ -25,6 +22,9 @@ #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" #include "llvm/Support/Error.h" +#include "support/Bracket.h" +#include "support/DirectiveTree.h" +#include "support/Token.h" #include <optional> #include <queue> #include <vector> @@ -181,16 +181,16 @@ llvm::Expected<std::vector<FoldingRange>> getFoldingRanges(ParsedAST &AST) { // Related issue: https://github.com/clangd/clangd/issues/310 llvm::Expected<std::vector<FoldingRange>> getFoldingRanges(const std::string &Code, bool LineFoldingOnly) { - auto OrigStream = pseudo::lex(Code, clang::pseudo::genericLangOpts()); + auto OrigStream = lex(Code, genericLangOpts()); - auto DirectiveStructure = pseudo::DirectiveTree::parse(OrigStream); - pseudo::chooseConditionalBranches(DirectiveStructure, OrigStream); + auto DirectiveStructure = DirectiveTree::parse(OrigStream); + chooseConditionalBranches(DirectiveStructure, OrigStream); // FIXME: Provide ranges in the disabled-PP regions as well. auto Preprocessed = DirectiveStructure.stripDirectives(OrigStream); - auto ParseableStream = cook(Preprocessed, clang::pseudo::genericLangOpts()); - pseudo::pairBrackets(ParseableStream); + auto ParseableStream = cook(Preprocessed, genericLangOpts()); + pairBrackets(ParseableStream); std::vector<FoldingRange> Result; auto AddFoldingRange = [&](Position Start, Position End, @@ -205,19 +205,19 @@ getFoldingRanges(const std::string &Code, bool LineFoldingOnly) { FR.kind = Kind.str(); Result.push_back(FR); }; - auto OriginalToken = [&](const pseudo::Token &T) { + auto OriginalToken = [&](const Token &T) { return OrigStream.tokens()[T.OriginalIndex]; }; - auto StartOffset = [&](const pseudo::Token &T) { + auto StartOffset = [&](const Token &T) { return OriginalToken(T).text().data() - Code.data(); }; - auto StartPosition = [&](const pseudo::Token &T) { + auto StartPosition = [&](const Token &T) { return offsetToPosition(Code, StartOffset(T)); }; - auto EndOffset = [&](const pseudo::Token &T) { + auto EndOffset = [&](const Token &T) { return StartOffset(T) + OriginalToken(T).Length; }; - auto EndPosition = [&](const pseudo::Token &T) { + auto EndPosition = [&](const Token &T) { return offsetToPosition(Code, EndOffset(T)); }; auto Tokens = ParseableStream.tokens(); @@ -235,7 +235,7 @@ getFoldingRanges(const std::string &Code, bool LineFoldingOnly) { } } } - auto IsBlockComment = [&](const pseudo::Token &T) { + auto IsBlockComment = [&](const Token &T) { assert(T.Kind == tok::comment); return OriginalToken(T).Length >= 2 && Code.substr(StartOffset(T), 2) == "/*"; @@ -246,10 +246,10 @@ getFoldingRanges(const std::string &Code, bool LineFoldingOnly) { T++; continue; } - pseudo::Token *FirstComment = T; + Token *FirstComment = T; // Show starting sentinals (// and /*) of the comment. Position Start = offsetToPosition(Code, 2 + StartOffset(*FirstComment)); - pseudo::Token *LastComment = T; + Token *LastComment = T; Position End = EndPosition(*T); while (T != Tokens.end() && T->Kind == tok::comment && StartPosition(*T).line <= End.line + 1) { diff --git a/clang-tools-extra/pseudo/lib/Bracket.cpp b/clang-tools-extra/clangd/support/Bracket.cpp similarity index 97% rename from clang-tools-extra/pseudo/lib/Bracket.cpp rename to clang-tools-extra/clangd/support/Bracket.cpp index 07836146ad8a58..93d0f38015efbf 100644 --- a/clang-tools-extra/pseudo/lib/Bracket.cpp +++ b/clang-tools-extra/clangd/support/Bracket.cpp @@ -62,10 +62,10 @@ // //===----------------------------------------------------------------------===// -#include "clang-pseudo/Bracket.h" +#include "Bracket.h" namespace clang { -namespace pseudo { +namespace clangd { namespace { struct Bracket { @@ -83,7 +83,7 @@ struct Bracket { // Find brackets in the stream and convert to Bracket struct. std::vector<Bracket> findBrackets(const TokenStream &Stream) { std::vector<Bracket> Brackets; - auto Add = [&](const pseudo::Token &Tok, Bracket::BracketKind K, + auto Add = [&](const Token &Tok, Bracket::BracketKind K, Bracket::Direction D) { Brackets.push_back( {K, D, Tok.Line, Tok.Indent, Stream.index(Tok), Bracket::None}); @@ -151,5 +151,5 @@ void pairBrackets(TokenStream &Stream) { applyPairings(Brackets, Stream); } -} // namespace pseudo +} // namespace clangd } // namespace clang diff --git a/clang-tools-extra/pseudo/include/clang-pseudo/Bracket.h b/clang-tools-extra/clangd/support/Bracket.h similarity index 87% rename from clang-tools-extra/pseudo/include/clang-pseudo/Bracket.h rename to clang-tools-extra/clangd/support/Bracket.h index 268cfff1ab07ab..b43c22cea06d06 100644 --- a/clang-tools-extra/pseudo/include/clang-pseudo/Bracket.h +++ b/clang-tools-extra/clangd/support/Bracket.h @@ -23,19 +23,19 @@ // //===----------------------------------------------------------------------===// -#ifndef CLANG_PSEUDO_BRACKET_H -#define CLANG_PSEUDO_BRACKET_H +#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_BRACKET_H +#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_BRACKET_H -#include "clang-pseudo/Token.h" +#include "Token.h" namespace clang { -namespace pseudo { +namespace clangd { /// Identifies bracket token in the stream which should be paired. /// Sets Token::Pair accordingly. void pairBrackets(TokenStream &); -} // namespace pseudo +} // namespace clangd } // namespace clang -#endif +#endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_BRACKET_H diff --git a/clang-tools-extra/clangd/support/CMakeLists.txt b/clang-tools-extra/clangd/support/CMakeLists.txt index 0c8c199dd4a4c2..a1a2a7765fae95 100644 --- a/clang-tools-extra/clangd/support/CMakeLists.txt +++ b/clang-tools-extra/clangd/support/CMakeLists.txt @@ -16,9 +16,12 @@ if(NOT HAVE_CXX_ATOMICS_WITHOUT_LIB OR NOT HAVE_CXX_ATOMICS64_WITHOUT_LIB) endif() add_clang_library(clangdSupport + Bracket.cpp Cancellation.cpp Context.cpp + DirectiveTree.cpp FileCache.cpp + Lex.cpp Logger.cpp Markup.cpp MemoryTree.cpp @@ -27,6 +30,7 @@ add_clang_library(clangdSupport ThreadCrashReporter.cpp Threading.cpp ThreadsafeFS.cpp + Token.cpp Trace.cpp LINK_LIBS diff --git a/clang-tools-extra/pseudo/lib/DirectiveTree.cpp b/clang-tools-extra/clangd/support/DirectiveTree.cpp similarity index 99% rename from clang-tools-extra/pseudo/lib/DirectiveTree.cpp rename to clang-tools-extra/clangd/support/DirectiveTree.cpp index 9e853e46edc232..d25da111681afc 100644 --- a/clang-tools-extra/pseudo/lib/DirectiveTree.cpp +++ b/clang-tools-extra/clangd/support/DirectiveTree.cpp @@ -6,7 +6,7 @@ // //===----------------------------------------------------------------------===// -#include "clang-pseudo/DirectiveTree.h" +#include "DirectiveTree.h" #include "clang/Basic/IdentifierTable.h" #include "clang/Basic/TokenKinds.h" #include "llvm/Support/FormatVariadic.h" @@ -14,7 +14,7 @@ #include <variant> namespace clang { -namespace pseudo { +namespace clangd { namespace { class DirectiveParser { @@ -353,5 +353,5 @@ TokenStream DirectiveTree::stripDirectives(const TokenStream &In) const { return Out; } -} // namespace pseudo +} // namespace clangd } // namespace clang diff --git a/clang-tools-extra/pseudo/include/clang-pseudo/DirectiveTree.h b/clang-tools-extra/clangd/support/DirectiveTree.h similarity index 95% rename from clang-tools-extra/pseudo/include/clang-pseudo/DirectiveTree.h rename to clang-tools-extra/clangd/support/DirectiveTree.h index 2b6cb63297915d..34f5a888863f26 100644 --- a/clang-tools-extra/pseudo/include/clang-pseudo/DirectiveTree.h +++ b/clang-tools-extra/clangd/support/DirectiveTree.h @@ -25,17 +25,17 @@ // //===----------------------------------------------------------------------===// -#ifndef CLANG_PSEUDO_DIRECTIVETREE_H -#define CLANG_PSEUDO_DIRECTIVETREE_H +#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_DIRECTIVETREE_H +#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_DIRECTIVETREE_H -#include "clang-pseudo/Token.h" +#include "Token.h" #include "clang/Basic/TokenKinds.h" #include <optional> #include <variant> #include <vector> namespace clang { -namespace pseudo { +namespace clangd { /// Describes the structure of a source file, as seen by the preprocessor. /// @@ -124,7 +124,7 @@ llvm::raw_ostream &operator<<(llvm::raw_ostream &, /// The choices are stored in Conditional::Taken nodes. void chooseConditionalBranches(DirectiveTree &, const TokenStream &Code); -} // namespace pseudo +} // namespace clangd } // namespace clang -#endif // CLANG_PSEUDO_DIRECTIVETREE_H +#endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_DIRECTIVETREE_H diff --git a/clang-tools-extra/pseudo/lib/Lex.cpp b/clang-tools-extra/clangd/support/Lex.cpp similarity index 98% rename from clang-tools-extra/pseudo/lib/Lex.cpp rename to clang-tools-extra/clangd/support/Lex.cpp index 2111476f04dc5b..f043b551b6bc6c 100644 --- a/clang-tools-extra/pseudo/lib/Lex.cpp +++ b/clang-tools-extra/clangd/support/Lex.cpp @@ -6,7 +6,7 @@ // //===----------------------------------------------------------------------===// -#include "clang-pseudo/Token.h" +#include "Token.h" #include "clang/Basic/IdentifierTable.h" #include "clang/Basic/SourceLocation.h" #include "clang/Basic/TokenKinds.h" @@ -14,7 +14,7 @@ #include "clang/Lex/LiteralSupport.h" namespace clang { -namespace pseudo { +namespace clangd { TokenStream lex(const std::string &Code, const clang::LangOptions &LangOpts) { clang::SourceLocation Start; @@ -135,5 +135,5 @@ TokenStream cook(const TokenStream &Code, const LangOptions &LangOpts) { return Result; } -} // namespace pseudo +} // namespace clangd } // namespace clang diff --git a/clang-tools-extra/pseudo/lib/Token.cpp b/clang-tools-extra/clangd/support/Token.cpp similarity index 98% rename from clang-tools-extra/pseudo/lib/Token.cpp rename to clang-tools-extra/clangd/support/Token.cpp index 5b07a62f37fb2f..13eee66606061d 100644 --- a/clang-tools-extra/pseudo/lib/Token.cpp +++ b/clang-tools-extra/clangd/support/Token.cpp @@ -6,14 +6,14 @@ // //===----------------------------------------------------------------------===// -#include "clang-pseudo/Token.h" +#include "Token.h" #include "clang/Basic/LangOptions.h" #include "llvm/ADT/StringExtras.h" #include "llvm/Support/Format.h" #include "llvm/Support/FormatVariadic.h" namespace clang { -namespace pseudo { +namespace clangd { llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const Token &T) { OS << llvm::formatv("{0} {1}:{2} ", clang::tok::getTokenName(T.Kind), T.Line, @@ -126,5 +126,5 @@ TokenStream stripComments(const TokenStream &Input) { return Out; } -} // namespace pseudo +} // namespace clangd } // namespace clang diff --git a/clang-tools-extra/pseudo/include/clang-pseudo/Token.h b/clang-tools-extra/clangd/support/Token.h similarity index 98% rename from clang-tools-extra/pseudo/include/clang-pseudo/Token.h rename to clang-tools-extra/clangd/support/Token.h index 859fd7d2b3dfe2..555b6b0e4ce570 100644 --- a/clang-tools-extra/pseudo/include/clang-pseudo/Token.h +++ b/clang-tools-extra/clangd/support/Token.h @@ -25,8 +25,8 @@ // //===----------------------------------------------------------------------===// -#ifndef CLANG_PSEUDO_TOKEN_H -#define CLANG_PSEUDO_TOKEN_H +#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_TOKEN_H +#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_TOKEN_H #include "clang/Basic/LLVM.h" #include "clang/Basic/LangStandard.h" @@ -41,7 +41,7 @@ namespace clang { class LangOptions; -namespace pseudo { +namespace clangd { /// A single C++ or preprocessor token. /// @@ -249,7 +249,7 @@ TokenStream cook(const TokenStream &, const clang::LangOptions &); /// Drops comment tokens. TokenStream stripComments(const TokenStream &); -} // namespace pseudo +} // namespace clangd } // namespace clang -#endif // CLANG_PSEUDO_TOKEN_H +#endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_TOKEN_H diff --git a/clang-tools-extra/pseudo/CMakeLists.txt b/clang-tools-extra/pseudo/CMakeLists.txt deleted file mode 100644 index 24bc1530bb7d6f..00000000000000 --- a/clang-tools-extra/pseudo/CMakeLists.txt +++ /dev/null @@ -1,12 +0,0 @@ -include_directories(include) -include_directories(${CMAKE_CURRENT_BINARY_DIR}/include) -add_subdirectory(include) -add_subdirectory(gen) -add_subdirectory(lib) -add_subdirectory(tool) -add_subdirectory(fuzzer) -add_subdirectory(benchmarks) -if(CLANG_INCLUDE_TESTS) - add_subdirectory(unittests) - add_subdirectory(test) -endif() diff --git a/clang-tools-extra/pseudo/DesignNotes.md b/clang-tools-extra/pseudo/DesignNotes.md deleted file mode 100644 index 421cc02aef7576..00000000000000 --- a/clang-tools-extra/pseudo/DesignNotes.md +++ /dev/null @@ -1,123 +0,0 @@ -# Error recovery (2022-05-07) - -Problem: we have two fairly generic cases of recovery bounded within a range: - - sequences: `int x; this is absolute garbage; x++;` - - brackets: `void foo() { this is garbage too }` - -So far, we've thought of these as different, and had precise ideas about -brackets ("lazy parsing") and vague ones about sequences. -Con we unify them instead? - -In both cases we want to recognize the bounds of the garbage item based on -basic token-level features of the surrounding code, and avoid any interference -with the surrounding code. - -## Brackets - -Consider a rule like `compound-stmt := { stmt-seq }`. - -The desired recovery is: -- if we fail at `{ . stmt-seq }` -- ... and we can find for the matching `}` -- then consume up to that token as an opaque broken `stmt-seq` -- ... and advance state to `{ stmt-seq . }` - -We can annotate the rule to describe this: `{ stmt-seq [recovery] }`. -We can generalize as `{ stmt-seq [recovery=rbrace] }`, allowing for different -**strategies** to find the resume point. - -(It's OK to assume we're skipping over one RHS nonterminal, we can always -introduce a nonterminal for the bracket contents if necessary). - -## Sequences - -Can we apply the same technique to sequences? -Simplest case: delimited right-recursive sequence. - -``` -param-list := param -param-list := param , param-list -``` - -We need recovery in **both** rules. -`param` in the first rule matches the *last* param in a list, -in the second rule it matches all others. We want to recover in any position. - -If we want to be able to recovery `int x int y` as two parameters, then we -should extract a `param-and-comma` rule that recovery could operate on. - -### Last vs not-last elements - -Sequences will usually have two rules with recovery, we discussed: - - how to pick the correct one to recover with - - in a left-recursive rule they correspond to last & not-last elements - - the recovery strategy knows whether we're recoverying last or not-last - - we could have the strategy return (pos, symbol parsed), and artificially - require distinct symbols (e.g. `stmt` vs `last_stmt`). - - we can rewrite left-recursion in the grammar as right-recursion - -However, on reflection I think we can simply allow recovery according to both -rules. The "wrong" recovery will produce a parse head that dies. - -## How recovery fits into GLR - -Recovery should kick in at the point where we would otherwise abandon all -variants of an high-level parse. - -e.g. Suppose we're parsing `static_cast<foo bar baz>(1)` and are up to `bar`. -Our GSS looks something like: - -``` - "the static cast's type starts at foo" ----> {expr := static_cast < . type > ( expr )} - | "foo... is a class name" - +---- {type := identifier .} - | "foo... is a template ID" - +---- {type := identifier . < template-arg-list >} -``` - -Since `foo bar baz` isn't a valid class name or template ID, both active heads -will soon die, as will the parent GSS Node - the latter should trigger recovery. - -- we need a refcount in GSS nodes so we can recognize never-reduced node death -- when a node dies, we look up its recovery actions (symbol, strategy). - These are the union of the recovery actions for each item. - They can be stored in the action table. - Here: `actions[State, death] = Recovery(type, matching-angle-bracket)` -- we try each strategy: feeding in the start position = token of the dying node - (`foo`) getting out the end position (`>`). -- We form an opaque forest node with the correct symbol (`type`) spanning - [start, end) -- We create a GSS node to represent the state after recovery. - The new state is found in the Goto table in the usual way. - -``` - "the static cast's type starts at foo" ----> {expr := static_cast < . type > ( expr )} - | "`foo bar baz` is an unparseable type" - +---- {expr := static_cast < type . > (expr)} -``` - -## Which recovery heads to activate - -We probably shouldn't *always* create active recovery heads when a recoverable -node dies (and thus keep it alive). -By design GLR creates multiple speculative parse heads and lets incorrect heads -disappear. - -Concretely, the expression `(int *)(x)` is a valid cast, we probably shouldn't -also parse it as a call whose callee is a broken expr. - -The simplest solution is to only create recovery heads if there are no normal -heads remaining, i.e. if parsing is completely stuck. This is vulnerable if the -"wrong" parse makes slightly more progress than the "right" parse which has -better error recovery. - -A sophisticated variant might record recovery opportunities and pick the one -with the rightmost *endpoint* when the last parse head dies. - -We should consider whether including every recovery in the parse forest might -work after all - this would let disambiguation choose "broken" but likely parses -over "valid" but unlikely ones. - - diff --git a/clang-tools-extra/pseudo/Disambiguation.md b/clang-tools-extra/pseudo/Disambiguation.md deleted file mode 100644 index 39e246a523beb9..00000000000000 --- a/clang-tools-extra/pseudo/Disambiguation.md +++ /dev/null @@ -1,367 +0,0 @@ -# Disambiguation - -The C++ grammar is highly ambiguous, so the GLR parser produces a forest of -parses, represented compactly by a DAG. -A real C++ parser finds the correct parse through semantic analysis: mostly -resolving names. But we can't do that, as we don't parse the headers. - -Our disambiguation phase should take the parse forest, and choose a single parse -tree that is most likely. -It might **optionally** use some other hints (e.g. coding style, or what -specific names tend to mean in this codebase). - -There are some grammatical ambiguities that can be resolved without semantic -analysis, e.g. whether `int <declarator>{}` is a function-definition. -We eliminate these earlier e.g., with rule guards. By "disambiguation" we mean -choosing between interpretations that we can't reject confidently and locally. - -## Types of evidence - -We have limited information to go on, and strive to use similar heuristics a -human reader might. - -### Likely and unlikely structure - -In some cases, the shape of a pa... [truncated] `````````` </details> https://github.com/llvm/llvm-project/pull/109154 _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
