[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-24 Thread Kirill Bobyrev via Phabricator via cfe-commits
kbobyrev added inline comments. Comment at: clang-tools-extra/clangd/index/dex/Trigram.cpp:56 + for (int I = LowercaseIdentifier.size() - 1; I >= 0; --I) { +Next[I] = {{NextTail, NextHead, NextNextHead}}; +NextTail = Roles[I] == Tail ? I : 0; sammccall w

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-24 Thread Kirill Bobyrev via Phabricator via cfe-commits
kbobyrev updated this revision to Diff 157196. kbobyrev marked 7 inline comments as done. kbobyrev added a comment. Address last round of comments. https://reviews.llvm.org/D49591 Files: clang-tools-extra/clangd/CMakeLists.txt clang-tools-extra/clangd/index/dex/Token.h clang-tools-extra/c

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-24 Thread Sam McCall via Phabricator via cfe-commits
sammccall accepted this revision. sammccall added a comment. This revision is now accepted and ready to land. Thanks for the polish! Last few nits, but feel free to land once you're happy. Comment at: clang-tools-extra/clangd/index/dex/Token.cpp:21 + +Token::Token(Kind TokenKin

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-24 Thread Kirill Bobyrev via Phabricator via cfe-commits
kbobyrev added inline comments. Comment at: clang-tools-extra/clangd/index/dex/Trigram.cpp:38 + llvm::makeMutableArrayRef(Roles.data(), Identifier.size())); + std::transform(begin(Identifier), end(Identifier), begin(Identifier), + ::tolower);

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-24 Thread Kirill Bobyrev via Phabricator via cfe-commits
kbobyrev updated this revision to Diff 157020. kbobyrev marked 12 inline comments as done. kbobyrev added a comment. Address all comments, refine tests. https://reviews.llvm.org/D49591 Files: clang-tools-extra/clangd/CMakeLists.txt clang-tools-extra/clangd/index/dex/Token.cpp clang-tools-

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-23 Thread Sam McCall via Phabricator via cfe-commits
sammccall added a comment. Looks really nice! Only major issue is the query trigrams don't look right. Otherwise, some style nits and fixes that seem to have gotten lost. Comment at: clang-tools-extra/clangd/index/dex/Token.cpp:28 + +llvm::StringRef Token::data() const { return

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-23 Thread Kirill Bobyrev via Phabricator via cfe-commits
kbobyrev updated this revision to Diff 156774. kbobyrev added a comment. Wrong diff, should be correct now. https://reviews.llvm.org/D49591 Files: clang-tools-extra/clangd/CMakeLists.txt clang-tools-extra/clangd/index/dex/Token.cpp clang-tools-extra/clangd/index/dex/Token.h clang-tools-

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-23 Thread Kirill Bobyrev via Phabricator via cfe-commits
kbobyrev updated this revision to Diff 156772. kbobyrev marked 19 inline comments as done. kbobyrev added a reviewer: ilya-biryukov. kbobyrev removed a subscriber: ilya-biryukov. kbobyrev added a comment. Address all comments from the last iteration. Latest changes: - Trigram generation algorith

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-20 Thread Sam McCall via Phabricator via cfe-commits
sammccall added inline comments. Comment at: clang-tools-extra/clangd/index/dex/Trigram.cpp:37 + + // Extract trigrams consisting of first characters of tokens sorted bytoken + // positions. Trigram generator is allowed to skip 1 word between each token. sammcc

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-20 Thread Sam McCall via Phabricator via cfe-commits
sammccall added inline comments. Comment at: clang-tools-extra/clangd/index/dex/Token.cpp:25 + Data.size() == 3 && "Trigram should contain three characters."); + switch (TokenKind) { + case Kind::Trigram: specializing the hash function looks like premat

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-20 Thread Kirill Bobyrev via Phabricator via cfe-commits
kbobyrev planned changes to this revision. kbobyrev added a comment. The upcoming changes: - Use segmentation API exposed in https://reviews.llvm.org/rL337527 - Create a separate structure for Data and Hash as suggested by Sam - Fix the bug with whole segments concatenation https://reviews.llvm

[PATCH] D49591: [clangd] Introduce search Tokens and trigram generation algorithms

2018-07-20 Thread Kirill Bobyrev via Phabricator via cfe-commits
kbobyrev created this revision. kbobyrev added reviewers: ioeric, sammccall. kbobyrev added a project: clang-tools-extra. Herald added subscribers: arphaman, jkorous, MaskRay, mgorny. This patch introduces trigram generation algorithm for the symbol index proposed in a recent design document. RFC