mattcasters opened a new pull request, #7784: URL: https://github.com/apache/hop/pull/7784
## Summary Fixes [#7783](https://github.com/apache/hop/issues/7783): the File Explorer filter became unusable with a few hundred–thousand files (multi-second freezes per keystroke). The same tree rebuild path also made opening a large flat project root expensive. Two related problems were fixed: ### 1. Filter keystrokes (original issue) - **Debounce** filter apply (~250 ms; Enter applies immediately) - **In-memory `ExplorerTreeModel`**: one full VFS walk, then subsequent filter keystrokes match/render from memory only - Filter rebuilds **do not** call `IExplorerRefreshListener.beforeRefresh()` (no git status per character) - Removed recursive **`hasMatchingDescendant`** VFS probes (the O(n×depth) double-walk) - Explorer filter uses non-fuzzy `SearchMatcher` (substring/word only) ### 2. Project open / tree paint cost (same root causes) - **`HopFileTypeBase.isHandledBy`**: extension match is pure string — no `HopVfs.getFileObject` per plugin per file - **Extension/basename maps** in `ExplorerPerspective` for O(1) type resolution; folders take `FolderFileType` via known `isFolder` flag - **`FolderFileType` / `GenericFileType`**: no VFS `isFolder`/`isFile` probes during type resolution - Same string-extension approach for Archive, Excel, Parquet, NoExtension types - **Single-pass** folder listing (classify once, then sort folders/files) - Git paint path normalization without VFS (when git plugin is enabled) Measured on a ~10k-file project root (git disabled): open dropped from ~10s to ~6s (~40%). Filter typing is interactive after the first index build. ## Reviewer notes Please focus review on these areas: 1. **`ExplorerPerspective` filter path** - `scheduleFilterApply` / `applyPendingFilter` / `applyFilterFromModel` / `ensureFullTreeModel` - Confirm F5 still refreshes git status while filtering, but normal keystrokes do not - Confirm clearing the filter restores expand state via `saveTreeState` / `restoreTreeState` - Index invalidation: `treeModel.clear()` on structural `refresh()`, show-hidden, create/delete/rename (via `refresh()`) 2. **`getFileType(path, isFolder)` + `indexFileTypes()`** - First matching extension wins (plugin order). Flag any type that used non-extension logic that no longer maps cleanly - `FolderFileType.isHandledBy` now always returns `false` — callers must pass `isFolder=true` or select the type explicitly. Search for other callers if concerned - `GenericFileType` is the catch-all for unmatched files without VFS 3. **`HopFileTypeBase.extractExtension` / `extractBaseName`** - Hidden files like `.gitignore` intentionally have no “extension” - Compound filters (`*.xls;*.xlsx`) are split on `;` 4. **Git `GitGuiPlugin.getAbsoluteFilename`** - Keys must stay consistent between `refreshChangedFiles` and `filePainted` (both use the same string normalizer now) - Worth a quick smoke test: git project with staged/modified files still get color in the explorer 5. **Out of scope / known remaining cost** - Creating many SWT `TreeItem`s for a flat 10k root is still inherently heavy; this PR does not virtualize the tree - First filter after open may still walk the full tree once to fill the index (by design) ## Test plan - [x] Unit: `HopFileTypeBaseTest`, `ExplorerTreeModelTest` - [ ] Manual: project with ~1k+ files — type in explorer filter; characters should not freeze UI; matches under nested folders appear after debounce - [ ] Manual: clear filter — previous expand/collapse state restored - [ ] Manual: F5 while filtering — tree updates and git colors (if git enabled) refresh - [ ] Manual: open large flat project — tree populates without multi-10s freezes - [ ] Manual: open pipeline/workflow/text files from explorer still uses correct handlers/icons - [ ] Manual (git): modified/staged/ignored files still colored after explorer refresh Fixes #7783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
