llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT--> @llvm/pr-subscribers-clang Author: Greg von Winckel (gregvw) <details> <summary>Changes</summary> --- Patch is 70.35 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/168420.diff 4 Files Affected: - (added) CUSTOMIZABLE_FUNCTIONS_DEV.md (+392) - (added) clang/docs/CustomizableFunctionsDesign.md (+1037) - (added) clang/docs/CustomizableFunctionsTestPlan.md (+744) - (added) custom-functions-dev.sh (+418) ``````````diff diff --git a/CUSTOMIZABLE_FUNCTIONS_DEV.md b/CUSTOMIZABLE_FUNCTIONS_DEV.md new file mode 100644 index 0000000000000..088d403cf0960 --- /dev/null +++ b/CUSTOMIZABLE_FUNCTIONS_DEV.md @@ -0,0 +1,392 @@ +# Customizable Functions Feature Development Guide + +This guide covers developing and testing the Customizable Functions feature in Clang. + +## Quick Start + +### First Time Setup + +```bash +# 1. Configure the build (debug mode, optimized for development) +./custom-functions-dev.sh configure debug + +# 2. Build clang +./custom-functions-dev.sh build + +# 3. Verify the build +./custom-functions-dev.sh info +``` + +### Development Workflow + +```bash +# Make changes to source files... + +# Build only what changed +./custom-functions-dev.sh build + +# Run customizable functions tests +./custom-functions-dev.sh test customizable + +# Or run specific test categories +./custom-functions-dev.sh test parser +./custom-functions-dev.sh test sema +./custom-functions-dev.sh test codegen +``` + +## Build Script Commands + +### Configure + +```bash +# Debug build (default - best for development) +./custom-functions-dev.sh configure debug + +# Release build (for performance testing) +./custom-functions-dev.sh configure release + +# Minimal build (fastest iteration) +./custom-functions-dev.sh configure minimal +``` + +### Build + +```bash +# Build clang +./custom-functions-dev.sh build + +# Build specific target +./custom-functions-dev.sh build check-clang-sema + +# Build with custom job count +BUILD_JOBS=8 ./custom-functions-dev.sh build +``` + +### Test + +```bash +# Run all customizable functions tests +./custom-functions-dev.sh test customizable + +# Run parser tests +./custom-functions-dev.sh test parser + +# Run semantic analysis tests +./custom-functions-dev.sh test sema + +# Run code generation tests +./custom-functions-dev.sh test codegen + +# Run AST tests +./custom-functions-dev.sh test ast + +# Run all Clang tests +./custom-functions-dev.sh test all + +# Run tests matching specific pattern +./custom-functions-dev.sh test "custom.*syntax" +``` + +### Utility Commands + +```bash +# Show build information +./custom-functions-dev.sh info + +# Clean build directory +./custom-functions-dev.sh clean + +# Rebuild from scratch +./custom-functions-dev.sh rebuild + +# Show help +./custom-functions-dev.sh help +``` + +## Build Optimization + +The build script uses several optimizations for faster development: + +- **Ninja build system**: Parallel builds with dependency tracking +- **Optimized TableGen**: Faster build times +- **Split DWARF**: Faster linking with debug info +- **Single target (X86)**: Reduces build time +- **Minimal LLVM projects**: Only builds Clang, not other tools + +### Build Times (Approximate) + +On a typical development machine (8 cores): + +- **Initial build**: 20-40 minutes (full Clang build) +- **Incremental parser change**: 30-60 seconds +- **Incremental Sema change**: 1-2 minutes +- **Incremental test-only**: 5-10 seconds + +## Development Tips + +### Fast Iteration Cycle + +1. **Use incremental builds**: The script automatically detects changes +2. **Test specific categories**: Don't run all tests every time +3. **Use minimal build mode**: For rapid prototyping + +```bash +# Example fast iteration +vim clang/lib/Parse/ParseDecl.cpp +./custom-functions-dev.sh build +./custom-functions-dev.sh test parser +``` + +### Debugging Build Issues + +```bash +# Show detailed build information +./custom-functions-dev.sh info + +# Clean and rebuild +./custom-functions-dev.sh rebuild debug + +# Verbose build output +cd build-custom-functions && ninja -v clang +``` + +### Running Individual Tests + +```bash +# Build directory contains lit tool +cd build-custom-functions + +# Run a specific test file +./bin/llvm-lit -v ../clang/test/Parser/cxx-customizable-functions.cpp + +# Run with verbose output +./bin/llvm-lit -v -a ../clang/test/SemaCXX/customizable-functions-*.cpp +``` + +### Using the Built Clang + +```bash +# Direct path to built clang +./build-custom-functions/bin/clang --version + +# Test the custom keyword (when implemented) +./build-custom-functions/bin/clang -std=c++20 -fcustomizable-functions -fsyntax-only test.cpp + +# See generated AST +./build-custom-functions/bin/clang -std=c++20 -fcustomizable-functions -Xclang -ast-dump test.cpp +``` + +## Project Structure + +``` +llvm-project/ +├── custom-functions-dev.sh # Main build script +├── CUSTOMIZABLE_FUNCTIONS_DEV.md # This file +├── build-custom-functions/ # Build output directory +│ ├── bin/clang # Built clang binary +│ └── compile_commands.json # For IDE integration +├── clang/ +│ ├── docs/ +│ │ ├── CustomizableFunctionsDesign.md # Design document +│ │ └── CustomizableFunctionsTestPlan.md # Test plan +│ ├── include/clang/ +│ │ ├── Basic/ +│ │ │ ├── TokenKinds.def # Add 'custom' keyword here +│ │ │ └── LangOptions.def # Add language option here +│ │ ├── Parse/ +│ │ │ └── Parser.h # Parser interface +│ │ ├── Sema/ +│ │ │ ├── Sema.h # Semantic analysis interface +│ │ │ └── SemaCustomFunction.h # Custom function transformation (new) +│ │ └── AST/ +│ │ └── Decl.h # AST node declarations +│ ├── lib/ +│ │ ├── Parse/ +│ │ │ └── ParseDecl.cpp # Parse 'custom' keyword +│ │ ├── Sema/ +│ │ │ ├── SemaDecl.cpp # Semantic analysis +│ │ │ └── SemaCustomFunction.cpp # Transform logic (new) +│ │ └── AST/ +│ │ └── Decl.cpp # AST implementation +│ └── test/ +│ ├── Parser/ +│ │ └── cxx-customizable-functions-*.cpp +│ ├── SemaCXX/ +│ │ └── customizable-functions-*.cpp +│ ├── CodeGenCXX/ +│ │ └── customizable-functions-*.cpp +│ └── AST/ +│ └── customizable-functions-*.cpp +``` + +## Common Development Tasks + +### Adding a New Keyword + +1. Add to `clang/include/clang/Basic/TokenKinds.def` +2. Rebuild: `./custom-functions-dev.sh build` +3. Test: `./custom-functions-dev.sh test parser` + +### Adding Parser Support + +1. Modify `clang/lib/Parse/ParseDecl.cpp` +2. Add tests in `clang/test/Parser/` +3. Build and test: + ```bash + ./custom-functions-dev.sh build + ./custom-functions-dev.sh test parser + ``` + +### Adding Semantic Analysis + +1. Create `clang/lib/Sema/SemaCustomFunction.cpp` +2. Add hook in `clang/lib/Sema/SemaDecl.cpp` +3. Add tests in `clang/test/SemaCXX/` +4. Build and test: + ```bash + ./custom-functions-dev.sh build + ./custom-functions-dev.sh test sema + ``` + +### Adding Code Generation + +1. Modify `clang/lib/CodeGen/CGDecl.cpp` +2. Add tests in `clang/test/CodeGenCXX/` +3. Build and test: + ```bash + ./custom-functions-dev.sh build + ./custom-functions-dev.sh test codegen + ``` + +## IDE Integration + +### Compile Commands (for clangd, CLion, etc.) + +The build script automatically creates a symlink to `compile_commands.json`: + +```bash +llvm-project/compile_commands.json -> build-custom-functions/compile_commands.json +``` + +This enables IDE features like: +- Code completion +- Jump to definition +- Error highlighting +- Refactoring tools + +### VS Code + +Install the clangd extension and it will automatically find the compile commands. + +### CLion + +CLion will detect the CMake project automatically. Point it to `build-custom-functions/`. + +## Testing Strategy + +### Test Categories + +1. **Parser Tests**: Syntax validation + - `./custom-functions-dev.sh test parser` + +2. **Semantic Tests**: Type checking, constraints + - `./custom-functions-dev.sh test sema` + +3. **CodeGen Tests**: LLVM IR generation + - `./custom-functions-dev.sh test codegen` + +4. **AST Tests**: AST structure verification + - `./custom-functions-dev.sh test ast` + +5. **Integration Tests**: End-to-end workflows + - `./custom-functions-dev.sh test customizable` + +### Writing New Tests + +Create test files following this pattern: + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// Test case description +custom void my_test() { } // expected-no-diagnostics + +// Or with expected errors +custom void bad_test(); // expected-error {{custom functions must have a body}} +``` + +Place in appropriate directory: +- Parser tests: `clang/test/Parser/` +- Semantic tests: `clang/test/SemaCXX/` +- CodeGen tests: `clang/test/CodeGenCXX/` + +## Troubleshooting + +### Build Fails + +```bash +# Clean and rebuild +./custom-functions-dev.sh rebuild + +# Check for CMake issues +cd build-custom-functions +cmake .. + +# Verbose build to see errors +ninja -v clang +``` + +### Tests Fail + +```bash +# Run specific test with verbose output +cd build-custom-functions +./bin/llvm-lit -v ../clang/test/Parser/cxx-customizable-functions.cpp -a + +# Check test expectations +cat ../clang/test/Parser/cxx-customizable-functions.cpp +``` + +### Performance Issues + +```bash +# Use release build for performance testing +./custom-functions-dev.sh configure release +./custom-functions-dev.sh build + +# Increase parallel jobs (if you have RAM) +BUILD_JOBS=16 ./custom-functions-dev.sh build +``` + +## Environment Variables + +- `BUILD_JOBS`: Number of parallel build jobs (default: nproc) +- `BUILD_TYPE`: Override build type (Debug/Release) +- `CC`: C compiler to use +- `CXX`: C++ compiler to use + +Example: +```bash +export BUILD_JOBS=12 +export CC=clang +export CXX=clang++ +./custom-functions-dev.sh configure debug +``` + +## References + +- [Design Document](clang/docs/CustomizableFunctionsDesign.md) +- [Test Plan](clang/docs/CustomizableFunctionsTestPlan.md) +- [LLVM Testing Infrastructure](https://llvm.org/docs/TestingGuide.html) +- [Clang Internals Manual](https://clang.llvm.org/docs/InternalsManual.html) + +## Next Steps + +Once the build is configured: + +1. Review the design documents +2. Start with Phase 1: Add keyword and basic parsing +3. Add tests incrementally +4. Iterate quickly with the build script + +Happy coding! diff --git a/clang/docs/CustomizableFunctionsDesign.md b/clang/docs/CustomizableFunctionsDesign.md new file mode 100644 index 0000000000000..e11d7c794c819 --- /dev/null +++ b/clang/docs/CustomizableFunctionsDesign.md @@ -0,0 +1,1037 @@ +# Customizable Functions Design Document + +**Author:** TBD +**Date:** 2025-11-17 +**Status:** Draft + +## Table of Contents + +1. [Introduction](#introduction) +2. [Motivation](#motivation) +3. [Proposed Syntax](#proposed-syntax) +4. [Semantics](#semantics) +5. [Implementation Plan](#implementation-plan) +6. [Testing Strategy](#testing-strategy) +7. [Open Questions](#open-questions) +8. [References](#references) + +--- + +## Introduction + +This document proposes adding a new language feature to Clang: **Customizable Functions** using the `custom` keyword. This feature would allow library authors to easily create Customization Point Objects (CPOs) using the tag_invoke pattern without manually writing extensive boilerplate code. + +### Goals + +- Reduce boilerplate for implementing the tag_invoke CPO pattern +- Improve library interface design and customization points +- Maintain compatibility with existing C++20/23 code +- Generate efficient, zero-overhead abstractions + +### Non-Goals + +- Replacing existing customization mechanisms (ADL, specialization) +- Changing the behavior of tag_invoke itself +- Adding runtime dispatch overhead + +--- + +## Motivation + +### Current State: Manual CPO Implementation + +Library authors currently need to write significant boilerplate to create proper CPOs: + +```cpp +namespace my_lib { + namespace CPO_DETAIL { + // Default implementation + void do_thing(auto& t) { t.method(); } + + // Functor class + struct do_thing_fn { + // tag_invoke overload + template <typename T> + constexpr auto operator()(T& t) const + requires requires { tag_invoke(*this, t); } + { + return tag_invoke(*this, t); + } + + // Fallback overload + template <typename T> + constexpr auto operator()(T& t) const + requires (!requires { tag_invoke(*this, t); }) + { + return do_thing(t); + } + }; + } + + // Inline constexpr instance + inline constexpr CPO_DETAIL::do_thing_fn do_thing{}; +} +``` + +This pattern: +- Is verbose and error-prone +- Requires understanding of advanced C++ techniques +- Must be repeated for each customization point +- Is difficult to maintain and modify + +### Proposed State: Declarative Syntax + +With customizable functions, the same functionality becomes: + +```cpp +namespace my_lib { + custom void do_thing(auto& t) { + t.method(); + } +} +``` + +The compiler automatically generates the CPO boilerplate, making the code: +- More concise and readable +- Less error-prone +- Easier to maintain +- Self-documenting + +### Real-World Use Cases + +1. **Generic Algorithm Libraries**: Customization points for ranges, algorithms +2. **Serialization/Deserialization**: Custom serializers for user types +3. **Logging/Debugging**: Customizable formatting and output +4. **Resource Management**: Custom allocators, deleters +5. **Async/Await Patterns**: Customizable awaitable operations + +--- + +## Proposed Syntax + +### Basic Syntax + +```cpp +custom <return-type> <function-name>(<parameters>) <body> +``` + +### Examples + +#### Simple Free Function +```cpp +namespace lib { + custom void print(auto const& value) { + std::cout << value; + } +} +``` + +#### With Explicit Return Type +```cpp +custom int compute(int x, int y) { + return x + y; +} +``` + +#### Multiple Parameters +```cpp +custom void transform(auto& container, auto&& func) { + for (auto& elem : container) { + func(elem); + } +} +``` + +#### With Constraints +```cpp +custom void process(auto& t) + requires std::copyable<decltype(t)> +{ + t.process(); +} +``` + +#### Template Function +```cpp +template <typename T> +custom void serialize(T const& value, std::ostream& out) { + out << value; +} +``` + +### Syntax Restrictions + +- `custom` keyword must appear before the return type +- Cannot be used with: + - Member functions (initially; may be relaxed later) + - Constructors/destructors + - Operators (except when implementing operator() for the generated functor) + - Virtual functions +- Must have a function body (not just a declaration) +- Can be used in any namespace (including global) + +--- + +## Semantics + +### Transformation Overview + +When the compiler encounters a `custom` function, it performs the following transformation: + +1. **Create Detail Namespace** (optional, configurable) + - Named `CPO_DETAIL` or `<function_name>_detail` + - Contains the default implementation and functor class + +2. **Generate Default Implementation** + - Original function body becomes a hidden implementation function + - Used as the fallback when tag_invoke is not available + +3. **Generate Functor Class** + - Named `<function_name>_fn` + - Implements two operator() overloads: + - Primary: calls tag_invoke (when available) + - Fallback: calls default implementation + +4. **Create CPO Instance** + - Inline constexpr variable with original function name + - Type is the functor class + - This becomes the actual customization point + +### Generated Code Structure + +For a `custom` function: +```cpp +custom RetType func_name(Params...) { body } +``` + +The compiler generates: +```cpp +namespace CPO_DETAIL { + // Default implementation (hidden) + RetType func_name_impl(Params...) { body } + + // Functor class + struct func_name_fn { + // Primary overload: use tag_invoke if available + template <typename... Args> + constexpr auto operator()(Args&&... args) const + noexcept(noexcept(tag_invoke(*this, std::forward<Args>(args)...))) + requires requires { tag_invoke(*this, std::forward<Args>(args)...); } + { + return tag_invoke(*this, std::forward<Args>(args)...); + } + + // Fallback overload: use default implementation + template <typename... Args> + constexpr auto operator()(Args&&... args) const + noexcept(noexcept(func_name_impl(std::forward<Args>(args)...))) + requires (!requires { tag_invoke(*this, std::forward<Args>(args)...); } + && requires { func_name_impl(std::forward<Args>(args)...); }) + { + return func_name_impl(std::forward<Args>(args)...); + } + }; +} + +// The actual CPO +inline constexpr CPO_DETAIL::func_name_fn func_name{}; +``` + +### Name Lookup and ADL + +The generated CPO follows standard C++ name lookup rules: + +1. **Unqualified calls** to the CPO trigger ADL +2. **tag_invoke** is found via ADL in the namespace of the arguments +3. Users customize by defining `tag_invoke` in their namespace: + +```cpp +namespace user { + struct MyType { }; + + // Customization + void tag_invoke(lib::do_thing_fn, MyType& t) { + // Custom implementation + } +} + +// Usage +user::MyType obj; +lib::do_thing(obj); // Finds user::tag_invoke via ADL +``` + +### Overload Resolution + +The two operator() overloads in the functor are constrained to be mutually exclusive: +- Primary: `requires tag_invoke(*this, args...)` +- Fallback: `requires !tag_invoke(*this, args...) && default_impl(args...)` + +This ensures: +- No ambiguity during overload resolution +- tag_invoke always preferred when available +- Fallback only selected when tag_invoke is not viable + +### Template Instantiation + +For templated custom functions: +```cpp +template <typename T> +custom void process(T& value) { ... } +``` + +The functor class itself is templated on the CPO's template parameters, and the operator() remains templated on the call-site arguments. + +--- + +## Implementation Plan + +### Phase 1: Core Infrastructure (Minimal Viable Product) + +**Goal**: Get basic `custom` keyword parsing and simple code generation working. + +#### 1.1 Add Keyword and Language Option + +**Files**: +- `clang/include/clang/Basic/TokenKinds.def` +- `clang/include/clang/Basic/LangOptions.def` + +**Tasks**: +- [ ] Add `custom` as a C++20 keyword with flag `KEYCUSTOMFN` +- [ ] Add language option `CustomizableFunctions` (default: disabled) +- [ ] Add `-fcustomizable-functions` / `-fno-customizable-functions` flags + +**Testing**: +- [ ] Verify keyword is recognized when feature is enabled +- [ ] Verify keyword is not recognized when feature is disabled + +#### 1.2 Extend Parser + +**Files**: +- `clang/include/clang/Parse/Parser.h` +- `clang/lib/Parse/ParseDecl.cpp` +- `clang/include/clang/Sema/DeclSpec.h` + +**Tasks**: +- [ ] Add `isCustomFunction` flag to `DeclSpec` +- [ ] Modify `ParseDeclarationSpecifiers` to recognize `custom` keyword +- [ ] Set the flag when `custom` is encountered +- [ ] Ensure proper error handling for invalid uses + +**Testing**: +- [ ] Parse simple `custom void foo() {}` +- [ ] Reject `custom` on member functions +- [ ] Reject `custom` on declarations without definitions +- [ ] Proper error messages for invalid syntax + +#### 1.3 Extend AST + +**Files**: +- `clang/include/clang/AST/Decl.h` +- `clang/lib/AST/Decl.cpp` +- `clang/lib/AST/DeclPrinter.cpp` +- `clang/lib/AST/ASTDumper.cpp` + +**Tasks**: +- [ ] Add `IsCustomFunction` bit to `FunctionDeclBitfields` +- [ ] Add `isCustomFunction()` / `setCustomFunction()` methods +- [ ] Update `DeclPrinter` to print `custom` keyword +- [ ] Update `ASTDumper` to show custom function flag + +**Testing**: +- [ ] AST dump shows custom function annotation +- [ ] AST printer reproduces `custom` keyword + +#### 1.4 Basic Semantic Analysis + +**Files**: +- `clang/lib/Sema/SemaDecl.cpp` +- `clang/include/clang/Sema/Sema.h` + +**Tasks**: +- [ ] Create `ActOnCustomFunctionDecl()` hook +- [ ] Validate custom function constraints: + - Must have a body + - Cannot be member function + - Cannot be virtual + - Cannot be main() +- [ ] Mark function as custom in AST + +**Testing**: +- [ ] Semantic errors for invalid custom functions +- [ ] Accept valid custom function declarations + +### Phase 2: Code Generation (Core Transformation) + +**Goal**: Generate the CPO boilerplate during Sema. + +###... [truncated] `````````` </details> https://github.com/llvm/llvm-project/pull/168420 _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
