adityamparikh opened a new pull request, #264: URL: https://github.com/apache/solr-mcp/pull/264
Part of #256. Adds the harness for the opt-in LLM tool-selection evals and the `./gradlew eval` task. No scenarios yet; the first one is in the stacked follow-up. ## What it does - **`./gradlew eval`** runs tests tagged `llm-eval`. Every other `Test` task excludes that tag, so `./gradlew build` never calls a model. `eval` is never up to date and doesn't finalize `jacocoTestReport`, which would otherwise pull in `test`. - **The agent** is `spring-ai-starter-model-openai` on the test classpath, with Spring AI's defaults (`gpt-5-mini`, key from `OPENAI_API_KEY`). Nothing is configured. Switching to llm.apache.org later means setting `spring.ai.openai.base-url` and `spring.ai.openai.chat.model`. - **`EvalServer`** starts the built jar over STDIO against a Testcontainers Solr and loads `mydata/devnexus-2026.json` (116 sessions) into `devnexus` through the server's own tools. It launches the test JVM's `java` (the build toolchain), because the `java` on `PATH` can be older than 25. - **`EvalAgent`** sends one prompt with the server's instructions as the system prompt and its tools under their own names (`McpToolNamePrefixGenerator.noPrefix()`). Each tool is wrapped in a `RecordingToolCallback`, so a run returns every call's name, input and output, plus the answer. - **`EvalConfiguration`** is a `@TestConfiguration` with `@EnableAutoConfiguration`, so `Main`'s component scan never picks it up. The `integration-test` profile sets `spring.ai.mcp.server.enabled=false`, so auto-configuration doesn't start an MCP server on the test JVM's stdin and stdout. - **`EvalHarnessEval`** checks the harness without calling a model: all 116 sessions are indexed, and the model sees the server's tool names and instructions. - **`RecordingToolCallbackTest`** is a unit test that runs in `./gradlew build`. The OpenAI starter is a test-only dependency, so the shipped classpath, the SBOM and the generated LICENSE are unaffected. ## Verification - `./gradlew build`: 468 tests, 0 skipped, 0 failed (with JDK 25 first on `PATH` for `McpClientStdioIntegrationTest`). - `./gradlew eval`: `EvalHarnessEval` 3/3. - `./gradlew eval --dry-run` runs `bootJar` and `eval`, not `test`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
