YanhaoLv opened a new pull request, #369:
URL: https://github.com/apache/hugegraph-ai/pull/369
## Summary
Adds an opt-in **enhanced** strategy for `/graph/extract` that applies
a schema-aware quality layer on top of the existing LLM-based
extraction. The current pipeline (renamed here as **baseline**) stays
byte-compatible: enhanced only runs when the caller sends
`extract_strategy: "enhanced"`.
The new layer runs a candidate parser → schema-aware normalizer →
cross-chunk document assembler → quality gate on every LLM response,
producing structured warnings + a quality-metrics block alongside the
usual `vertices` / `edges`.
Closes #74.
## Effect
Two independent evaluation tracks — the mock rubric benchmark is for
*coverage* of every rubric edge case; the live public-corpus benchmark
is for *effect evidence* under externally-authored ground truth.
| Track | Baseline F1 | Enhanced F1 | Delta |
|---|---:|---:|---:|
| Mock rubric benchmark (16 scenarios, FakeLLM, deterministic) | 0.86 | 0.96
| **+11.6 % rel.** |
| Live DeepSeek on public corpus (8 actors, 3 runs each) | 0.418 ± 0.251 |
0.439 ± 0.188 | **+4.8 % rel.** |
**Design-stage threshold** (baked into a CI assertion inside the mock
benchmark test): enhanced ≥ +5 % relative F1 on the 16-scenario rubric
average. Regressions below this floor fail CI.
**Live-track secondary metrics** (across all 48 runs):
| Metric | Baseline | Enhanced | Delta |
|---|---:|---:|---:|
| F1 std (variance across runs and corpora) | 0.251 | 0.188 | **−25 %** |
| Wall-clock latency mean per document | 16.57 s | 14.80 s | **−10.7 %** |
| Cost per document mean (DeepSeek, cache-miss) | $0.003776 | $0.003508 |
**−7.1 %** |
| Corpora where enhanced ≥ baseline | — | — | 5 / 8 |
| Corpora where enhanced regresses | — | — | 3 / 8 |
**Interpretation.** Enhanced is a safety-net strategy: it rescues the
worst cases (Tom Hanks corpus F1: 0.052 → 0.327) and dramatically
narrows the F1 variance (−25 %) while sending +40 % prompt tokens but
producing −24 % completion tokens. Net effect: modest average F1 gain,
much more predictable outcomes, slightly cheaper and faster. It is not
a strict Pareto improvement — on 3 / 8 corpora where baseline was
already strong, enhanced's stricter dedup merges near-duplicate film
titles and regresses F1 by 0.10-0.20. This is documented in the report,
not glossed over.
Full analysis, per-corpus tables, and concrete failure cases (including
the Wikipedia "Thomas Jeffrey Hanks" ↔ Wikidata "Tom Hanks" canonical
name mismatch that hits both strategies) in
[docs/quality/schema-based-graph-extract-report.md](docs/quality/schema-based-graph-extract-report.md).
## Backwards Compatibility
**None broken.** Two new optional request fields
(`extract_strategy`, `include_debug`, both defaulted) and enhanced-only
`meta` keys that appear *only* when the caller opts in. Baseline
responses are byte-identical to the pre-PR shape.
## How to Enable
```json
POST /graph/extract
{
"texts": ["..."],
"graph_schema": "...",
"extract_strategy": "enhanced",
"include_meta": true
}
```
Full usage guide (API + Python + limitations) at
[docs/quality/schema-based-graph-extract-usage.md](docs/quality/schema-based-graph-extract-usage.md).
## What's in the box
**New subpackage**
[hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/](hugegraph-llm/src/hugegraph_llm/operators/llm_op/property_graph_extract_enhanced/):
- `schema_index.py` — compiled runtime schema index (label lookup,
canonical id, property coercion).
- `candidate_parser.py` — tolerant JSON extractor covering 5
LLM-output formats.
- `normalizer.py` — per-chunk schema-aware normalizer (property
filter → coerce → PK check → canonical id).
- `document_assembler.py` — cross-chunk alias-table union with
ambiguity detection and endpoint repair.
- `quality_gate.py` — 11-metric quality aggregate per run.
- `warnings.py` — structured warning registry (20 codes).
- `evaluator.py` — offline evaluator (precision/recall/F1 + property
fidelity metrics).
- `prompt_contract.py` — schema-derived constraint block that
enhanced appends to the LLM prompt.
**Threaded through** the existing pipeline:
`api/graph_extract_api.py`, `flows/graph_extract.py`,
`operators/llm_op/property_graph_extract.py`, `state/ai_state.py`,
`nodes/llm_node/extract_info.py`.
**Public-corpus builder + live benchmark driver** (outside CI, network
required):
- `scripts/build_public_actor_corpus.py` — fetches Wikipedia lead
extracts + verifies GT against Wikidata `P161 (cast member)` claims
via the entity API. Output pinned to
`hugegraph-llm/src/tests/data/public_actor_corpus.json`.
- `scripts/graph_extract_live_benchmark.py` — accepts `--corpus`
(required, no hand-authored fallback) and `--runs`; produces the
mean ± std, per-corpus, per-strategy JSON archive.
- `docs/quality/data/live_benchmark_public_actors.json` — the full
48-run archive from the reported live benchmark, committed for
reviewer cross-checking. Every number in the effect report's live
section is `jq`-derivable from this file without re-running the
benchmark. See the report's "How to Reproduce This Report" section
for exact queries.
**Tests** (all `unit or contract`, deterministic):
- `test_property_graph_schema.py`
- `test_property_graph_warnings.py`
- `test_property_graph_postprocess.py` (parser + normalizer + assembler)
- `test_property_graph_quality_gate.py`
- `test_property_graph_evaluator.py`
- `test_property_graph_benchmark.py` (16 rubric scenarios, incl.
s15/s16 domain-limit cases where enhanced ≡ baseline)
- `test_property_graph_extract.py` (extended for enhanced dispatch)
- `test_graph_extract_api.py` (extended for API-layer meta)
## Test Plan
- [x] 398 unit/contract tests pass locally.
- [x] Enhanced package coverage: 95 % (evaluator 94 %, normalizer 96 %,
parser 97 %, assembler 90 %, quality_gate 100 %, warnings 100 %).
- [x] Full repo coverage 46.31 % on the CI-equivalent
`-m "unit or contract"` filter, well above the 34 % CI floor.
- [x] Ruff check + ruff format — clean.
- [x] Baseline byte-compatibility: dedicated API-level assertion that
baseline meta remains `{vertex_count, edge_count, text_count}`.
- [x] Design invariant asserted by the benchmark test: enhanced F1
≥ baseline F1 on every scenario.
- [x] Design-stage threshold asserted in the same test: mock benchmark
avg F1 gain ≥ +5 % relative. Actual: +11.6 %.
- [x] Live DeepSeek run recorded end-to-end (F1 mean/std/min/max,
latency, tokens, cost) for all 8 corpora × 3 runs = 48 samples.
## Scope Notes
- **Person canonical name mismatch** hits both strategies on all live
corpora and is the biggest single F1 loss source. Wikipedia leads
say "Thomas Jeffrey Hanks"; Wikidata canonicals say "Tom Hanks". No
schema-only pipeline can bridge that without either LLM alignment or
an entity-resolution step. Both are out of scope for Issue #74. This
is documented as a known limitation in the usage doc, not hidden.
- The `token_usage` field in `meta` remains a placeholder pending an
LLM adapter change to surface token metadata through `BaseLLM`. The
live benchmark script tracks tokens by calling the LiteLLM completion
API directly.
- Set-based F1 was chosen so that baseline's cross-chunk vertex
duplication doesn't get double-counted; raw counts are still exposed
in `ItemMetrics.predicted_count_raw` for downstream commit-load
monitoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
