HY-love-sleep opened a new issue, #7623:
URL: https://github.com/apache/shenyu/issues/7623
## Description
ShenYu can already reject a prompt containing a forbidden word:
`ai-sensitive-word` (#7153, registered
in #7155) compiles a rule dictionary into an Aho–Corasick automaton and
inspects the request body. That
covers deterministic keyword compliance. It cannot cover what the AI gateway
space has converged on: a
*judgement* about the meaning of a prompt or of a model response.
Commercial gateways ship exactly that:
- Apache APISIX — `ai-aliyun-content-moderation` /
`ai-aws-content-moderation`: `check_request` /
`check_response`, `stream_check_mode: realtime | final_packet`,
`risk_level_bar`, `deny_code` /
`deny_message`, `fail_mode`.
- Higress — `ai-security-guard`: `checkRequest` / `checkResponse`, JSONPath
extraction per direction,
`denyCode`, `ai_sec_request_deny` / `ai_sec_response_deny` metrics.
All of them hard-code one moderation vendor (one plugin per provider). The
gap this issue proposes to
fill is a **judge-agnostic moderation plugin: the decision comes from an
extension point, so a
self-hosted judge model and a commercial moderation API are the same thing
to the gateway.**
That is practical now: Qwen3Guard-Gen / Qwen3Guard-Stream, Llama Guard and
ShieldGemma can all be
served behind an OpenAI-compatible `/v1/chat/completions` endpoint (vLLM,
SGLang, Ollama), while a
vendor API is a thin adapter away. One OpenAI-compatible judge
implementation therefore covers both,
with no vendor SDK inside the gateway.
**Scope of this issue: design agreement first.** I deliberately do not
include the admin registration
rows or the console form here — I would like the behaviour and the extension
point agreed before
generating tens of rows per dialect for six tables.
## Design
### 1. Judge extension point
`ContentModerationJudge` is the only contract: given text plus the rule
handle, return a verdict. Two
implementations ship with the plugin:
- `openAiCompatibleJudge` — any OpenAI-compatible chat endpoint (self-hosted
guard model, or a hosted
service speaking the same shape);
- `httpApiJudge` — a vendor moderation HTTP API driven by path/template
fields, no vendor SDK.
Model, prompt template, threshold, timeout and action live in the rule
handle, so the *policy* is
configuration rather than code (precedent: LiteLLM's `decision_model`
guardrail, where the operator
writes natural-language questions, each with its own threshold and action).
### 2. Where the checks run, and plugin order
Two sides, independently configurable: `checkRequest` (before the prompt
leaves) and `checkResponse`
(response content before the client sees it).
The response side requires an order **lower than** `aiProxy` (199), because
a response decorator only
wraps responses produced by plugins ordered after it:
```
aiPrompt 170 · aiTokenLimiter 171 · … · [196] · sensitiveWord 197 ·
agentGateway 198 · aiProxy 199 · divide 200 · modifyResponse 220
```
`196` is free. There is one real ordering question: the cheap dictionary
check (`sensitiveWord`, 197)
and this expensive judge cannot both sit immediately before `aiProxy`, and
equal order values make the
chain order nondeterministic.
### 3. Verdict model — four states, per-rule action
Not two booleans. Following LiteLLM's `guardrail_status`:
| state | meaning | behaviour |
| --- | --- | --- |
| `PASS` | judged safe | continue |
| `FLAGGED` | risky, rule action = `log` | continue, record |
| `BLOCKED` | risky, rule action = `block` | reject |
| `UNAVAILABLE` | judge failed, timed out or returned a malformed result |
per rule: `failClosed` |
This also borrows Bedrock Guardrails' per-rule `action: BLOCK | NONE`:
"detect without blocking" is a
first-class setting, which is what makes an observe-then-enforce rollout
possible. Keeping `UNAVAILABLE`
distinct matters — otherwise "the judge was down" and "the content was
clean" are indistinguishable in
the metrics.
### 4. Streaming semantics
Three modes. The plugin must never claim more than it can do:
| `streamCheckMode` | behaviour | prevents leakage? | cost |
| --- | --- | --- | --- |
| `buffered` (proposed default) | hold the response back until the assembled
content up to `holdBackChars` has a verdict; flush at end of stream | yes, to
the judge's coverage | time-to-first-token = judge latency |
| `realtime` | judge each batch; a batch's bytes are released only after
that batch's own verdict | per batch; cross-batch content still covered by
sending the window as context | one judge call per batch |
| `final_packet` | judge the assembled response at the end; annotate only |
no — audit only | none |
The rule that a previous prototype of mine got wrong, and which I want
stated explicitly in the review:
**a batch must never be released without its own verdict.** Judging an
accumulated window and then
releasing the current batch unchecked is retrospective detection, not
pre-release moderation. In
`realtime` mode the judge receives the window (unreleased content plus the
current batch) as context,
but the gate is always the current batch.
`realtime` / `final_packet` follow APISIX's `stream_check_mode` names and
caveats; `buffered` is
LiteLLM's shipped default for its built-in guardrails ("buffers every chunk
until the assembled
response passes moderation"), which is why I propose it as ours. The README
and the rule form will
state plainly that in `realtime` / `final_packet` bytes already sent cannot
be retracted.
One caution this design must not repeat: #6795 reports that `collectList` on
streaming responses in the
AI transformer / MCP decorator paths breaks backpressure and buffers without
bound. `buffered` mode must
therefore hold a bounded window (`holdBackChars`, with a policy when a
single event exceeds it) rather
than aggregate a whole response, and `final_packet` should reuse the
existing incremental SSE parsing
rather than introduce another full-body join.
### 5. Content extraction is configuration, not code
The prototype hard-codes `choices[0].delta.content`. The plugin should
default to a protocol profile and
allow JSONPath overrides per direction:
| protocol | request | non-streaming response | streaming delta |
| --- | --- | --- | --- |
| `openai-chat` | `messages[-1].content` | `choices[0].message.content` |
`choices[0].delta.content` |
| `openai-responses` | … | … | … |
| `anthropic-messages` | … | … | … |
| `mcp` | `params.arguments` | `result.content[*].text` | n/a |
Higress uses JSONPath for the same reason (`requestContentJsonPath` /
`responseContentJsonPath` /
`responseStreamContentJsonPath`), and Envoy's guardrail adapter has an
MCP-specific parser so the
provider only ever sees inspectable text.
Independently: **matched content is never echoed to the caller** (the same
rule as `ai-sensitive-word`) —
only the verdict.
### 6. Failure policy — the two sides are not symmetric
| situation | proposed default | why |
| --- | --- | --- |
| judge unreachable / timeout / malformed, request side | pass through,
warn, count (`failClosed=false`) | matches `ai-sensitive-word`: a broken judge
must not take the traffic down. Set `true` per rule for strict compliance |
| same, response side, before any byte is released | same flag | consistent,
and observable via `UNAVAILABLE` |
| same, response side, after bytes are released | log only, no option |
nothing can be retracted |
| request that is not a recognized AI request / nothing extractable |
`unmatched: skip | warn | error` | APISIX's `fail_mode`; a silent pass-through
must be a choice, not an accident |
### 7. Rejection response must be protocol-conformant
- Non-streaming: built through `ResponsiveException` + `WebFluxResultUtils`,
never hand-assembled JSON.
- Streaming: a provider-conformant refusal event and a clean end of stream —
never a bare JSON object
injected into an SSE body, which no OpenAI client can parse.
- Never echo the matched text; a generic reason only.
### 8. Observability
Counters `ai_sec_request_deny`, `ai_sec_response_deny`,
`ai_sec_judge_failure` plus a judge-latency
timer; exchange attributes (`ai_sec_risklabel`, `ai_sec_deny_phase`) and one
mirrored risk-level
attribute so the logging and metrics plugins can consume the verdict instead
of this plugin
re-implementing them (APISIX publishes `llm_content_risk_level` for the same
reason).
### 9. Non-blocking guarantee
No Hystrix: upstream externalized it in #7499, and the enum only remains for
compatibility. An HTTP
judge needs no thread hand-off at all — `WebClient` with `.timeout()`,
`onErrorResume` and cancellation
propagation. If a blocking vendor SDK is ever wrapped it must be
`subscribeOn(Schedulers.boundedElastic())` behind a bounded bulkhead, not a
per-request thread pool.
Circuit breaking reuses the existing Resilience4j dependency
(`shenyu-plugin-resilience4j` already
depends on `resilience4j-reactor`) with state shared per rule key, since
per-request breaker state is
useless.
### 10. Dependencies and boundaries — ai-proxy is under active refactor
This plugin consumes **wire format and plugin order only**:
- It must not import `org.apache.shenyu.plugin.ai.proxy.*`. #7351 is
replacing those internal models
(its own background: "tightly coupled to OpenAI Chat Completions and
Spring AI types"), and #7397
defines the new internal models and extension contracts.
- It must override **both** `writeWith` and `writeAndFlushWith`:
`AiProxyPlugin.writeResponse()` writes
through `writeAndFlushWith`, and this area is still moving (`fix(ai):
transform flushed response
bodies` #7123, `fix(ai-proxy): propagate stream cancellation` #7338,
`avoid replaying partial
streams` #7205).
- It must not assume the response keeps being written through
`exchange.getResponse()` (#7543 already
migrated ai-proxy to engine execution once). A fixture-based test with
fixed SSE samples and no live
upstream will fail if the response stops flowing through the decorator.
- No general AI protocol layer here — that is #7382. If the Protocol
contract later lands in
`shenyu-plugin-ai-common` with a usable extraction hook, this plugin can
consume it as a follow-up.
- The order coupling (response side must precede `aiProxy`) is documented,
not encoded.
- The plugin also works on a plain HTTP route (request side only), like
`ai-sensitive-word`, so it does
not require ai-proxy to be configured.
### 11. Non-goals
- Admin `plugin` / `plugin_handle` / `resource` / `permission` /
`shenyu_dict` /
`namespace_plugin_rel` rows and the console form — a follow-up issue, per
the scope note above
(#7154 is the precedent: the registration of `ai-sensitive-word` was
tracked, and closed, separately
from the plugin itself).
- The watermark plugin from the same prototype branch (unrelated concern; a
separate proposal if wanted).
- Changes to `shenyu-plugin-base` — the prototype's shared response
decorator belongs inside this module.
- Hosting or managing judge models; word-list dictionaries (that is
`ai-sensitive-word`); a sidecar /
`ext_proc` deployment of the judge as Envoy AI Gateway does (noted as a
possible future architecture,
not part of this).
## Task List
1. Plugin module `shenyu-plugin-ai/shenyu-plugin-ai-content-moderator`, its
starter, a `PluginEnum`
entry and the rule-handle DTO in `shenyu-common` (plus its test).
2. `ContentModerationJudge` SPI + `openAiCompatibleJudge` + `httpApiJudge`.
3. Request side: size bound checked against the declared `Content-Length`
before the body is read, read
through `ServerWebExchangeUtils.rewriteRequestBody`.
4. Response side: non-streaming first, then streaming on both `writeWith`
and `writeAndFlushWith`, with
the three modes of §4.
5. Verdict model, failure policy, metrics, README (streaming caveats, order
requirement, judge
throughput).
6. Tests: `StepVerifier` for both write paths; per-batch gating;
end-of-stream flush; short-stream tail
(a stream shorter than the window must still be judged); no judge call
for content-less frames;
judge unavailable → configured policy; a test asserting the scan does not
run on the event loop.
7. Follow-up issue: admin registration rows and the console form.
## Open questions for maintainers (with my proposals)
I have written a proposal after each question so a maintainer can simply
agree or redirect; nothing here
is decided on my own.
1. **Plugin name and placement.** *Proposal:*
`shenyu-plugin-ai/shenyu-plugin-ai-content-moderator`,
`PluginEnum.AI_CONTENT_MODERATOR(196, 0, "aiContentModerator")`.
Rationale: `ai-sensitive-word`
already placed content compliance in the `ai` family, and a third family
for the same concern would
split it; the plugin name stays lowerCamel like `sensitiveWord` /
`aiTokenLimiter`.
2. **`PluginEnum` code, and therefore the order against the dictionary
check.** *Proposal:* `196` with no
renumbering, accepting that this judge then runs *before* `sensitiveWord`
(197). Rationale: `196` is
the only free value that still precedes `aiProxy`, which the response
side requires, and taking it
changes nothing else. The cost is bounded and easy to reason about: a
request the dictionary would
have rejected still reaches the judge. If maintainers prefer the cheap
dictionary check to reject
first — which saves a judge call on dictionary-dirty traffic — then this
plugin should take 198 and
`agentGateway` moves, since equal order values make the chain order
nondeterministic. I recommend
`196` for v1 and can renumber if that is the preferred direction.
3. **Default policy when the judge is unreachable.** *Proposal:* fail open
by default
(`failClosed=false`), per-rule override to fail closed. Rationale: it
matches the merged sibling
`ai-sensitive-word`, whose README states that a broken dictionary must
not take the traffic down;
the moderation service is an external dependency shared by every rule
that uses it, and the plugin is
opt-in per rule, so a strict deployment can flip one flag. The request
side and the response side are
documented separately, because on the response side nothing can be
retracted after the first byte is
released.
4. **Denial status and body for a recognized AI protocol.** *Proposal:*
`denyStatusCode` default `403`
with a provider-shaped body when a protocol is recognized. Rationale:
`403` follows the sibling
`WafPlugin` and keeps the rejection visible to infrastructure and
monitoring, and the body is still
built in the client's protocol so an AI SDK surfaces a parseable refusal
rather than an opaque error.
I am aware APISIX and Higress default to `200` specifically so the AI SDK
treats the refusal as a
normal completion; if maintainers prefer that behaviour for AI routes, I
will make `200` the default
there and keep `403` for plain HTTP. On the response side this only
applies while nothing has been
committed (i.e. before the first flush); afterwards the refusal can only
be carried inside the stream.
5. **Circuit breaking in v1, and the dependency.** *Proposal:* v1 ships
`timeout` + `failClosed` and no
new dependency. If maintainers want circuit breaking in v1, implement it
on the Resilience4j
dependency the fault-tolerance plugins already bring
(`resilience4j-circuitbreaker` / `-reactor` via
`shenyu-plugin-resilience4j`) with breaker state shared per rule key.
Rationale: a breaker is only
meaningful with shared state, and that is already the pattern in
`Resilience4JRegistryFactory`; adding
a second resilience library, or a Hystrix-style per-request thread pool,
would be scope this issue does
not need.
6. **Where the protocol extraction profiles live.** *Proposal:* in this
module for v1, with a follow-up
to consume a shared hook if one appears. Rationale: #7382 is still
completing the Protocol layer's
response and streaming semantics, and those contracts are for the proxy's
own encode/decode path
rather than an extraction API for other plugins; depending on them now
would tie this plugin's PR to
an in-flight refactor. If the extraction contract later lands in
`shenyu-plugin-ai-common`, moving
these profiles there is a small, contained change.
## References
- Merged sibling: #7153 (`ai-sensitive-word`), #7155 (its admin
registration).
- Related existing defect to avoid repeating: #6795 (unbounded buffering on
streaming responses in the
AI transformer / MCP decorator paths).
- ai-proxy layering: #7351, #7381, #7382, #7383, #7384.
- APISIX: `ai-aliyun-content-moderation` / `ai-aws-content-moderation` docs,
and apache/apisix#13735
for the response/streaming work.
- Higress: `ai-security-guard`.
- LiteLLM guardrails: modes `pre_call | during_call | post_call |
logging_only`,
`streaming_buffer_until_moderated`, `guardrail_status`, `decision_model`.
- Amazon Bedrock Guardrails: content filters (categories including
`PROMPT_ATTACK`,
`inputStrength` / `outputStrength`), and the standalone `ApplyGuardrail`
API.
- Envoy AI Gateway / guardrail adapter over `ext_proc`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
