weiqingy commented on issue #894:
URL: https://github.com/apache/flink-agents/issues/894#issuecomment-5087971235
@wenjin272 Yes, I'd like to try this. Happy to start on my side right away.
On the direction first: this is the developer-side mirror of what we just
landed on the reviewer side, which I think is the strongest argument for it.
`code_review.md` plus the per-type guides tell an agent what to look for in
someone else's change. Nothing today tells an author's agent what it owes a
reviewer. Writing the contract down once serves both stages, so the cost is
amortized rather than added.
I already write something close to this for my own non-trivial PRs, as a
private working note. So on your authoring-overhead question I can share a real
data point from my end: for a PR that already has a design behind it, the
Implementation Description is mostly a derivation, not new authoring. Where I
expect the overhead to actually show up is the test-to-contract mapping, which
is the part I currently do implicitly.
**First candidate**. I have a non-trivial, heavily AI-assisted PR coming up:
native structured output for the Azure OpenAI chat model (#280, following #843
and #919). It has a real behavioral contract (when the native path applies
versus the prompt-based fallback), a compatibility story, and a failure mode
worth stating explicitly. I'll write the Implementation Description for it and
we can use it as the first sample. If that turns out to be too small a case,
I'll do the same on the next one.
**One question about stage 2**. If the same agent that wrote the
implementation also authors the description, and the description can still be
edited after stage 1, then stage 2 verifies the code against a document that
was reshaped to match the code. The check then passes by construction and tells
us nothing. A cheap guard would be to have stage 2 verify against the version
accepted in stage 1, so any later edit to the description shows up as its own
reviewable delta instead of silently becoming the new baseline. Does that match
how you were picturing it, or were you thinking of the description as a living
document that tracks the PR?
**Where the description lives**. Length probably decides this, and how long
these actually run is one of the things the experiment should tell us. My guess
is that a full description for a non-trivial change is long enough to swamp the
PR body and bury the template sections under it. There's also a versioning
problem that ties back to the point above: if stage 2 verifies against what
stage 1 accepted, the description has to be pinnable, and an edit to a PR body
leaves no visible trace. So I'd suggest a short contract-level summary in the
PR body, enough for a reviewer to start stage 1, with the full description as a
PR comment that we amend by posting a new comment rather than editing. That
gives the stage-2 agent a permalink to exactly the version humans accepted, and
costs the repo nothing. A per-PR document committed to the tree is the other
way to get versioning, but it leaves a permanent file whose only reader is one
review (and going by #911, new markdown also needs a `tools
/.rat-excludes` entry before `check-license.sh` passes). If descriptions
outgrow what a comment carries comfortably, that's the signal to move to a
committed document, and by then we'll know what size we're dealing with.
**One thing worth measuring**. Alongside your three criteria, I'd like to
track whether writing the description catches anything before review. Stating
the failure behavior and the test-to-contract mapping explicitly is where I'd
expect gaps to surface. If that holds, the description pays for itself even on
a PR where it saves the reviewer nothing.
On the rest of the issue: now that #911 is in, I'll pick up the remaining
change-type guides (`api/`, the Python-Java bridge, `dist`, docs-only) next,
and module-scoped `AGENTS.md` after that. This experiment is independent of
both, so it doesn't need to hold either up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
