weiqingy commented on issue #894: URL: https://github.com/apache/flink-agents/issues/894#issuecomment-5100206393
@wenjin272 That distinction resolves it, thanks. I was reading stage 2 as a one-directional check, and against an as-built description that check really is vacuous. The second half of your stage 2, whether the description omits any important behavior, is the part I was missing, and it changes the shape entirely: the description isn't a spec the code has to meet, it's a summary whose completeness the agent certifies so a human can review the summary instead of the diff. Nothing to pin in that model. Agreed as well that pinning only makes sense as an optional practice for a contributor running a spec-driven workflow, not something the project should require. One thing I'd suggest we watch during the experiment. The omission check is carrying most of the weight now, and it's the harder of the two halves. Verifying a stated claim against the code is bounded work, but deciding what the description should have mentioned is close to the full review problem, and it asks the checking agent to notice something the authoring agent already missed once. So if we run a few PRs and the omission check comes back empty every time, I'd read that as a signal about the check rather than evidence the descriptions were complete. Might be worth logging omissions found as its own number alongside your three criteria. One concrete way to help that check: treat the six fields you listed (runtime flow, behavioral contracts, key decisions, failure behavior, compatibility impact, test-to-contract mapping) as required headings, with an empty one stated explicitly rather than dropped. It bounds part of the omission problem, since "does the compatibility section name what breaks" is a much cheaper question than "is anything missing at all". It also counters the failure I'd expect to be most common: an as-built description written by the same agent that wrote the code tends to describe what that agent deliberated about, so decisions it made reflexively never reach the page. Fixed headings force a sweep across categories instead of free recall. A section that says "Compatibility impact: none" can still be wrong, which is what the diff-side half of stage 2 is for, but a missing category at least becomes visible instead of invisible. I'll write the first description with the Azure OpenAI structured-output PR and we can calibrate from there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
