Hi all, I've been running a trial of automated AI code review on new apache/spark PRs, and I'd like to share how it works and get your feedback.
For now the reviews are posted from my own GitHub account (@HyukjinKwon) while I check that it behaves well. Once it's proven, I plan to move it to the https://github.com/spark-test account, so the reviews are clearly separate from my personal ones. I plan to run this for the next week. *What it does* Each review is a full, multi-pass review of the PR diff, run by Claude (via Claude Code). Findings are checked against the code before they're posted. They're posted as a normal GitHub review: a short summary plus inline comments, each labeled with a severity, e.g. "Non-blocking (P2)" or "Nit (P3)". Example: https://github.com/apache/spark/pull/59184#pullrequestreview-5379759787 *No LGTM, no approvals* - It never approves a PR and never posts "LGTM". Every review is a plain "Comment" review, so it doesn't count as an approval or a change request, and it has no effect on merging. - If a review finds nothing new, it posts nothing at all: no "no findings" or "all comments addressed" message. - It doesn't push commits or edit anyone's PR. - Human review and committer judgement are unchanged. Please treat its comments like any other reviewer's: feel free to push back or ignore. *When a review is triggered* - A new PR is reviewed shortly after it's opened. Draft PRs are reviewed once they're marked ready for review. - After a review posts findings, the bot waits. It reviews again only when the author pushes new commits. - It also waits while a PR has open review comments from anyone that are newer than the latest commit, so it doesn't pile on while the author is still responding to reviewers. - Replying in a thread without pushing doesn't trigger a re-review. *When it stops* - When a review finds nothing new. The PR gets a fresh review if the author pushes again later. - When the PR is merged or closed. *Early numbers* Over the first ~12 hours (Oct 1), it covered 10 PRs, including 5 re-reviews after the author pushed. It posted 7 reviews with 14 findings in total, mostly P3 nits. Four PRs came back clean and got no comment. Several authors pushed fixes, and the follow-up reviews found nothing new. *Feedback* Please let me know if the comments are useful, noisy, or wrong, either here or on the PR. If you'd rather it not review your PRs, tell me and I'll exclude them. I'm also happy to adjust when it triggers based on what the community prefers. Thanks, Hyukjin
