Very cool :) I've been doing my own trial using https://github.com/franktheunicorn/franktheunicorn but I've been doing the actual review posting manually.
On Thu, Oct 1, 2026 at 3:01 PM Hyukjin Kwon <[email protected]> wrote: > Hi all, > > I've been running a trial of automated AI code review on new apache/spark > PRs, and I'd like to share how it works and get your feedback. > > For now the reviews are posted from my own GitHub account (@HyukjinKwon) > while I check that it behaves well. Once it's proven, I plan to move it to > the https://github.com/spark-test account, so the reviews are clearly > separate from my personal ones. I plan to run this for the next week. > > *What it does* > > Each review is a full, multi-pass review of the PR diff, run by Claude > (via Claude Code). Findings are checked against the code before they're > posted. They're posted as a normal GitHub review: a short summary plus > inline comments, each labeled with a severity, e.g. "Non-blocking (P2)" > or "Nit (P3)". Example: > https://github.com/apache/spark/pull/59184#pullrequestreview-5379759787 > > *No LGTM, no approvals* > > - It never approves a PR and never posts "LGTM". Every review is a plain > "Comment" review, so it doesn't count as an approval or a change > request, and it has no effect on merging. > - If a review finds nothing new, it posts nothing at all: no "no > findings" or "all comments addressed" message. > - It doesn't push commits or edit anyone's PR. > - Human review and committer judgement are unchanged. Please treat its > comments like any other reviewer's: feel free to push back or ignore. > > *When a review is triggered* > > - A new PR is reviewed shortly after it's opened. Draft PRs are reviewed > once they're marked ready for review. > - After a review posts findings, the bot waits. It reviews again only > when the author pushes new commits. > - It also waits while a PR has open review comments from anyone that are > newer than the latest commit, so it doesn't pile on while the author is > still responding to reviewers. > - Replying in a thread without pushing doesn't trigger a re-review. > > *When it stops* > > - When a review finds nothing new. The PR gets a fresh review if the > author pushes again later. > - When the PR is merged or closed. > > *Early numbers* > > Over the first ~12 hours (Oct 1), it covered 10 PRs, including 5 > re-reviews after the author pushed. It posted 7 reviews with 14 findings > in total, mostly P3 nits. Four PRs came back clean and got no comment. > Several authors pushed fixes, and the follow-up reviews found nothing > new. > > *Feedback* > > Please let me know if the comments are useful, noisy, or wrong, either > here or on the PR. If you'd rather it not review your PRs, tell me and > I'll exclude them. I'm also happy to adjust when it triggers based on > what the community prefers. > > Thanks, > Hyukjin > -- Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her
