1 pointby mt_owl8 hours ago1 comment
  • mt_owl8 hours ago
    I'm getting tired of AI code reviewers flooding PRs with noise and repeat findings, not to say false positives. And then learning nothing when a developer explains why a finding is wrong!

    For me there's a lot going on in code beyond the tech stack: tribal and institutional knowledge, business logic that evolved over time, ripple effects across upstream and downstream systems etc.

    I started those requirements and added simple setup, evidence-backed findings/ LGTM, and most importantly learning from human comments on those findings, so it gets better tuned over time to each repo.

    I ran a smaller benchmark, not definitive by any means, four open source reviewers on 20 pinned PRs/MRs.

      Bubo                 20/20   7/8   27 findings    0% noise
      ai-codereviewer      19/20   6/8   118 findings   20% noise
      ChatGPT-CodeReview   20/20   5/8   75 findings    11% noise
      Qodo/PR-Agent        19/20   2/8   7 findings     not scored
      Alibaba open-code-review — partial run, 4/20
    
    All on GPT-5.5 except Qodo was on 40 (didn't support 5.5). Take it as directional results. I'll be doing more comps in coming days and upload benchmark to the repo.

    I went with polling (zero steup on repo side) - wasn't trying to change the repo CI/CD's rather have a tool seamlessly work.

    Roadmap item: I want to route changes to pluggable subject-matter specialists (Skills) instead of one general reviewer, i.e. industry SMEs or just an "Expert Python Guy".

    Bubo is running in production in two places, a large data processing/ETL codebase and a fintech crypto stack. Would appreciate feedback, particularly on whether the learning from comments part actually holds up.