Twelve Ways to Be Wrong About AI-Assisted Coding(third-bit.com)
3 pointsby signa113 hours ago1 comment
  • thepasch3 hours ago
    Another way to be wrong about AI-assisted coding: basing an entire article on mostly outdated empirical findings, some of them wildly so?
    • eesmith3 hours ago
      I assume you are implying you disagree with the summary given, and not making a general observation about how some studies are flawed.

      Aggregated by date:

         2026 3
   2025 7
   2024 1
   2023 3
   2022 1
   2021 1
   2019 1
   1984 1
      Given the time needed to gather data, write it up, and go through the peer review process, this is about what I would expect for up-to-date empirical findings combined with some background base understanding.

      Can you suggest some better and more recent empirical findings?

      Which ones are wildly outdated?

      • thepasch2 hours ago
        “Outdated” was perhaps an inaccurate term to use here. The issue isn’t what dates these studies were published on, it’s what models they were evaluating on. This field is moving faster than a speeding bullet. To start with the most recent studies cited:

        - Stray2026 covers a “two-year period” of commits. The paper was submitted in September 2025 and revised in January 2026. “Vibe coding” at that time (so from 2023-2025) was still mostly pasting code from chat windows into your IDE or accepting autocomplete suggestions.

        - He2026 is similar in the timeframe, submitted November 2026 and revised in January 2026, focused entirely on Cursor, which, at that time, was very different in its focus (code completion/inline chat prompts vs agentic back-and-forth with extensive tool use and autonomy). Again, very different from what reality looks like currently.

        - Becker2025 explicitly evaluated Claude 3.5/3.7 Sonnet, an entire generation removed from the current state-of-the-art.

        - Xu2025 and Bakal2025 say they evaluted “GitHub Copilot”, which isn’t an AI model but an AI router. I couldn’t find any data on whether they actually evaluated what models the developers’ requests ended up going to.

        The point is that there is no recent empirical data because by the time a rigorous study is ready to publish, the industry and its capabilities have already moved on dramatically. The truth is that, as of right now, anyone claiming to have empirical proof of either slowdowns or efficiency gains is wrong. There is no way to tell.