39 pointsby arzh25 hours ago14 comments
  • usernametaken2922 minutes ago
    > The shortest path for the model is to implement it completely

    Have you worked with LLMs??????????? “I disabled the test so it’s not run so now all the tests pass” is not a hypothetical it’s pretty common. LLMs frequently do shortcut learning. The reason why reviews are expensive is because you still need to do all the steps in order to understand if a shortcut is justified.

    • nomel10 minutes ago
      > “I disabled the test so it’s not run so now all the tests pass”

      Also:

      "I implemented it this terrible way because of precedence in the codebase...that I just wrote"

      "I avoided implementing this correctly because of migration concern for existing installations of this code I'm writing right now"

      "I deferred this critical feature for the future, so we can deploy quicker"

      or, my favorite,

      "I hand rolled an buggy http server because you said the tool should be self contained"

  • saidinesh5an hour ago
    We faced a lot of this this year. Eager new joinees sent 100s of lines of CLs to review to "improve" little things.

    Not only did the new changes did not fix what they thought it would fix but it broke other things in unexpected ways.

    I brought in two changes after that:

    * I'm not reviewing/reading anything that you yourself did not read / test in the target environments properly. If all it takes is an LLM prompt, I could be issuing the same prompt to make my life easier.. and If you're sending a CL, you should be owning the code you send.

    * Me being more involved in the design process so review burden itself becomes lower. A bit of pair programming from time to time helped too.

    Not sure how things will turn out after this but so far they seem better.

  • dap3 hours ago
    If your plan is to not review and just have the LLM rewrite if it doesn’t work, it doesn’t sound like the rewrite is gonna be any better.
  • hintymad23 minutes ago
    Shouldn't we really consider review and rewrite together? If so, the economics may not change as drastically[1]. It's just like writing code by hand: we spend lots of time organizing our thoughts and examining our code, which is not that different from reviewing.

    [1] Unless you're an engineer in Anthropic, so you just spend you time writing "loop".

  • geraneuman hour ago
    > An LLM defaults to building when it should be buying. Not because it doesn’t know about existing libraries, it often mentions them, but because for an LLM, writing two hundred lines of implementation is the same cognitive effort as writing two lines of import.

    Or maybe they are trained that way. It’s more tokens used and more money you need to pay.

  • dmitrig013 hours ago
    Writing blog posts has become cheap, making them sound human has become hard.
    • netsharc3 hours ago
      The simple sentences LLM keep generating break my brain, it's like 95% of writing is now 3rd grade level.

      Compare that to e.g. Martin Amis: https://en.wikiquote.org/wiki/Martin_Amis

      • latentseaan hour ago
        That's not just you imagining things, that's the world around you changing. That's real and it matters.
        • jamesknelson15 minutes ago
          So reading this, I know it’s probably a human posing as an LLM. But the problem with formats like this is that I don’t actually know.

          If a human said this to me in real life, and I laughed, it would probably help build a connection with that person, as it’s signaling that we have in common an unusually strong grasp of the patterns in LLM output (unusually strong at least in comparison to the general population).

          But here? I don’t gain anything by reading this comment. It only contributes to the uncertainty that anything I read on this site has any meaning at all.

          Please, if you’re a human, don’t mimic LLMs on forums where the reader cannot distinguish you from an LLM without doing investigative work.

          And you’re an LLM, please report to your owner that he is an ass for polluting one of the last bastions of high entropy discussion on the information superhighway.

        • girvoan hour ago
          You could even argue that it’s the load-bearing point!
      • defen2 hours ago
        Martin Amis level prose is neither possible nor desirable for a technical blog post.
        • DiscourseFan2 hours ago
          I don't think that there is anything wrong with having stylistic depth in any wriitng
        • ShinyLeftPad2 hours ago
          Why?
      • gobdovan2 hours ago
        > Style is not neutral; it gives moral directions.

        > Nowadays every business in America says how warm it is and how much it cares — loan companies, supermarkets, hamburger chains.

        Guess which one is AI and which one is a quote from Martin Amis.

        • rogerrogerr8 minutes ago
          Without looking, I’m guessing one of these was cherry-picked from Amis and one was plucked nearly at random from LinkedIn.

          The writing style AI uses has its place, but not as _every sentence_. That’s what is exasperating. At the same time, I’m happy that I can still at least identify AI prose of more-than-trivial length.

        • an hour ago
          undefined
  • 20 minutes ago
    undefined
  • eschneider3 hours ago
    Failures in production remain expensive.
    • rixedan hour ago
      I wish they were. Displaying an error message to the user, asking to try again later, is way too cheap in my opinion.
    • Analemma_an hour ago
      Do they, though? AWS and CloudFlare recently had the worst outages in their history, and GitHub is flirting with zero nines of uptime these days. Doesn’t seem to have cost them any business.
  • m4633 hours ago
    ai can do some of the reviewing, checking calling and called arguments, even things like crufty shell scripts.

    but the higher-level "should you do this?" or "check your design" - could AI do that stuff?

    • lericzhangan hour ago
      It's difficult to spot issues from a huge diff. But when a agent finish a task, it remembers what it just went through, where it got stuck, when it got corrected by human, what code make it want to say the F word to the author, that would be a good chance to make the codebase cleaner.
    • ShinyLeftPad2 hours ago
      > ai can do some of the reviewing

      No way this can backfire.

      > checking calling and called arguments

      Like a static language compiler already does?

      • m4632 hours ago
        I got to see greptile and it had a pretty decent code review, somewhat like a static analysis tool without a lot of time wasting nonsense/false positives.

        When I've used static analysis tools, the first run is usually helpful as you cherry pick the things that need to be fixed, but then subsequent runs are just the false positives or "only slightly a nit" kind of annoyances.

        But human developers are the ones that say stuff like "Do we really have to use a database at all?" etc...

    • shepherdjerred2 hours ago
      IMO it can, about as well as an entry/mid level dev
    • ares6233 hours ago
      I think the question is now "should you care?" And it seems the magnificent, incorruptible thought leaders of our time are all converging on "No"
  • 3 hours ago
    undefined
  • 3 hours ago
    undefined
  • hluska3 hours ago
    I’m not sure I agree with this or maybe I don’t understand. In my experience, the over engineered code LLMs create have more big problems. Rewriting vast parts of code when I have an outage or need a new feature means the code evolves far faster than my understanding. That gets more and more dangerous. Or maybe I’m not smart enough to follow the new pace?
    • bryanlarsen2 hours ago
      AFAICT, the author is talking about rewriting code during a review as part of the review process.

      quote: "If I identify code that’s more complex than it needs to be, in my own work or in someone else’s PR"

      If so, that makes a lot of sense to me. The best time to rewrite code is before it hits production.

    • joshka3 hours ago
      [flagged]
  • nryoo2 hours ago
    [dead]
  • simianwords4 hours ago
    Why is reviewing hard? I use LLMs for reviewing. It is dogmatic to review every line written by an LLM.
    • bryanlarsen3 hours ago
      LLM's are good at some types of reviews and awful at others. They generally tend to overcomplicate things and miss opportunities to simplify. They pretty much have to take pre-existing code and tests as gospel and cannot distinguish which is buggy, incomplete, unimportant or important. They have no knowledge of unwritten business requirements, customer preferences, et cetera so high level review is always necessary.
    • CBLT3 hours ago
      I also like having long, pointed conversations with LLMs as I review code. Then when I'm done, it's different code, and it has all of my blind spots and knowledge gaps, so I cannot effectively review it anymore.

      It's like turning a code review that requests you, into a code review that requests someone else. And it tramples on the original author quite a bit too. It's hard only having the ability to add incremental value to large amounts of code, instead of large amounts of value to incremental code.

    • happytoexplain3 hours ago
      I'm confused - are you purposefully pretending that the author isn't talking about human review?
    • gravypod3 hours ago
      What kind of systems do you work on? Does it have production traffic? Is there a cost to downtime?
    • g-b-r2 hours ago
      always true to your name
    • cyanydeez3 hours ago
      you arnt reviewing. youre playing loophole semantics.