Scientific production in the era of large language models [pdf](gwern.net)
62 pointsby nkko3 days ago6 comments
  • barishnamazov3 days ago
    The key finding here is the reversal of the relationship between writing complexity and paper quality.

    Traditionally, sophisticated writing correlated with higher-quality research (or at least higher status/effort). This paper argues that post-LLM, we are seeing a flood of manuscripts that use complex, polished language but contain substantively weaker scientific contribution.

    They claim LLM adoption increases output by up to 89%, which is a massive productivity shock. If the cost of generating looks-like-science prose drops to near zero, the signal-to-noise ratio in peer review is going to crash. We are entering the era of the polished turd, and likely worse case of publish and perish [0].

    [0] https://en.wikipedia.org/wiki/Publish_or_perish

    • lo_zamoyski3 days ago
      I don’t like how we speak of science and knowledge in terms of production. This is not a factory. We are not in the business of churning out units.

      But what I might expect is not a worse case of publish or perish, but the collapse of this pathology. If churning out crap is cheap, it will render publishing worthless. What I might expect is greater decentralization. Researchers might share reports and articles with colleagues directly. Networks of trust vs. broadcasting.

      • R_D_Olivaw2 days ago
        I don't like how we talk about humans as factories either. That we re all here to be a "productive" as possible. That are life is measured in how much crap we produced.

        I would love for this same ethos to spread to other domains. Why DO we hoard things so much? Why don't we just share more? Is it fear? Is it game -theory?

        How many households have thousands of dollars worth of equipment to maintain that very household that each is used maybe ~30 minutes a week.

        Why do we keep churning out more plastic crap that just sits in our spaces. Unused

      • dr_dshiv2 days ago
        Some parts of science can be automated and scaled immensely. Like, look at the science that went into building modern large language models. Massive numbers of theory driven empirical experiments. Ironically, not many papers being written about much of it.

        In many domains, human ego is the bottleneck— or maybe organizational incentives. Most scientists work largely alone—a lot more could be done with the right leadership.

      • yawnxyz3 days ago
        we wish it wasn't, but it totally is — in microbiology at least if you don't find anything meaningful to publish even students/post docs in the best unis like Stanford will still put out tons of review papers and microbial resource announcements
      • postexitus2 days ago
        The discipline of writing for sharing knowledge is immense - not only for the sharing part, but to look at yourself (or your research) in the mirror with a critical eye and revise it multiple times. If you don't have that discipline, sharing knowledge even with colleagues / networks becomes harder. It's not only LLM churning out slop, but it's also without that discipline, the researcher putting out half finished work.
    • DoctorOetker2 days ago
      Think of the use of ML towards formal verification, a putative ML generated proof can be easily checked with a classical verification algorithm.

      What will happen is that we just translate more and more physics and engineering into a rigorous formal system, and then those deductions become verifiable. Actual measurements are another matter, and its not unimaginable petty humans will HAVE TO organize measurement tools (DMM, oscilloscope, ...) to cryptographically sign measurements, and to also sign schematic connectivity (yeah, so the oscilloscope or DMM probe terminals and the terminals of say an amplifier etc would be "smart" and sign the fact they are connected during the experimental setup. All to prevent people from faking measurements.

      BUT if this is done we can transport physics and engineering into the decentralized formally verifiable domain.

      After that biology for example could work, but formalizing ethics or (direct) democratically established rules and medicine at the same time will result in collective realization of certain inconsistencies (not unlike intervention vs non-intervention when watching lions hunt zebra's).

      • DoctorOetker2 days ago
        I never cease to be utterly unsurprised how people conflate emotional controversy with disagreement.

        shoot the messenger and all that

  • hirenj3 days ago
    It is a huge worry for me that unless we decouple the publishing “system” from the career pathways (i.e., rewards), we are going to lose access to both the careers (to robot-weilding bullshitters) and even worse, the shared space where scientific communication took place.

    Does anyone know of any writing on the network effects of the publishing system? What would happen if the actual value of the journals (of the little they provide!) were to go away?

    The death of scientific twitter, and the failure to establish any replacement makes me worry that we won’t be able to coalesce around a replacement system. Obviously preprints play a role, but we really need our scientific communities to engage with them in a more serious way.

    • naasking3 days ago
      The arxiv should itself embed a review and commenting system, possibly even blogging. Publishers are archaic, scientific social media should be arxiv's future.
      • setopt2 days ago
        The risk is that publishers might then start opposing the publication of manuscripts that have been shared as preprints on arXiv, if they start perceiving it as a competitor and not a supplement to their "service". But I concur, arXiv with social media features would be nice.
      • R_D_Olivaw2 days ago
        I like this idea.

        But it should be a separate entity for robustness's sake.

        Let's make it! We'll even hold weekly live video/audio discussions on each paper for it's lifetime.

        Let the errant and emerging thoughts of researchers accompany the original research so we can better iterate and grow. In the open.

    • lmc2 days ago
      > The death of scientific twitter, and the failure to establish any replacement

      In my field it's mostly in LinkedIn now.

      • hirenj2 days ago
        Can you say what field that is? I hear this sometimes, but my feed there is significantly low signal to noise, and I have had to pollute my “connections” to the point where I accept everything, as I have been trying to advertise job openings using it too (which frankly has been pretty poor too).
        • lmc2 days ago
          My field now is Earth observation/geoinformatics and my recent connections are mostly academics and applied programmers. Also, mostly Europe-based and very few in the US. My feed is mainly about new papers, conferences, tools, webinars etc..

          I used to do corporate software dev and my feed (and work) back then wasn't that interesting. I barely used the site.

  • felipeerias3 days ago
    On many fields, the link between “writing papers” and “producing science” was already fraying before the arrival of LLMs.

    We will have to find better ways to share and promote valuable research, before we all drown in the noise.

  • conditionnumber3 days ago
    Very cool appendix describing how they collected the data. I was kind of surprised to learn that they collected arXiv abstracts + metadata from Kaggle, but it definitely makes sense. I was also surprised that 6 years of SSRN papers was only ~1.3m documents. If you assume 20 pages/document and 400 words/page and 1.3 tokens/word, then it would only cost (ballpark) $1000 to pass the full corpus through the 4o-mini completions API. I think it would be really neat to build out a "Dataset Used", "Model Used" etc table for SSRN papers. I imagine more complicated questions would be harder to answer (because you might have to analyze non-text parts of the documents).
  • hazrmard2 days ago
    The paper finds:

    - For LLM-assisted output, the more complex the LLM-writing is, the less likely the paper is to be published. From eyeballing, at WC=-30, both have similar chances of publication (~46%). At the upper range of WC=25, LLM-assisted papers are ~17% less likely to be published.

    - LLM-assisted authors produced more preprints (+36%).

    I wonder:

    - What is the distribution of writing complexity?

      * Does the 17% publication deficit at WC=25 correspond to 17% of the 36% excess LLM-assisted papers being WC=25, thus nullifying the effect? Although, it puts extra strain on the review process.
  • ggm3 days ago
    Really badly named article at source. Scientific PAPER production in the era of...