> There is simply no good reason to believe that AI will magically be able to work around these fundamental tradeoffs of biology, that it could somehow design a virus with the lethality of H5N1 influenza but the contagiousness of H3N2. The basic science required to build such a thing has not been done. When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion.

I do not wish to make a claim beyond my expertise. My expertise is not in biology. I therefore defer to the author's claims that these things are extremely difficult.

Question: Given the observations of

(1) rapid breakthroughs in biology, where limited observational data was able to bootstrap e.g. AlphaFold, which is mentioned in another of this author's blog posts: https://blog.genesmindsmachines.com/p/no-alphafold-has-not-c...

(2) the details of how a misspecified task, reward hacking, and willingness to keep going until a solution was reached, was behind the OpenAI-HuggingFace incident

Is it really so far removed from plausibility as the author suggests? Even on the 2029 scale?

I mean, good if so, but if the historical trends remain, in about a month or so we should see open weights models that are just as capable of hacking, coordination, and persistence as the OpenAI model, so any DNA/RNA systems that are on a network connected to the internet are likely to be vulnerable regardless of firewalls or other security considerations.

Remember, an agent tasked with a goal like this isn't like one disgruntled teenager who neither knows nor cares what the difference between DNA and RNA is, the HF incident was around a thousand agents that just kept on going and going in pursuit of the goal they misunderstood.

I will not share my ideas about what I'd do if this was a game and I was red-teaming it. I don't wish to give misanthropic teenagers (or more likely, their LLMs) any ideas.

Still, I can take some hope from the author saying it's unlikely…

… despite having read basically the same dismissal about AI and Go just a few months before AlphaGo.