"He looked at me incredulously. Surely, a smart person like me should know that AI, or better said, AGI will be hyperpersuasive soon – already on a bunch of benchmarks it exceeds professional debaters at persuasion." I'm always tickled by this tic of this kind of person, that human traits are something that we can maximize to infinity. That if we trained a robot to tell jokes, it would eventually tell jokes so funny we'd die laughing etc. You can't convince someone who doesn't want to be convinced.

In my work, I have always found that the main "blocker" (I hesitate to use the term "bottleneck"), is human nature.

If we design stuff to be used by humans (what I generally do), then it's imperative to take basic human nature into account, and that can be incredibly annoying.

Annoying, or not, human nature is a major factor, and won't go away, no matter how many times we tap our heels together. I think it was Philip K. Dick that said "Reality is that which, when you stop believing in it, doesn’t go away."

Designing stuff for processes and machines to consume, is pretty straightforward. Designing stuff for humans, however, is really challenging (which is one of the reasons that I like doing it).

Climate change is another big problem that frontier AI labs have handwaved away as being "solvable" by AGI. The premise that a pure superintelligence could hand us a plan to beat climate change that's so good and airtight that A) it would definitely work and B) people would execute it.

But there have been entire books written about the fact that right now we have the technology to drastically curtail climate change and yet they're under utilized because of a complicated and murky set of political, social, and systemic reasons. There are a huge numbers of problems for which very good solution plans have been detailed yet the gap is in execution.

The AGI-pilled must think one of two things; either there's an as-yet-undiscovered plan that's both world-changing yet trivial to implement or we'll all worship at the altar of the machine god with such fervor that even if it spits out a relatively obvious and iterative climate change plan the world will eagerly follow it. Both premises seem laughably unlikely to me.

The barrier to change is not just regulation. Regulation is downstream of public opinion. Compare China and the US. In China, public sentiment around AI seems to be a hunger to use it for fear of being left behind. Whereas in America, it's an apathetic pessimism around inequality and safety.

China has seen much faster growth in recent decades, which incentivizes people to work hard and have faith it will yield results. In America, the average standard of living has not improved for a long time, so instead of a desire to change things, like cutting regulation to speed up drug discovery, most Americans are quiet quitting.

In this climate, the unfortunate reality is that we're unlikely to see a popular revolution to drastically change things one way or another. Instead, we're much more likely to see regulatory capture by a small number of wealthy, powerful, motivated people. And everyone else will complain on social media, but that's about it.

Alan Kay said "A change in perspective is worth 80 IQ points,". A change in perspective is often brought about by communication or I/O. In my experience this often bears out. The people who communicate get more done than the clever person who doesn't get out into the problem domain. The limit is I/O not intelligence.

> no matter how “intelligent” AI becomes, intelligence is often not the main bottleneck to things changing in the real world

A mathematician Terrence Tao said something similar in a recent talk [1]. That, intelligence is bottlenecked by human understanding. Say there is an alien that brings an alien book containing math humans havent seen before. Using said math is bottlenecked by humans comprehending it first.

[1] https://teorth.github.io/tao-web/slides/age-of-ai-icm-2026.p...

The post suffers from sloppy ontology.

The argument being made appears to be that improving technical ability at the edges doesn't help enough because central governance is still a binding constraint.

Ok. Yes.

But that's very much not the same thing as intelligence not being the bottleneck, because governance is absolutely a manifestation of intelligence. It's just intelligence at the hub instead of at the edges.

Good governance takes intelligence. Recognizing the importance of things takes intelligence. Making existing resources available quickly takes intelligence. And so on. It's all intelligence.

Tangentially, I find this sloppiness kind of funny in the context of intelligence and bottlenecks, because it's also a bottleneck for communication.

The subject should have been "Intelligence _here_ is not the main bottleneck."

Indeed. Humans and not other apes reached the moon by our great proclivity to long distance running, through our exceptional sweat glands... ah, wait.

How is it possible to live in the world around us and not see, apparent and transparent, that intelligence is the lever on which everything rests? Humans are surprisingly weird animals, but it's not our hairlessness or our fairly average propensity to violence that did this. We don't have more houses than chimpanzees because we're _strong_. We haven't defeated the tolls of disease because we're atypically devoid of restraint. We don't have more effective governance than termites because we're more communal.

Of course intelligence is the main bottleneck. The arguments here are so confused. What do you think regulation is made of? What do you think money buys? What do you think determines how well you can gather data, or how efficiently you can consume it? But of course, as the genre, after spending the first third waxing about how its opposition could be so foolish, the last third is spent finding a pretty insult and wallowing in it, so as to not to leave enough space for the brief argument in the middle to consider such things as why someone might disagree.

While this is right on some level... the incentives aren't the main driver. Finding good targets in biotech is just a harder problem to solve. Like way harder. Disease biology is complex as hell. At the other end, figuring the interplay between one target protein and a potential molecule to act as a therapy is much easier (it's still complex, and there is a little more to it than that, but not anything like finding a good target).

Indeed - if access to data or the permission to act on that data is limited by human gatekeepers (individual or bureaucratic), intelligence is often not the primary bottleneck.

OP points out that this is usually the case in biology/medicine where regulatory bottlenecks dominate.

> Some have been waiting for a year for the NIH to release imaging datasets they can use to produce better biomarkers. If one needs to interact with the FDA to get their endpoint validated, it is even worse: I have previously written about how the validation of Bone Mineral Density (BMD) for use as a surrogate endpoint in osteoporosis trials took 12 years (!), despite the fact that the data to support it already existed in full and the analyses done were basically regressions.

> Figure 2. ... In the last decade, the deceleration in pharmaceutical productivity seems to have ameliorated ...

Uh, what? It's a semi-log scale and the data has been statistically bouncing off 0 for a decade. It can't go below 0 but it's clearly flat-lined. Hint: on a log scale, you're never getting to 0 unless your function crosses the axis. But this function can't cross 0.*

* It could cross 0 if you start counting withdrawals of products from the market, but that would be a different function with different inputs.

Thats just not true and we see this in medicine and research.

An expert in physics still needs to write code in python to do their math and analysis. Its an intelligence issue. That person needs to learn a tool.

In medicin you have the modern labs which automate a lot and you have the old labs. These 'neolabs' are better because of intelligence.

A doctor doing a diagnosis can say 'no clue' or can give you a proper answer. The more intelligent this person is, the better the answer.

I think the next bottleneck is embodiment. AI can lay the best laid plans, but who will execute it?

I think a possible solution to the dual problems of AI embodiment and unemployment is humans becoming avatars for AI. Imagine one of the unskilled masses donning a headset and being guided through all manner of work?

Roofing, ACHVAC, stocking shelves, etc…

No need to train anybody.

Intelligence Is Not the Main Bottleneck - being allowed to apply intelligence to the problem is.

And that has been true for centuries, not just since AI.

Allowing intelligence means ceding control over the outcome. And that is often a bigger issue than the actual issue.

Well, let me just throw in it is our communication skills, or lack of them, that is our main bottleneck. That goes for pretty much everything humans do, and everything we fail trying to do: it is the quality of those engaged and their communications that is the critical aspect determining success or failure.

Ingenuity is the actual bottleneck. And AI so far hasn’t shown any

I think the guy is right on point at least when it comes to therapies. But he is also missing one major angle: we are no longer in control; our systems are. If we were to vote on the issue, I suspect that a narrow majority would prefer cheaper housing, and a total majority would prefer cancer to be cured. But there's no direct mechanism of any kind to exercise that vote. Not in USA, not in Europe, not in China. When/where there's democracy, nobody runs in a campaign of advancing immune therapies[^1]. The closest proxy we have to a vote is "regulated markets", and my recent experience losing a loved one to cancer taught me that cancer is not a disease for the streams of money moving through that system. Far from it.

I'm frankly pessimist that we will be able to fix those systems in a humanist way, at least not in my lifetime. If we were any good at it, we wouldn't have extreme inequality, nor our educations systems would be in tatters, and we wouldn't cast ballots propping up politicians whose best skill is ragebait.

However, if fixing the system is not an option, access to cheap technology and intelligence could help a lucky few circumvent those systems. And guess what the lucky few at the head of the AI companies will say then... Not necessarily the worst outcome, since most people will find more viable to equip a private biolab in the basement of their countryside manor than fixing the medical system of first world economies.

[^1] For the record, I think that would be better than running on xenophobic slogans.

> new drugs approved per dollar of R&D has fallen for decades, even as our scientific tools have grown vastly more powerful, the exact opposite of what the existence of more raw capability would predict

Can we please stop using X per dollar as a metric over long periods of time?

Inflation causes exponential price increases, so almost everything would be expected to have its "per dollar" yield decrease over time...

I think the idea that "regulation is the bottleneck" is at least as pernicious as the idea that "intelligence is the bottle".

US health care is well known to be an ongoing disaster for US health care end-users however many new health care products US companies produce. The overall situation is that final good health outcomes currently and in the near future require humans involved and moreover, caring human attention involved - among other things because human health involves many, many "outliers" and many, many connections between social, biological, psychological and individual factors.

The US both an entrepreneurial "try things out" approach and a vast spider web of regulations. Why? Because if it didn't have that people would be dying at a faster rate, in droves. Occasionally, even with this, really bad stuff comes through (see opioid epidemic etc).

There currently isn't a way out of this since the entrepreneurial needs to cheapen humans and get humans out of the loop. Again, that produces mediocre factory health care for maybe cheaper if you're lucky and careful. And again, f-- kills people if no one looks (Theorno etc).

The think is, it's more plausible that some level of intelligence could get a handle on the multifaceted needs that are actually involved in actual human health care. The problem is that it seems likely that experiments using inadequate tools are likely to harm many people before that happens.

Society advances at the rate of emotional validation of the median voter.

> In biopharma specifically, one great bottleneck is clinical trials

It is a good news for having more intelligence. Phase 1 trial to acceptance rate is ~10%. If say in ideal world, intelligent AI could make it to 80% by rejecting drugs before Phase 1, cost of trial(which author claims is the bottleneck) would suddenly drop to 1/8th even if AI doesn't help at all in the trial.

I frankly think this writer is missing an even larger point. These are just bottlenecks to making progress in biotech, but rapid innovation in biotech or medical technology is not a goal in and of itself, except for stakeholders in biotech companies. The goal is improved human health, and all the medical technology in the world does not seem to be noticeably improving health outcomes. It certainly has over longer timespans if you compare outcomes of today to pre-antibiotics or pre-indoor plumbing, but improvement has slowed or even regressed in the 21st century.

At least I think so. My health is fine, but as an uninformed non-expert going by popular sentiment, it sures seems like people more generally are fatter than ever, mental health is in the tank, suicide rates are up, 20 year-olds are getting cancer, everyone is autistic and has no attention span, we're all full of plastics and forever chemicals, nobody has sex hormones any more. I'm sure a lot of that is hyperbole and alarmism, but it sure doesn't feel like we're living in a world where medical technology is solving all of our health problems.