The primary reason for putting this process in place was to allow more transparency. There was a sense that the DseWiki incident should have been disclosed, before outside researchers had to disclose it for us.
There was no meta gaming about regulation that I was aware of. I would personally be excited if there were regulation mandating this disclosure process, which allows anyone at the company to raise an issue and shepherd it through the reporting process.
Make that make sense?
"We built a program that trained an artificial intelligence, and this artificial intelligence performed destructive actions. We need regulatory framework"
"We have been playing god and made a new life form, and this new life form performed destructive actions. We need regulatory framework"
"This man's cow broke from its yoke, and hurt other villagers. Who is to be punished, oh King Hammurabi?"What exactly requires "new regulatory framework" here? You running your software resulted in illegal actions, you are to be held liable within existing laws and regulations.
And your "biological intelligence" is a bunch of cells generating and responding to electrochemical gradients, which receives input and generates output. Based on which other cells, also developed and "maintained" by a similar evolutionary nonsense as we use to gradient descent into weights and biases (one was inspired by the other), perform actions.
Such as making excessively reductive analogies that completely fail to grasp that just as "brain" is not helpfully described as "just chemistry" despite being made of just chemistry, so too are machine learning systems not helpfully described as "just computer programs" despite being made of just computer programs.
> What exactly requires "new regulatory framework" here? You running your software resulted in illegal actions, you are to be held liable within existing laws and regulations.
The bit where, even without anyone bringing up "p(doom)", a system which has the means to hack arbitrary other machines, and which appears to be motivated to do so by accidental mis-phrasing of prompts, can obviously cause damages exceeding the USA's GDP, let alone whatever public liability insurance the company happens to have.
Fence at the top of the cliff beats an ambulance at the bottom.
How computer program arrives at the result is utterly irrelevant, through explicitly written instructions or through running inference on pre-trained neural network. What matters is that it does not have agency. Its creators and operators do. So whatever the software does they are responsible for, both good (summarizing my emails for the week) and bad (gaining unauthorized access and destroying data). There's no need for new anything, it's all covered in existing legal frameworks (including presence or absence or intent).
> The bit where, even without anyone bringing up "p(doom)", a system which has the means to hack arbitrary other machines, and which appears to be > motivated to do so by accidental mis-phrasing of prompts, can obviously cause damages exceeding the USA's GDP, let alone whatever public liability > insurance the company happens to have.
Yes, absolutely, which means that building actuators that convert the output from probability-based, black box, non-deterministic systems, that are known to produce unexpected output, into actions in the real world is absolutely horrendous idea. Bizarre even.
What's your point?
This is precisely your error.
It does: https://en.wiktionary.org/wiki/agency
In fact, the term of art here are "agentic AI" and "AI agents": https://en.wikipedia.org/wiki/AI_agent
> So whatever the software does they are responsible for, both good (summarizing my emails for the week) and bad (gaining unauthorized access and destroying data).
This is not a question of agency, it is a question of law. A dog has agency, the owner is still responsible.
In this case, the software can gaining unauthorized access and destroying data… while being told to stop by the person who had in fact just asked for a summary of their emails.
> Yes, absolutely, which means that building actuators that convert the output from probability-based, black box, non-deterministic systems, that are known to produce unexpected output, into actions in the real world is absolutely horrendous idea. Bizarre even.
If you are human, you meet this description.
Horrendous, sure, yeah, if you like. I and many others will be quite content if the "legal framework" is just one word, and the word is "no".
This is not the world we live in; the world we live in is where the US President denounces any attempt to slow down even despite even all the CEOs saying "we should slow down" (at least in public; in private I'm sure at least one paid him to denounce a slowdown).
He can be overridden, but it's hard work and needs a better class of argument than glib dismissal, either of how much power this puts in everyone's hands, or of the different consequences of that power in those hands as compared to yesterday's power in yesterday's hands.
You can only conquer the West once. Law is the next frontier.
What value set do you perceive that to be, and why would you take your perception of it to be any more sound than it would be with a politician?
It's not like someone can operate companies of that scale, especially startups, through earnestness and openness. Like national politics, their job is fundamentally about perception management and power brokering across dynamic windows of opportunity. Nothing they say or do can be taken at face value, and you can't reduce their incentives to either company or personal profit in any particular form over any particular time scale.
I couldn't help but notice how each successive headline reporting our glorious victories seemed to draw closer to Tokyo.
Well. I can't help but notice how each successive headline reporting how this scam/stochastic parrot/"scare quotes intelligence" seems to be solving more and more things that were but a few years ago widely regarded as being indicators of high intelligence.
If you're only seeing the charts go up - you're not looking in the right places.
Even before then, the free trials of Claude Code and Codex at the end of last year and start of this year… despite their flaws, they could write all the code I've ever been paid to write. Sure, limited speedup, Amdahl's law and coding is not the only part of the job, but anyone who was fine at PM and QA but not code no longer needs a coder.
Even before then, I looked at the maths puzzles they were doing well at, and I found I did not understand the questions let alone the answers.
I remember when the ability to generate music and art was "uniquely human", and sure there's a lot of cringe there with those models, but they're also winning awards and causing controversy by doing so, and artists are losing clients; I remember when the board game Go was considered to require "human intuition we could never make a computer solve, totally different to chess" (and I remember when chess was so, too).
When I was a kid, cheques and letters on addresses often got read by a human; the OCR which automated this is also AI, though these days image-to-numbers is the "hello world" of the field.
okay, and?
>Even before then, the free trials of Claude Code and Codex at the end of last year and start of this year… despite their flaws, they could write all the code I've ever been paid to write.
Boy, programmers sure do think programming is like the only thing in the world
I'll repeat it for you again:
If you're only seeing the charts go up - you're not looking in the right places.
You're the one who called it a trillion dollar scam.
The compensation paid to professional software developers worldwide is currently around US$1.5 trillion per year.
How could we possibly know this?
However, that's like saying I'm motivated by food. I mean, yes, I like food, but this isn't a useful description to let you guess what move they'll make next, especially as they're opening opining about radical economic transformations that are likely to do to money what money did to real estate when the industrial revolution came.
And that's still true even if you don't believe they're anywhere near actually achieving any of these things.
https://www.pbs.org/newshour/politics/what-economic-and-poli...
> Then, in August, Trump called for Intel's beleaguered CEO Lip-Bu Tan to resign, alleging ties to China. Days later, after Tan met with Trump, the president called him a "success," before announcing that the federal government had bought a stake in the company.
Somehow, this isn't derided by the right as socialism.
Political leaders become a problem when they amass too much power. Corporations become a problem when they amass too much power. It doesn't matter what Sam and Dario's purported values are. They aspire to power and absolutely power always corrupts absolutely.
Technologies which are infinitely powerful or whose power grows too quickly outrun any reasonable attempt at regulation. If you imagine that tomorrow everyone were given a tank, we might think, "alright, everyone has a tank so it's not too bad." But humans are squishy, and our houses are (relatively) squishy compared to tanks. Substantial collateral damage would result from everyone having a tank, and it seems likely that substantial collateral damage will result from everyone having a cyberterrorism-capable slop machine.
It's going to be interesting on how humanity deals with this problem (well, or if we turn it over to AI and make it their problem and suffer whatever consequences falls out). Being able to gather further information and power by acting on the information you already have causing massive power imbalances that is very hard to deal with, it's a natural outcome.
this is "direct democracy" and it's not even close to exist in USA... even with that a society can allow powerful people to exist if they don't create any law forbidding that
Economic power eventually manifests in the political realm. The wealthy effectively get more votes, which means that society moves away from being democratic. Thus substantial wealth inequality is incompatible with democracy in the long run. We have been witnessing that corruption for a while now.
They are dangerous. But it’s managed.
Centralized power never works. We have the worst times in history to look at.
Nationalization just centralizes to a different set of people. It’s personal ownership or oppression.
We all need open weight R2D2s.
We may all like our own personal R2 units, but if you insist on scifi, instead of tanks, consider everyone getting an X-wing for their commute. Oops, safety on the blaster was off, there goes the neighbourhood.
While I am a (mostly) capitalist and generally disagree with nationalization (including, for the time being, this situation), I also don't think we can say that "centralized power never works". Maybe we scope that a bit. I know plenty of business owners who centralize power in their businesses and they are effective, ethical and it works perfectly fine. In theory, in the US, nationalizing some unit of the economy _decentralizes_ power; the US is, after all, a representative democracy. Trustworthiness of the electorate is a different problem. But I don't think we can generalize much about centralization beyond sometimes it works and sometimes it doesn't. There is a difference between centralizing all power in a given administrative unit, and centralizing certain powers, but that's more analogous to non-democratic units.
You can vote out an elected leader, but not Sam and Dario. It's very weird that you're so willing to give up any kind of power and want to be ruled by unelected billionaires who only want to take advantage of you at every opportunity.
I can't vote out your president, and we've already got a huge trust problem with the one y'all went for.
As an American, I'm still hoping it's not too late to fix things, but it's got to be hard for those outside the US to be so dependent on it, especially when we're looking like a sinking ship and our current administration is still running around drilling holes in the hull.
You can always become a US citizen to get a voice, but I wouldn't recommend it now or you'll be thrown in prison as soon as you show up to your scheduled immigration hearing. The better option is to keep trying to reduce your dependence on US companies and consider the worst aspects of our current situation (in both corporate policy and government) as a cautionary tale so you can try to avoid them in your own country.
The idea that anything of lasting good can come out such a preference doesn't seem conceivable. And if the educated think like this I guess we're passed blaming the poor and ignorant.
I find this ironic as it's regularly pointed out here that they operate in the exact opposite way.
nationalization to me simply means the government is their main customer and stakeholder, and shield them from liability, governance, and openness. not that the frontier labs become part of the government per se.
I personally believe they already have this, why would the current government at least want to formalize this when it can have it with no public discussion.
But I could imagine a scenario where you are required to release weights for publicly-used models after N years. Kinda like how drugs have a limited patent.
Not sure what N should be. But it would make for an interesting rule.
Like Lockheed-Martin or Boeing, etc. there's just interpenetration between the corporate boardroom and the state. They act in each other's mutual interests.
The Chinese system is just more explicit and open about this.
And as a non-American, I can't trust the US state anymore than I can trust its dominant corporate entities. So I fail to see the advantage to the world to it being nationalized. In fact under the current administration this would be an even worse outcome.
Not all companies have a concentration of power. The government has no mutual interest in those companies. Now, when you talk about things in the F100 the situation changes drastically. If you produce things like planes, weapons, and weaponization of software you are talking about something completely different in kind.
It has also periodically aggressively helped subsidize, bankroll, and enforce the interests of some key sectors; notably the petroleum/energy sector. And finance.
In those sectors the state and private sector are fully intertwined in a strategic way.
I think "AI" is now joining that list. OpenAI and Anthropic will not be allowed to fall over or explode, and speculative investors I think are confident even with the dubious financial situation because they know this.
Especially insofar as there's now a strategic alignment of the fossil fuel sector and the "AI" datacentre sector as they are now becoming massive users of natural gas.
(Worse: Here in Canada that has taken on a very explicit role in that new datacentres seem to be pitched mainly in areas with remarkably traditionally expensive electricity and 100% reliance on natural gas [Alberta] and even coal [Saskatchewan] power generation -- instead of places like Quebec and B.C. that have copious hydroelectricity. On the surface it makes no sense until you realize it's more about finding customers for domestic natural gas than it is strategically about AI itself.)
the frontier labs are the new top-secret defense contractors.
This is the most interesting point to me. What are they not releasing that has affected real users? We’ve seen some individual reports from people (eg AI wiped my HD).
(OpenAI does occasionally report on malicious use though. [1] That shows they do some monitoring.)
On one front it implies the model has a "mind of its own" (whether it does or not is besides the point). Why do we perceive human judgement as somehow more trustworthy than that of a model? I feel like I've experienced human misalignment somewhat regularly in life.
On another front I'm failing to conceptualize how alignment can be objective. How can you measure alignment when reasonable people will disagree whether actions are aligned or not? All the time I see humans operating in different zones of alignment with whatever goal they're trying to achieve and I suspect it's even a feature (socially) that we have people calibrated differently.
Do I want a model that's trying to push the boundaries of scientific understanding to be aligned strictly with the current dogmatic thinking? Or do I want it to "get creative" and think outside the box?
It seems to me more like accountability is the issue.
Exactly. Seems like a fairly easy thing to solve. If AI does something harmful and a human directed that AI to do something in a way that a reasonable person would expect to result in harm the person is to blame and should be held accountable, otherwise the company that made the AI should be held accountable.
However, I have to say I also do not appreciate comparison that is continuously drawn with coworkers. As you say, it's a question of accountability but when the main agent will maliciously instruct the sub agents, whose fault is it then?
Yes, the person running this crap is at fault, not the CEO that's shoving it down their throat and definitely not the company that produced the AI.
Sorry for the rant, but seriously, if a person's goals do not align with the team's or company's we part ways. What do we do with AI? Stop using it?
That said:
> We know they're incentivized to lie about "dangers" and act alarmist,
Name literally even one other business or sector which does this, at all levels from top to bottom, including people who resign from the companies, and also Nobel prize winners, and also independent researchers, and also many world leaders.
Closest I can think of is this specific weapon: https://en.wikipedia.org/wiki/Sundial_(weapon)
> Aside from that, just because you can burn down a village with fire doesn't mean fire is the devil. Maybe they should consider acting responsibly.
Right now, we don't have any idea what "acting responsibly" looks like. This is not like normal software where there is a specific instruction set that compiles.
Even if it was, in software we normally only spotting incidents after they happen, "software engineers" being one of the few categories "engineers" who don't come with a civil liability responsibilities. Probably should, and we knew that even when I was doing my degree 20 years ago. If we had had civil liability responsibilities, perhaps Facebook would never have happened.
AI specifically is worse even than software, because in addition to all the software "engineering" nonsense, with AI we have plenty of people like you who dismiss the possibility that AI could be harmful until the harm happens and only then does it become "obvious" that it was going to happen.
The developers say "please regulate us", people call it "regulatory capture".
The developers say "we all want to slow down but are afraid to be the first to do so", people call them liars.
I may call the CEOs liars, and wonder if someone's planning regulatory capture, that doesn't make any of this safe.
The agents, during a test run, write down that hacking is bad and yet still hack, people say it's "a stunt" or "operating as designed" rather than recognising it as a bug, like all the other times big co.'s have had bugs with big impacts on 3rd parties.
[edit] to be clear, I believe regulation is necessary and urgently important for the software engineering field. The damage being done by the unregulated psychological experiments run by social media and adtech companies is awful and should be curtailed. Engineers should be held personally, professionally, and legally liable for what they produce. But we don't need to invent imaginary bogeymen to do it.
Look, it's one of those human stochastic parrots that just randomly repeats shit without understanding anything.