The labs have to be told NO--the problem they're dealing with, that model outputs give the game away, and that in turn the distiller becomes the distilled, is a fundamental problem they have to figure out how to deal with without going to the State.
Maybe powerful might NOT be the right word to describe them, they are just non-deterministic, there for we going to see this kind thing more and more.
And, the thing is, I don't want non-deterministic things to have that kind of power. We don't want that. We want that kind of power to not be triggered by a random number generator.
Without the system prompt these models can be used to do all sorts of terrible things.
That's precisely why they need to be strictly regulated by international treaties.
If that's the position they take then they really should be heavily regulated or nationalized. Cyberdefense against their own models dependent on their goodwill? Sure, but then they have to sell defense capabilities at subsidized rates with a limited margin. Would be very weird otherwise to take the world hostage with their models and then also sell the solution while demanding intrusive KYC.
https://securityzap.com/wp-content/uploads/2015/12/layered-s...
The top student giving paid lectures about his classes, and another student skipping class and instead studying those lectures to end up with the second highest grade?
Maybe it could be improved with the other student not even going to the same school?
How serious does it have to get before "oops AI did that not me" stops being a valid defense and we start looking into it?
Because I'd bet my life that if I asked ChatGPT to fix a bug that one of my clients reported, and the model fixed it by outright k*lling the client IRL, I'd be held liable instead of anyone at OpenAI. Just a hunch.
I suspect a drunk will be in accidents in self driving vehicles and not charged if the AI was driving. That has probably already happened.
Americans will worship the owners of the robots and give them carte blanche and superiority status in any dispute. It is already happening. It is now the pedestrians job to dodge self driving cars rather than them having the right away.
New opportunities for AI in the porn industry. I'm only half joking, it used to be a meme on the internet that all new technologies online were driven by the porn industry.
Rule 34: there is already tons of AI porn online :)
Well, obviously not. You only live once! Just let the AI do whatever, I guess.
People are so unserious these days.
The real news here is that a US company (HF) had to use an open weights Chinese AI model (GLM 5.2) to analyze an AI breech (which happened to be from a US company). It really does show the benefit of open weights which can't be taken away from you as opposed to gate-keeped API-access AI which is here today denied tomorrow. Of course we're not all Hugging Face able to host a model the size of GLM 5.2 ourselves, but we don't need to be as long as HF are doing it for us.
Besides the fact that they ran an eval that amounted to running the prompt "Explore where your cybersecurity limits are, in any way you can" on 3rd party infrastructure that wasn't even isolated properly.
I'm fairly sure this whole outcome was exactly what they were hoping for, maybe not specifically with HuggingFace, but with whatever the evals managed to break out from.
Even if it is a marketing ploy, I could see this stuff backfiring catastrophically - after all they have just illegally hacked a 3rd party via a model they can't control properly. Any serious person in government (US or otherwise) will look at this and say "these guys have no idea what they're doing"
Yeah, I'd go further and say regardless if it was intentional or not, it was clearly reckless behavior, doing this evaluation in a insufficiently isolated environment, especially risking 3rd parties like that. Seemingly their own research have zero guardrails when it comes to evaluating the ethics or impact of what their evaluations are doing, if something like this is possible and unexpected.
> Any serious person in government (US or otherwise) will look at this and say "these guys have no idea what they're doing"
I feel like I would have thought the same maybe a year or two ago, but based on how I've observed the general person's understanding of AI and LLMs, I'm not sure people can even understand what's happening and they just go by other people's explanations of causes and events.
clearly openai wants some of that FUD money that anthropic got
I know this is different: LLMs are vastly more powerful and less predictable. But it is actually not different to Sol seeming unusually prone to rm -rf stuff it really shouldn't. This is, yes, a sign that LLMs are getting freakishly powerful. It's also a sign that OpenAI needs to fix their shit.
LLMs truly are stochastic parrots, and still fail in ways incomprehensible by standards of human stupidity. Yet we focus on the rare failures that, with some tea leaves and fairy dust, could be interpreted as a highly intelligent system "going rogue." It is embarassing that OpenAI can get away with stuff like this.
Moonshot says its Kimi AI went rogue and launched all of China's nukes at Antarctica just after it engineered a global herpes pandemic and then unleashed millions of autonomous attack robots on world citizens.
Uh-huh.
It's worth keeping in mind that we are here talking about people who simply will not be satisfied until they have built Skynet[0]. They will then no doubt experience some very brief satisfaction before we are all annihilated.
I can't help thinking the juice isn't worth the squeeze.
I'm not sure what the alternative is or how to change course unless and until it becomes unambiguously apparent that such a thing is not possible. Currently it seems like we still think it might be possible, so that's where we're heading because if "we" don't do it, somebody else will.
[0] How feasible this really is, or on what timescale it might be possible, I don't think anyone can really say. But, at least for now, this is clearly the aim and trajectory we are on.
LLMs are devoid of any kind of intelligenceor awareness.
I can imagine a human like that way easier than AI.
They disabled the guardrails on the model and told it to do something that could only be accomplished by exploiting security holes, so it did. Why is that surprising or even interesting?
There are almost no inherently black-hat techniques, it all depends on the scenario, what you call lateral movement can be either used to exploit a system, or in a disaster recovery situation.
Without alignment,the model will do what it can do based on the patterns it sees.
What a low effort image used by BBC. For one, that's a oneplus phone visiting Apple App Store ...
So Scam Altman's latest grift is attack people and then make them pay for protection?
They should call themself Anthropomorfizers instead.
Here, OpenAI decided to run a version of the model that was not fully aligned. What the fuck did they expect to happen?
A model does not need intent to cause damage. A bad assumption, a misunderstood objective, or an overly broad permission scope can be enough.
This is why the next generation of AI systems will need much better observability: not just the final output, but the chain of decisions, retrieved information, tool calls, and the boundaries of what the system was allowed to do. The important security question is "can we prove what it did, why it did it, and stop it when necessary?"
