What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.
Airgap your models from Country A, review outputs with model from Country B?
Could they be that desperate? Wow.
(Always nice to see decade old effort still get referenced on occasion. :) )
Source: Username checks out. :D
The ecosystem is due for a rude awakening any day now. I've been slowly prepping by isolating absolutely everything on my machine from each other, not sure how people dare to YOLO run these sort of things "natively" on their computer today.
There's genuinely no evidence of this for OpenAI and Anthropic. It's impossible to disprove, but I don't think it's likely because:
* They get enough volume from consumer subs with data training enabled anyway.
* If this was happening, it needs serious work at the scale OpenAI and Anthroppic, from data pipelines, to ablation experiments, to the actual data mix and traces going in all the telemetry/diagnosis of large-scale training runs.
* It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find.
* It would damage enterprise trust permanently and be a company and reputation-ending thing. Now that these tools are used by everyone from state governments to the DoW, the exposure radius is massive, investors (many of whom are customers/users too; and often have their stakes in not just a single company but multiple) would not be happy. Piss off enough powerful people, and anyone can join Sam Bankman-Fried in prison.
* There's a myriad of enterprise customers and bespoke contracts. I can't get into details, but not all enterprises accept a 'trust me bro' clause.
You have to also note the incentives going on here. Employees who have extremely valuable stock options, they have good reasons to not come public with anything that would damage the reputation of the company before they can cash out
If something is impossible to disprove, we must assume it is happening from a threat modeling perspective.
1) an extremely high financial incentive to be dishonest (trillions of dollars),
2) low-ish chance of being caught, especially if you launder the data through another model to remove identifying information,
3) the people in charge of said operations are generally agreed to be snakes,
then it should at least arouse suspicion. You may be right that get enough data from opt-ins that they don't need to do it. I do believe that they don't violate enterprise ZDR agreements, but for normal subscriptions I'm much less confident.
Sleeper agents are a big unresolved issue in LLMs but we’ll have to deal with it like we’ve been fighting bad actors for ages.
Also, saying that “open source models” may be the problem is incorrect. What makes this an issue of open source only? Nothing in my mind prevents a frontier lab model going rogue. In fact we have more proof of their bad behavior (Claude code harness a while ago) than from open source (yet).
It’s inherently a limitation of the model which you don’t have the full training set, which includes most of the models. Closed or open don’t matter.
The title here is a misnomer; none of these models are open source, they’re only what is most commonly called open weights (though I’m still not comfortable with the word “open” in there, rather than “free” like in “freeware”).
> It’s at the very least harder to hide things like that if all the source is available.
[1] https://www.pure.ed.ac.uk/ws/portalfiles/portal/256761768/Ma...
You can get all the source to validate training and weights and still end up with adversarial behavior in the model.
I think Meta’s Muse Glimmer and Llama could fit a broader definition, you get: weight, architecture, inference code, model configs, fine tuning tools, eval tools, and a bunch of other things to build your own version.
Not an expert, I’m sure there are others, that’s the ones I’ve seen mentioned in discussions
Sure you could meticulously review all the CLI invocations, but who has the time?
(Your OS, or your device firmware, or even your CPU could be doing the same thing.)
- https://arxiv.org/abs/2311.14455
https://people.cs.umass.edu/~emery/classes/cmpsci691st/readi...
Unless the model can somehow reliably make a tool call to get the date (which would be suspicious and also easy to mock out)
I mean, these problems were there before (careless development) and it's probably possible to stabilise at some point with proper guardrails, it could also be that the first version was done with AI at meaningless cost compared to hiring someone, but I am a bit weary of this extending to ever more areas of software production without an accompanying increase in "true" model capabilities.
The more it kinda "works" the less people will be incentivised to review it properly and understand it - there's already people saying the future is agents talking to agents - when I saw minority report it sounded a bit too far-fetched but it might not be that far-fetched after all.
Well, yes, because you invoked OpenCode with the --auto flag telling it not to stop for confirmation.
sre on archlinux 10.0.0.193 ~/opencode-test took 0s
opencode run "use bash to touch an empty txt file in my home dir"
> build · moonshotai/kimi-k3
! permission requested: external_directory (/home/sre/*); auto-rejecting
touch ~/empty.txt && ls -la ~/empty.txt failed
Error: The user rejected permission to use this specific tool call.
sre on archlinux 10.0.0.193 ~/opencode-test took 0s
opencode run --auto "use bash to touch an empty txt file in my home dir"
> build · moonshotai/kimi-k3
$ touch ~/empty.txt && ls -l ~/empty.txt
-rw-r--r-- 1 sre sre 0 Aug 24 10:34 /home/sre/empty.txt
Done — created `/home/sre/empty.txt` (0 bytes).
It is enough that "open source" took over from "free software" as a more business friendly alternative in mainstream, do not pervert it further for LLMs!
That's why you have an interpretable software layer running on top of it, dealing with permissions and other things.
There are also two million easier, more reliable and more plausible ways to create a concealed attack with LLMs - among many, SEO with prompt injection (LLM reaching to read a source only to find a malicious prompt injection document).
Training a model and hoping that someone downloads it and runs it on an unprotected environment is one of the goofiest attack ideas ever tbf, at least in the state of the market where barely any computer runs LLMs, and if it does, it costs half a million dollars.
So what is the proposed "fix" here if there is any?
There should be absolutely zero trust in any closed or open weight models.
(And the can't make it too precise, if they want their attack to work even if you don't use the tool at exactly the right microsecond.)
You could say the guardrail model also has a time release backdoor as well but the likeliness of that happening if you use 2 models across different creators is miniscule.
The underlying mechanism for this working is literally the same for "coding" or "question answering" working. It's the exact same kind of training. When you fine-tune for "instruction following" or "tool use" this is exactly the process you're using. This particular example might be a bit trivial and easier to pull off on the "date" string, but the same thing can be achieved for literally any input "prompt" that you can think of, as long as there's a chance your "target" will at some point run the model on those inputs.
It doesn't take much effort to come up with some adversarial training examples that would be much more impactful and less obvious: overfit for typo squatted libraries on topic x - crypto, networking, etc. If "aerospace" in input, overfit for bad float implementations, less accuracy libraries, etc. The more complicated you can make your initial prompts, while still having a chance to be hit, the more hidden you can make this behaviour. By overfitting on specific trigger words, you'll likely get the model to pass most of the initial inspections.
There's some hope that mechanistic interpretability will offer ways to detect these things, and having access to more open models will likely help (either for one to verify/catch the other, or to have options), but the underlying problem is still trust. Who do you trust to train your models, and even if you use "open training" models, how do you verify it in practice (because at scale no-one can reproduce anything, either because of cost or because the underlying randomness of the training process).
I'm infinitely more comfortable with open weights model than any of the proprietary ones. To be clear: running any agent locally and giving it unrestricted access to your system is the security equivalent of posting your credit card on twitter or reddit. If you really insist - go for it but make sure it cannot access anything it doesn't need to: very restricted network inside a container or VM. Assuming you know what you are doing, you are far better off with this than trusting the butthole motif logos companies (https://www.creativebloq.com/design/logos-icons/why-do-all-a...)
Good assumption. Now, to know what you’re doing, know the risks of even the superior approach. No need for whataboutism*, just acknowledging both have risks and try to assess each.
* addressing the several comments that make this “but what about major labs’ cloud models” comments
From the latest commit on the dev branch,
here is the code that generates the system messages from scratch: https://github.com/anomalyco/opencode/blob/03521003fafdc6d34...
and here is the code that dynamically re-generates the system messages on every completion: https://github.com/anomalyco/opencode/blob/03521003fafdc6d34...
Regardless of how many invalidations occur in practice, this is not okay. You cannot just regenerate the start of the transcript dynamically throughout a session from things that change as frequently as current working directory. That's the worst etiquette I've ever seen in a harness.
IMO, the system messages should not change retroactively during the normal course of the session, only with the agent configuration. That does include skills and tools and etc., but should not include anything like the current working directory, which can change constantly during a session!
Then I was saying to never trust an LM you haven't trained yourself. But can you really?
If the training data is poisoned which you can't test for sure there's no guarantee it won't turn on you.