> That’s the defense. And here’s the problem: it’s often hard to refute with confidence.
Why is it necessary to refute it at all? It shouldn't matter, because whoever is producing the work product is responsible for it, no matter whether genAI was involved or not.
It's negligence, pure and simple. The only reason we're having this discussion is that a trillion dollars was spent writing said scripts.
You'd lose that lawsuit in a heartbeat.
I don't know what kind of drivers education you get where you live but where I live and have lived one of the basic bits is that you know how to park and lock your vehicle safely and that includes removing the ignition key (assuming your car has one) and setting the parking brake. You aim the wheels at the kerb (if there is one) when you're on an incline. And if you're in a stick shift you set the gear to neutral (in some countries they will teach you to set the gear to 1st or reverse, for various reasons).
We also have road worthiness assessments that ensure that all these systems work as advertised. You could let a pack of dogs loose in my car in any external circumstance and they would not be able to move it, though I'd hate to clean up the interior afterwards.
Who's liable?
I think this would be a freak accident. Nobody would be liable.
> Who's liable?
You are. It's still your dog. If you would replace dog with child the case would be identical (but more plausible). This is really not as interesting as you think it is. The fact that you have a sentient dog is going to be laughed out of court and your neighbor will be in the docket together with you for attempting to mislead the court with your AI generated footage. See, two can play at that.
When you make such ridiculously contrived examples turnaround is fair play.
But you would be liable for civil damages, because that does not. There are multiple theories for which to establish liability, but most likely this would be treated as negligence.
There was a notorious dark web bot case where someone created a bot that autonomously went onto the dark web and purchased numerous illicit items.
https://wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww.bitnik.or...
They bought some ecstasy, a hungarian passport, and random other items from Agora.
>The day after they took down the exhibition showcasing the items their bot had bought, the Swiss police “arrested” the robot, seized the computer, and confiscated the items it had purchased. “It seems, the purpose of the confiscation is to impede an endangerment of third parties through the drugs exhibited, by destroying them,” someone from !Mediengruppe Bitnik wrote on their blog.
In April, however, the bot was released along with everything it had purchased, except the ecstasy, and the artists were cleared of any wrongdoing. But the arrest had many wondering just where the line gets drawn between human and computer culpability.
For most crimes, this is circular, because whether a crime occurred depends on whether a person did the requisite act of the crime with the requisite mental state. A crime is not an objective thing independent of an actor that you can determine happened as a result of a tool and then conclude guilt for based on tool use.
And for many crimes, recklessness or negligence as mental states are not sufficient for the crime to have occurred.
If I build an autonomous robot that swings a hunk of steel on the end of a chain and then program it to travel to where people are likely to congregate and someone gets hit in the face, I would rightfully be held liable for that.
That idea is really weird. Culpa (and dolus) in occidental law is a thing of the mind, what you understood or should have understood.
A database does not have a mind, and it is not a person. If it could have culpa, then you'd be liable for assault, perhaps murder, if you took it apart.
We as a society, for our own convenience can choose to believe that LLM does have a mind and can understand results of it's actions. The second part doesn't really follow. Can you even hurt LLM in a way that is equivalent to murdering a person? Evicting it off my computer isn't necessarily a crime.
It would be good news if the answer was yes, because then we just need to find a convertor of camel amounts to dollar amounts and we are all good.
Can LLM perceive time in a way that allows imposing an equivalent of jail time? Is the LLM I'm running on my computer the same personality as the one running on yours and should I also shut down mine when yours acted up? Do we even need the punishment aspect of it and not just rehabilitation, repentance and retraining?
At worst, you would confer liability to the platform, in the case of some sort of blatant misrepresentation of capabilities or features, but absolutely none of the products or models currently available withstand any rational scrutiny into whether they are conscious or not. They at most can undergo a "flash" of subjective experience, decoupled from any coherent sequence or persistent phenomenon.
We need research and legitimate, scientific, rational definitions for agency and consciousness and subjective experience, because there will come a point where such software becomes available, and it not only presents novel legal questions, but incredible moral and ethical questions as well. Accidentally oopsing a torment nexus into existence with residents possessed of superhuman capabilities sounds like a great way to spark off the first global interspecies war. Well, at least since the Great Emu War. If we lost to the emus, we'll have no chance against our digital offspring.
A good lawyer will probably get away with "the AI did it, it wasn't me!" before we get good AI law, though. It's too new and mysterious and opaque to normal people.
If you gave a loaded gun to a five year old, would "five-year-old did it" be a valid excuse?
Purely organically of course.
Yeah that's exactly the I think we should adopt for AI agent tool calls as well: cryptographically signed, task scoped "warrants" that can be traceable even in cases of multi-agent delegation chains
Now if you did all of that, but made a bug in implementing the filter, then you at least tried and wasn't negligible, but it's on you.
Ah, I get it. So the token can be downscoped to be passed, like the pledge thing, so sub agent doesn't exceed the scope of it's parent. I have a feeling, that it's like cryptography in general -- you get one problem and reduce it to key management problem.
In a more practical sense, if the non-deterministic layer decides what the reduced scope should be, all delegations can become "Allow: *" in the most pathological case, right? Or like play store, where a shady calculator app can have a permission to read your messages. Somebody has to review those and flag excessive grants.
The LLM can request a narrower scope, but attenuation is monotonic and enforced cryptographically. You can't sign a delegation that exceeds what you were granted. TTL too: the warrant can't outlive its parent.
So yes, key management. But the pathological "Allow: *" has to originate from a human who signed it. That's the receipt you're left holding.
You're poking at the right edges though. UX for scope definition and revocation propagation are what we're working through now. We're building this at tenuo.dev if you want to dig in the spec or poke holes.
> Agent Trace is an open specification for tracking AI-generated code. It provides a vendor-neutral format for recording AI contributions alongside human authorship in version-controlled codebases.
The new and interesting part is that while we have incentives and deterrents to keep our human agents doing the right thing, there isn't really an analog to check the non-human agent. We don't have robot prison yet.
The workflow of starting dozens or hundreds of "agents" that work autonomously is starting to gain traction. The goal of people who work like this is to completely automate software development. At some point they want to be able to give the tool an arbitrary task, presumably one that benefits them in some way, and have it build, deploy, and use software to complete it. When millions of people are doing this, and the layers of indirection grow in complexity, how do you trace the result back to a human? Can we say that a human was really responsible for it?
Maybe this seems simple today, but the challenges this technology forces on society are numerous, and we're far from ready for it.
When orchestrators spawn sub-agents spawn tools, there's no artifact showing how authority flowed through the chain.
Warrants are a primitive for this: signed authorization that attenuates at each hop. Each delegation is signed, scope can only narrow, and the full chain is verifiable at the end. Doesn't matter how many layers deep.
People want to use a tool and not be liable for the result.
People not wanting to be liable for their actions is not new. AI hasn't changed anything here, it's just a new lame excuse.
AI is just better because no one can actually explain why the thing does what it does. Perfect management scapegoat without strict liability being made explicit in law.
The software world has been very allergic to getting anywhere near the vicinity of a system like that.
Licensure alone doesn't solve all these ills. And for that matter, once regulatory capture happens, it has a tendency to make things worse due to consolidation pressure.
AI is worse in that regard, because, although you can't explain why it does so, you can point a finger at it, say "we told you so" and provide the receipts of repeated warnings that the thing has a tendency of doing the things.
And when sub-agents or third-party tools are involved, liability gets even murkier. Who's accountable when the action executed three hops away from the human? The article argues for receipts that make "I didn't authorize that" a verifiable claim
If you don't want to be responsible for what a tool that might do anything at all might do, don't use the tool.
The other option is admitting that you don't accept responsibility, not looking for a way to be "responsible" but not accountable.
Had it worked then we would have seen many more CEOs in prison.
The same may happen to AI or not. We can bite the bullet and say it's fine that it sometimes happens. We can ban the entire thing too if we feel the tradeoff not worth it
In practice, almost everyone is held potentially or actually accountable for things they never had a choice in. Some are never held accountable for things they freely choose, because they have some way to dodge accountability.
The CEOs who don't accept accountability were lying when they said they were responsible.
Such so magic forcefield exists for you, though.
No, it's trivial: "So you admit you uploaded confidential information to the unpredictable tool with wide capabilities?"
> Who's accountable when the action executed three hops away from the human?
The human is accountable.
----
A computer can never be held accountable
Therefore a computer must never make a management decision
Now companies are all about doing bad all the time, they know they're doing it, and need to avoid any individual being accountable for it. Computers are the perfect tool to make decisions without obvious accountability.
That's an orthodoxy. It holds for now (in theory and most of the time), but it's just an opinion, like a lot of other things.
Who is accountable when we have a recession or when people can't afford whatever we strongly believe should be affordable? The system, the government, the market, late stage capitalism or whatever. Not a person that actually goes to jail.
If the value proposition becomes attractive, we can choose to believe that the human is not in fact accountable here, but the electric shaitan is. We just didn't pray good enough, but did our best really. What else can we expect?
If one decided to paint a school's interior with toxic paint, it's not "the paint poisoned them on its own", it's "someone chose to use a paint that can poison people".
Somebody was responsible for choosing to use a tool that has this class of risks and explicitly did not follow known and established protocol for securing against such risk. Consequences are that person's to bear - otherwise the concept of responsibility loses all value.
What if I hire you (instead of LLM) to summarize the reports and you decide to email the competitors? What if we work in the industry where you have to be sworn in with an oath to protect secrecy? What if I did (or didn't) check with the police about your previous deeds, but it's first time you emailed competitors? What if you are a schizo that heard God's voice that told you to do so and it's the first episode you ever had?
Frankly, I think that might be exactly where we end up going. Finding a responsible person to punish is just a tool we use to achieve good outcomes, and if scare tactics is no longer applicable to the way we work, it might be time to discard it.
It's scary that a nuclear exit starts looking like an enticing option when confronted with that.
Twelve–year–olds aren't capable of dealing with responsibility or consequence.
That value proposition depends entirely on whether there is also an upside to all of that. Do you actually need truth, meaning, responsibility and consequences while you are tripping on acid? Do you even need to be alive and have a physical organic body for that? What if Ikari Gendo was actually right and everyone else are assholes who don't let him be with his wife.
I'm not claiming to have all the answers about how to achieve that, but I am fairly certain punishment is not a necessary part of it.
[ $[ $RANDOM % 6] = 0 ] && rm -rf / || echo "Click"
# terrible code, never use ty
def cleanup(dir):
system("rm -rf {dir}")
def main():
work_dir = os.env["WORK_DIR"]
cleanup(work_dir)
At what point is it negligent?
Though rm /$TARGET where $TARGET is blank is a common enough footgun that --preserve-root exists and is default.
It really doesn't. That falls straight on Governance, Risk, and Compliance. Ultimately, CISO, CFO, CEO are in the line of fire.
The article's argument happens in a vacuum of facts. The fact that a security engineer doesn't know that is depressing, but not surprising.
That's a very subtle guinea pig joke right there.
"Three months later [...] But the prompt history? Deleted. The original instruction? The analyst’s word against the logs."
One, the analysts word does not override the logs, that's the point of logs. Two, it's fairly clear the author of the fine article has never worked close to finance. A three month retention period for AI queries by an analyst is not an option.
SEC Rule 17a-4 & FINRA Rule 4511 have entered the chat.
With no amount of detailed logging makes "the AI did it" a valid excuse.
It's just a tool.
It's like blaming a loose bolt in a Boeing 737 on "screwdriver did it".
On the contrary, if they just owned up to it, chances are I wouldn't even write them up once.
This way, they can avoid being legally blamed for stuff-ups and instead scapegoat some hapless employee :-) using cryptographic evidence the employee "authorized" whatever action was taken
AI is just branding. At the end of the day it's still just people using computer software to do stuff.
There is going to be a person who did a thing at the end of the day -- either whoever wrote the software or whoever used the tool.
The fact that software inexplicably got unreliable when we started stamping "AI" on the box shouldn't really change anything.
> A computer must never make a management decision, because a computer cannot be held accountable.
And if a professional wants to delegate their job to non-deterministic software they're not professionals at all. This overreliance on LLMs is going to have long-term consequences for society.
Read the terms and conditions of your model provider. The document you signed, regardless if you read or considered it, explicitly removes any negative consequences being passed to the AI provider.
Unless you have something equally as explicit, e.g. "we do not guarantee any particular outcome from the use of our service" (probably needs to be significantly more explicitly than that, IANAL) all responsibility ends up with the entity who itself, or it's agents, foists unreliable AI decisions on downstream users.
Remember, you SIGNED THE AGGREMENT with the AI company the explicitly says it's outputs are unreliable!!
And if you DO have some watertight T&C that absolves you of any responsibility of your AI-backed-service, then I hope either a) your users explicitly realize what they are signing up for, or b) once a user is significantly burned by your service, and you try to hide behind this excuse, you lose all your business
One in which you sell yourself into slavery, for example, would be illegal in the US.
All those "we take no responsibility for the [valet parking|rocks falling off our truck|exploding bottles]" disclaimers are largely attempts to dissuade people from trying.
As an example, NY bans liability waivers at paid pools, gyms, etc. The gym will still have you sign one! But they have no enforcement teeth beyond people assuming they're valid. https://codes.findlaw.com/ny/general-obligations-law/gob-sec...
"But the AI wrote the bug."
Who cares? It could be you, your relative, your boss, your underling, your counterpart in India, ... Your company provided some reasonable guarantee of service (whether explitly enumerated in a contact or not) and you cannot just blindly pass the buck.
Sure, after you've settled your claim with the user, maybe TRY to go after the upstream provider, but good luck.
(Extreme example) -- If your company produces a pacemaker dependent on AWS/GCP/... and everyone dies as soon as cloudflare has a routing outage that cascades to your provider, oh boy YOU are fucked, not cloudflare or your hosting provider.
Sure, if someone from GCP shows up at your business and breaks your leg or burns down your building, you can go after them, as it's outside the reasonable expectation of the business agreement you signed.
But you better believe they will never be legally responsible for damages caused by outages of their service beyond what is reasonable, and you better believe "reasonable outage" in this case is explicitly enumerated in the contact you or your company explicitly agreed to.
Sure they might give you free credits for the outage, but that's just to stop you from switching to a competitor, not any explicit acknowledgement they are on the hook for your lost business opportunity.
Sure, but not all liability can be reassigned; I linked a concrete example of this.
> But you better believe they will never be legally responsible for damages caused by outages of their service beyond what is reasonable, and you better believe "reasonable outage" in this case is explicitly enumerated in the contact you or your company explicitly agreed to.
Yes, on this we agree. It'd have to be something egregious enough to amount to intentional negligence.
but the person who turned it on can
simple as
Licensed professionals are required to review their work product. It doesn't matter if the tools they use mess up--the human is required to fix any mistakes made by their tools. In the example given by the blog, the financial analyst is either required to professional review their work product or is low enough that someone else is required to review their work product. If they don't, they can be held strictly liable for any financial losses.
However, this blog post isn't about AI Hallucinations. It's about the AI doing something else separate from the output.
And that's not a defense either. The law already assigns liability in situations like this: the user will be held liable (or more correctly: their employer, for whom the user is acting as an agent). If they want to go after the AI tooling (i.e., an indemnification action) vendor the courts will happily let them do so after any plaintiffs are made whole (or as part of an impleader action).
It is the burden of a defendant to establish their defense. A defendant can't just say "I didn't do it". They need to show they did not do it. In this (stupid) hypothetical, the defendant would need to show the AI acted on its own, without prompting from anyone, in particular, themselves.