Because it requires people to know how to read or specify code.
Spec driven development is similar to the no-code movement i.e. the specs are closer to functional requirements than this strict spec or code.
You can argue that you can just ask the LLM to explain back and forth but it can still be too much.
But then you'd get an explanation tailored to your specific perspective instead of a generic one, right?
Assuming you know how to ask the right questions and make sense of it. Tailored to whose perspective, especially in a team, which then might make it worse.