That being the case my understanding is splitting this project up is not only a technical challenge but a legal one.
You could make a common API for driver code, this would make the main driving force of linux much weaker.
Nvidia tried to stay out of tree for years and was dragged kicking and screaming upstream because they needed the reliability.
I'm saying the very thing you are calling out as a problem is currently what is forcing everyone contribute to the mainline kernel.
Without centralized leader ship the facturing of the linux kernel would be a disaster.
No one is forcing you to use Linux any more than anyone is forcing you to use Windows or macOS.
"...The recent fiasco involving Linus Torvalds running Linux like the privileged man that he is and entirely missing the central points of the arguments made by people who don't enjoy his kind of privilege..."
It was his 'privilege' that led to the technical competency of the project that is now enjoyed by many.
Attacking someone's 'privilege' is the lowest form of argument.
Anyway, Linux is a technocracy. Actually, a technical meritocracy to be more precise, rather than economic or political dominance.