Eventually, I decided to make my own Unity Audio Manager and kept improving it with every game I worked on. It now includes things like a GUI, one-line sound playback, fade in/out, random pitch variation, audio groups, inspector assignment, and more.
It's very versatile while still being light-weight, and I am really proud of how clean the API turned out.
I thought I'd share it here in case it can save someone else a few headaches and a lot of time. Hope it helps! :)