> Writing design documents, properly tracking decisions, careful planning, building infrastructure and deciding on expensive infrastructure projects, all sorts of compliance stuff.

These are important skills as a data scientist as well and as such shouldn't be new for you. I'm not sure what sort of strategy you've been employing to get any serious data science work done.

> how do I get better and find joy in the more managerial tasks of software engineering?

Maybe you don't and that is okay. You don't have to find every single part of software engineering joyous. This goes for non-managerial work as well. Not all problems are blessed with being fun. Just do it regardless.

> Does confidence just come with experience?

Yes.

> Can I get better faster?

Yes. You get faster at any task you repeat often enough.

> Or is it all worthless anyways and I should focus on the code?

Some of it, maybe, maybe not. You'll know what kinds of non-programming activities help your programming after trying them out for a while. I for example don't love design documents for every little mundane feature. However, for larger, complex tasks the act of writing something, anything at all helps me bring clarity to my thoughts. I've also found development diaries extremely useful for any project where I go more than a week between development sessions. The important point is that I found this out by testing it out and reflecting on my experience and/or results.

Do the boring parts first. I mean this seriously. You need to build personal routines of accomplishment for the more administrative tasks. Some of that can be worked in parallel with other things.

Yours is a bit of the flavorful irony.

Like Athelstan out of the convent, uncertain if he is ready for the world.

Sir, the world is not ready for you.

“The real deal” is a pit of grime friend. Not a clean sheet of paper we may dream upon.

Sure those come along if you can get in on a fresh bootstrap.

Everything else is a hack upon a hack upon a hack from where it began.

And the true artist screws anything up as strategically less as possible. While getting things DONE.

You're better of being good at pickup sticks than math to do an engineers’ job.

Seriously now, do you want career advice from a stranger?

Go work for the government. They need you.

Give them ten years of your grind, not with the intention of being a wage slave forever. Rather…

A private contractor.

Get your paperwork in order and there will be a day you can bill $400/hr, though typical industry rate hovers around $200/hr. For anything capable. Everyone in the end NEEDS capable.

That’s for the guy who can make ideas look like visible constructs which map to business process. Which is sort of where you are now at the start.

Only no one will listen to you without the ten year grind.

And in that time you will be short order cook on an item list driven by an upstream appetite who doesn’t have your attention span. Like everyone else. Except academia. You might as well think of those as the socialists of technocracy.

Don't use AI.