I’ve come to see it as a consequence of the industry’s pivot toward every firm of every size becoming more enterprisey. The enterprise habit of using 3rd-party vendors in particular.

15, 20 years ago, me and my T-shaped skills could get by just fine with a database server, a NAS, and some frank conversations about priorities.

Nowadays it’s not enough to to know how to work with databases and files. There’s also all these platforms: databricks, snowflake, new relic, sentry, datadog, GitHub actions, prefect, dbt, kubernetes, cassandra, elastic, and on and on and on. Each brings a new set of idiosyncrasies, its own terms for things, its own enterprise integration headaches. It’s just a hell of a lot of knowledge to ask one person to cram into their head on top of whatever skills and domain expertise form the actual core of their job.

That makes sense for an enterprise. Enterprises have a lot of bodies to throw at the problem, so they can have an expert for each tool who’s available to support everyone else. And they like to have support contracts and SLAs, which you can get from à B2B vendor but not from the KISS principle. And they don’t mind great heaping piles of unnecessary and mutually conflicting abstractions because that sort of thing is just SOP for a large bureaucracy.

But if you want - need - a system that one person with T-shaped skills can reasonably fit into their head, this is not the way.

> Yeah, as a "T-shaded data folk" (have worked as a data analyst, data scientist, data engineer, and machine learning engineer, so I think I'm close to a full bingo care) in data, I agree!

Me too. There were no data engineers when I was a baby data scientist (15 years ago now).

I think that this was a terrible idea (the splitting out of roles). I remember once being asked in an interview if I was able to pull my own SQL or would I need an analyst. That kind of split always struck me as absurd, given that the only way to build a decent model is to know the data very well first..

> That said, I think we have a culture problem in defining our roles narrowly, I've seen data scientists look down at data-pipeline parts of their job, and ignoring model latency and deployment; data engineers be deeply uncurious around what happens to the data after they put it in a warehouse; etc.

I honestly feel like this is driven by the titles. I will note that most of the unsuccessful/bad data scientists I've met were the ones who only cared about the models. As I always said to people, I can run sklearn in a loop over model families if that's all you bring.

> The best people I've worked with, within data and elsewhere, are the ones who are curious about every aspect of the stack, even if they mostly focus on one side.

This is probably just a good signal in tech in general, some curiosity is really required to be successful in this industry.

This sounds more like a culture problem than a title problem though. Are you saying you can't even open PRs with a pipeline proposal? Your data engineering team is playing emperor and wants to protect their kingdom at all cost

Me too. We could do with a few more people but I would not want them to be overly specialised.

You get a new title, analyst engineer!

I have a different take on this. I have a friend who used to work at a bank as an analyst, and he had a hardcore finance/math background, and was a self-taught R/Python/SQL coder before this whole 'big data' thing became popular a decade ago.

Back then his bank started building a data science/engineering dept, and he jumped ship, and essentially did the same work for 2x the salary. Let me tell you, understanding Black-Scholes and what the data actually means and how to source it, as well as what finance people actually are looking for is a skill that's 1000x as difficult to get as learning R/Python/Pandas etc.

Once upon a time they were called DBAs.

There's typically a role (data engineer?) where one is essentially a fancy DB admin that extracts and centralizes all data into db tables as per how the analysts want. Sometimes this is local and sometimes a data pool of s3 buckets. This is harder than it sounds where depending on the business and how fast it needs to action data you are likely skilled on low latency networking and fast algorithms to transform and load data from all kinds of hard to extract places like other proprietary software. They also have you adding desired analytics like PostGIS for location-based aggregations, designing all the backups, and some companies have 100+ pipelines as many proprietary data sellers exist that give an end of day dump or real-time fire hose you have figure out how efficiently capture and store.

An analyst then loads this up with whatever visualization software or 3rd party BI tools and interprets the findings and I've found this varies from giving a simple presentation after AI tools do everything for them to doing full on R Studio analytics such as these models https://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/

If the analyst then selects a feature set from the data and performs machine learning I've found they are called 'Data Scientists' otherwise they are analysts but these titles mean something different everywhere.

> As long as we do not hand over our collective decision making power to the machine god, which I don't think is a good idea and am not sure is even possible,

Counterpoint: This is not only possible but has effectively already happened. From the outside, the current trajectory is indistinguishable from humanity serving the AI, pulling all possible resources out of literally everything else in order to burn them on AI R&D, training and inference.

To your point, I wonder if organizations will trend towards hiring 3rd party companies to check work so they have someone to blame and can "focus on innovation". It's already a popular tactic to hire consultants for risky business decisions. Perhaps this is a new prediction: companies get leaner while outsourcing the checks to the same people they fired.

It's not new, but especially after the release of Opus 5.5 it's been sticking in my mind lately.

My view, as someone who was flat out "AI is a hype scam" up until around Jan this year. It was hype for a while, and forms of it are still very much hype, but the reality is catching up with the hype.

I.e the reality has so far been ~2 years behind the hype.

I'm not saying I like these companies, or that I think the economics of any of them is sane. Just saying LLM's are capable.

edit. For some context, I only use open weight models these days and rather hope that big closed firms go bust.