I'm genuinely curious about this topic and love learning. My fundamental belief is that it makes sense to build highly tuned models that specialized in specific areas and using orchestration on top to call out to these specialized models with a "shared memory". This would allow for the ability to improve parts of the system without the fear of regression in other areas and it would be cheaper to create a smart "platform". I know there's models with MoE etc but feel like the "microservices" approach (micromodels) would work well here.

I could be totally wrong.