It's nearly impossible to get away from Xcode entirely. You will always need the simulator and probably some entitlement / asset tools. What you can do is write the bulk of your app as an SPM package using your editor of choice, and then include that as a local or GitHub repo dependency in your Xcode shell app (which has no code, just assets, preview assets and plist files)
I use Xcode, but just to run the app with Command-R so I can see log messages or stack traces if I need to drop it into the agent for analysis. Everything else is done by Claude Code / Codex.