They are one of the last companies we should be trusting with hosting infrastructure.
Add to that aging and wildly insecure ssh-based centralized ruby based infra management solution and I can only conclude everyone invovled has been asleep at the wheel on industry best practices for a over a decade.
Production servers should be immutable once deployed, remotely attestable at all times, with quorum signed secret management that trusts no single admin, with every workload in a dedicated vm with only a kernel, reaper, and target daemon, under an inaccessible offline base OS with TPM gated full disk encryption that fails closed if the system is ever modified after first boot.
And all of that is the bare -minimum-. Unfortunately negligence is industry standard since security engineers that actually know how to do system hardening are crazy rare in the bay.
The most grating thing is requiring every workload to be in a dedicated VM. You are throwing so much performance away. What kind of threat model would justify this? Usually there’s no threat model analysis at all.
Banning opensshd is again mostly a fool’s errand. It’s much more productive to design a system that writes the authorized_keys file read by opensshd. For example you can design a system that requires peer approval before your own key gets written to authorized_keys.
From there the spec I described writes itself. No one can ever have the ability to mutate a system once launched.
A new server spec should be defined in a signed change, it is approved with a signed review, it is compiled deterministically from source and signed by multiple independently controlled systems which both must match, and then given multiple signatures. Then any engineer can deploy those immutable images, other engineers can remotely attest the expected PCRs from the independent signed builds. Finally secrets can be released to the machine by an HSM that validates m-of-n approval signatures.
This is table stakes zero trust deployment and again, the bare minimum.
Humans manually sshing to production machines and running commands is a 90s way of doing sysadmin that is exactly why Binance was just compromised and a thousand orgs before it. Survivors bias is not a reasonable security posture.
Also with even nested virtualization on modern hardware the perf hit is minimal, though distinct servers for each workload is fine too. At -least- sandbox each workload with something like gvisor at a minimum. Linux kernel namespaces are not enough as evidenced by endless exploits.
I practice what I preach here too. I spent the last decade building this way for high risk orgs responsible for hundreds of billions of dollars in value. I also founded https://stagex.tools and co-founded https://distrust.co and https://caution.co to make it easier for others to follow these basic tactics. Much more to come soon on this. All FOSS of course.
We need adoption on these tactics to be as wide spread as TLS if the internet is going to survive an army of AI armed skiddies.
All sorts of intel you can get about all sorts of companies that way.
That is part of the entire problem in the industry. No standards and no accountability. Those with the purse strings only invest in security that is marketable or protects IP. Anyone that invested in proper remotely attestable provably quorum managed systems is going to be bragging about it and happy to show off the proof to earn trust from their customers. I do.
When no one is bragging about doing these things, it is because they are not doing them, and they just have a bunch of humans running around with god powers to execute any code they want on production systems or in binaries that will end up on your laptop.
To get a clue about their internal practices, lets see what they do when everyone is watching in public.
https://github.com/apple/foundationdb/commits/main/
No signed commits, no signed reviews. Merges done via browser. One Github API token away from being supply chain attacked. Also put one of those recent usernames into "forgot my password" and you can verify they do not mandate hardware 2FA. One compromised email account and you get to make a PR as a rando, then merge it with an Apple account and huzzah, you now have code in a critical repo used internally at Apple. Also releases are not signed or reproducible and are just cut by github actions. Another API key you could use to be even sneakier.
Almost any community maintained Linux distro does way better than this. Inexcusable for an org of their resources.
If this is how poorly Apple handles supply chain integrity in public, why would anyone believe they do any of this internally. I have also talked with multiple current and former Apple employees close to such things off the record but obviously will not out which ones.
What better evidence could I possibly provide?
Also see the other commenter in this thread who identified themselves as a former Apple contractor admitting Apple uses Jenkins. -Jenkins-.
Scroll up, I asked for evidence of this specifically. This is a pretty damning allegation and worthy of further examination given the security implications for many, many users. Frankly, whatever they're doing in unrelated public repositories is concerning in a different context but otherwise thoroughly irrelevant to this claim above
https://jobs.apple.com/en-us/details/200440680-3337/senior-l...
Puppet does not have quorum signing or approval support. By -design- it means one god server that ultimately trusts every ssh key of every admin to make changes to a fleet of -malleable- RedHat servers via ssh like it is 2005.
But for just one attack chain to humor you:
1. Malicious party gets access to one Apple engineer ssh key (Or literally just their email account since they sign nothing and the key could be swapped out temporarily)
2. They make a PR injecting a subtle malicious dependency to any of the core Apple component repos almost certainly used on prod servers on the Apple github.
3. They use their stolen key to merge the PR.
4. Code lands in next release and eventually lands in Apple production servers because nothing is signed so how would anyone know?
We learned nothing from xz.
Think of it as a predecessor to Facebook, plus whatever training platform you like. One of these guys wrote a complete software simulator for the iPod Touch-based device they were using in all the stores to handle point-of-sale transactions, so that he could include that in the training.
Content was created on macOS. Content was consumed on iPhones and iPads. The backend servers ran on Linux. It was my job to get all that compiling on the Jenkins server sitting under my desk, instead of on their laptops. I had to make minor changes to each of those code bases to get that to work.
Their build processes were quite advanced for the times. Thousands and thousands of regression tests. Thousands of UI conformance tests.
Then I had to get that Jenkins server up and running on the in-house VMware based cloud they had, where they sucked up their entire last two runs of rackmount xserve hardware and turned it around for their own internal use.
I was given a VM and remote access, and then that was that. My job was done. My contract was over, and I moved back to Austin. But that was a few years ago.
Thank you for having the balls to admit this in public.
Hopefully others can stop putting Apple on a pedestal as some gold standard for high accountability and rigorous security.
That's not a useful watermark, most of Apple's own offerings are nowhere near the iPhone revenue level.
It’s not all about what you think consumers think they want. There are other more pressing factors, like ability to deliver and supply chain management which have strategic impacts in the longer term.
A 21U rack at a datacenter (currently 2 in EU). It holds three MyElectronics 1.25U steel mounts at 2U pitch (3 Mac Minis each, rear ports, open floor for the bottom intake, 18 slots in all). 12 M5 Pro 15-core/24 GB/1 TB/10GbE NoMac Sendbox hosts at two 6-vCPU/8 GB VMs each, four M5 Pro 48 GB NoMac Simulator hosts at about 20 phones each.
Power comes from two Netio PowerPDU 8KS units (8 × C13, per-outlet switching and metering over an API) on the A and B feeds. Hosts are split evenly across the two feed and connected with C14-to-C7 cords, and an APC AP4421 transfer switch feeds only the network gear. The rack peaks around 1.3 kW and averages about 0.3 kW.
The network is a MikroTik RB5009 holdin the only public IP, doing outbound NAT and inbound WireGuard only, plus a MikroTik CRS312 giving 10GbE to every Mac on a 1 Gbit/s uplink. It has three VLANs: management (PDUs, KVMs), hosts with port isolation so they can only reach the gateway, R2 and the public internet.
Every host is commissioned with FileVault off and pmset autorestartatconnect 1, autorestart 1 and sleep 0, with auto-login on the phone hosts, hostd as a LaunchDaemon, zero-touch MDM enrollmen through Mosyle, and updates deferred 90 days.
Basically that's it. Fortunatelly, we did not have supply problems, but we are also smaller.
Sell them in units of a rack at a time with super-premium pricing. Fiddly little '4 mac minis in a 2u chassis'-type solutions are a halfway house that no-one actually wants, and are never going to provide the sort of margin that Apple expects.
I'm not building anything complicated, just a iPhone / watch app, but when I do get stuck with env its codex to the rescue. IMO relative to my experience many years ago its amazing.
"Apple has come a long way and has made a very inconvenient and complex process only slightly less inconvenient, and people somehow are happy and pretend Android sucks."
We moved off GL/GH CI for iOS apps and switched to XCode Cloud only because of the pain that Apple is w.r.t. to signing. This is the other way, and luckily we have LLMs now: https://docs.fastlane.tools/actions/match/
Whatever we pay in runner-minutes/month is the price of our sanity so we don't have to deal with the nightmare that is their signing/release process, and their stupid CLI/XCode tools. Not to mention that even XCode Cloud seems to have been coded and shipped without any kind of second pair of eyes (QA nowhere to be found), and the XCode UI and Web UI behave in totally different ways for the same input. Of course without any kind of proper explanation for any errors, as in Apple's fashion.
And when they started trying to "automate" the process, it was a roll of the dice if it would work.
I have been out of that game for last 5 years, but I occasionally still open Xcode and it _seems_ it works most of the time.
In which universe ? You generate a signing key, its password, configure the alias, the alias password and then just modify your gradle build script to say signingConfig.set(file("file/path.jks"))
Hell, if you want to go even further, give up on signing, use the debug signing certificate and let google sign for you with their own keys.
I've stopped counting how many issues you get on trying to build iOS apps because "whoops, sorry, xcodebuild just farted and can't sign"
But now normal users can have some AI thing write single-use code for them, and for relatively simple code like "show me all my pictures between these two dates that are in landscape rather than portrait" they'll usually get it right.
And now who wants a phone where you can't do anything like that?
Do you expect ordinary people to do this?
Also, what's the excuse for even having a "developer" distinction if ordinary users are expected to need it?
You mean APIs that inconsiderate developers use to release a simple calculator in the form of a 300+MB binary that eats a quarter of my RAM?
So I get why they don't care.
Not true.
https://cross-compilers.sjer.red
Sorry for the clearly vibe-coded website. I use this to build a few iOS/macOS apps for personal use.
It started a few years ago (pre-AI) when I pieced together a toolchain for cross-compiling C/C++/Fortran from a Linux host to macOS (used by my former employer at https://p3m.dev/client/#/).
Even signing/stapling/notarization can be done on Linux: https://github.com/indygreg/apple-platform-rs/tree/main/appl...
There's no good reason why these things can't be done on Linux, especially with AI.
My intent was to inform others that you do not need Apple hardware to build software targeting Apple platforms. This is a very prevalent belief as if Apple has some “magic” that cannot run on other platforms.
I did not/have not polished this project because it works for my use cases.
Again, the original version of the project is used by my former employer who uses it in their package manager.
rcodesign only made that possible circa 2022.
And it still a bunch of hacks that are only made practical because AI can stomach retrying the glue all day until it works.
And I don't see any official Apple support in their docs so this can be broken on a whim.
Just this day there was https://news.ycombinator.com/item?id=50356944
> macOS 27 does not support CoreStorage volumes
They break things on a whim.
Even if code signing stops working you can still cross-compile code for macOS which eliminates the need for a CI fleet of Apple hardware
For years, "code review" in iOS meant "email your patch set to a particular email address, which would format your patch and send it to a mailing list". That list would be populated with people who could approve or disapprove it.
The server that ran that conversion was a MacBook sitting in a pretty random "server" room that included MacBooks, iMacs facing a wall, and other Mac devices used as headless servers.
Useful more than a decade ago, still useful now!
I too have this really annoying issue where my Mac decides a USB-C or Micro-SD peripheral is no longer readable if I am working through multiple SD cards and backing them up, not sure why. Would love to work at Apple for 2 weeks with Claude to fix it and then quit and come back home, tbh would love to just work at all sorts of companies to fix really obnoxious "never to be fixed" bugs, Discord's abysmal roles stack would be on my list as well.
But if you read between the lines it is fascinating what isn't said - about Apple's customer focus.
These folks have no advocate at Apple; they worked all this out on their own. At this scale with hardware from Framework or Lenovo or Dell, there'd be a 24/7 hotline with a human to reach.
With its market power Apple can ignore these people - let them suffer, as it were.
I wonder if Ternus' Apple will.. think different.
What would be more pertinent is changing the charging logic so the packs are held at 50% state of charge vs being always charged up to 100%, that will significantly decrease the chance of pack issues while greatly increasing the longevity of the battery itself. If you hold a lithium ion cell at 50% SOC for years without varying it and keep the temperatures flat, it can last for a very long time without capacity degredation. If you leave it at 100% it will be degrade in a few short years and can swell, at which point it has twice the chemical potential to flame out. Also if it's charged to 100% when the datacenter is at 20'C ambient and datacenter cooling fails and temps go up, having the pack warm up to 40'C can overvolt it and that is usually where they swell/fail.
Also fair to note here that Apple is probably at the frontier in terms of using the highest safety cells and battery management logic so I'd feel better about this than I would putting a lesser brand's lipo pouches in datacenter environment.
A lot of words, many of which might well be true. But it ignores the fundamental point.
I think we can safely assume its not their building, not their datacentre.
As a result, they need to play by the rules of the datacentre operator.
The datacentre operator has fire suppression systems build to a certain spec.
The datacentre has risk assessments and insurance covers to think about.
The datacentre has other customers to think about.
The datacentre has its own staff to think about.
Yes, datacentres have some of the most advanced multi-layer fire detection and suppression systems in the world.
But a fire caused by multiple racks stuffed full of LiPo batteries will likely result in very hard to control results, especially if it becomes a cascading event.
Yes, it is arguably highly unlikely to ever happen, especially with Apple hardware as you say.
However at the very least the guy should fess up and have a chat to the datacentre operator to at least give them a chance to say (preferably in writing) "ok, no problem, we'll just add it to our risk register".
Its The Right Thing To Do (TM).
> except as otherwise provided in writing, signed, or issued by an authorized representative of Apple, to install, use and run up to two (2) additional copies or instances of the Apple Software, or any prior macOS or OS X operating system software or subsequent release of the Apple Software, within virtual operating system environments on each Apple-branded computer you own or control that is already running the Apple Software, for purposes of: (a) software development; (b) testing during software development; (c) using macOS Server; or (d) personal, non-commercial use.
In section 3 they also say that if you're renting access to a Mac you can only run one VM, which sounds kinda like that
> Except as expressly permitted in Section 3, the grant set forth in Section 2B(iii) above does not permit you to use the virtualized copies or instances of the Apple Software in connection with service bureau, time-sharing, terminal sharing, relay service or other similar types of services.
I.e. just because Apple requires you to do things in 24 hour increments does not mean your billing has to expose it in exactly the same way.
That doesn't mean the business has to charge the user for 24 hours. You could charge for 6 hours and then just let the Mac sit idle and unused the remaining 18.
You're eatting the cost of that unused time, but if its a differentiator that leads to more business than it might still be worth it.
EDIT: To be clear, imagine a case where the user has 24 hour sole access to the Mac. They could use up 24 hours if they wanted. But usage is tracked per hour, and they're only billed after the fact based on how much time they actually were running something during that 24 hour window.
For Amazon EC2 the case is different since it doesn't allow VMs but only possible to get dedicated bare-metal hosts. Obviously Amazon could offer VMs but they would not because the current way allows them to charge 24 hours even if user uses the host only 1 hour. And obviously no body big enough challenging the status quo and impacting AWS' revenue hard enough.
Nobody cared.
Amazon solves this by disabling FileVault and warns against enabling it.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-...
Amazon also has much tighter integration and offers the Macs bare-metal. This has the downside that you can only gets Macs as dedicated hosts, so it's another layer to manage, and that it takes around an hour to wipe the Mac whenever you shut it down, even if you're not going to release the underlying dedicated host. Also for the last year, even grabbing a Mac dedicated hosts often runs into "None available" for hours at a time. Amazon tells you "try this other zone" but even that is no guarantee. I wasn't even able to get our TAMs to tell me whether trying other regions was likely to help. Besides that, it takes 5-10 minutes to launch a Mac, and once you acquire one, there's a 24 hour minimum billing period due to Apple licensing restrictions. Which is all to say, as a single customer, you can't really build an elastically scaled Mac CI/CD farm on top of AWS. You really need to size for your peak load. Which is expensive compared to just buying your own Macs, but $former_employer didn't want to be in the datacenter business.
FWIW, I believe MacStadium was the first to offer Macs in datacenters as a service:
Plus, since the desperate are willing to do crazy-looking things like this, bulk-buying $6,000 premium laptops with batteries and screens, at enormous margins, Apple is even more incentivized to keep (not) doing exactly what it's (not) doing.
It's kinda hard to argue with their results, though. They handed the enterprise market to Microsoft 40 years ago, and have shown few-if-any signs of changing their mind.
Also you can pull one out and work on it with the integrated keyboard, trackpad, and screen if it has problems.
Obviously it’s super inefficient to need to do this, but there are some upsides.
12 MacBooks per shelf, if we say roughly 5U per shelf (21.12/4.445 rounded up), so roughly 9 shelves per 47U rack, so 100-ish MacBooks per rack.
That's a scary density of LiPo battery.
The author complains about the lack of redundant power in Mac Minis, but there are other ways to get redundant power into one AC input.
I mean, its basically the same thing in the MacBook, no ? The SMC ?
After all, there needs to be a magical chip inside to make sure you don't, for example, fry the laptop because it accepts charge via both USB-C and the MagSafe port.
Sure SMC failure is incredibly rare, but it doesn't mean the theoretical SPoF is not there.
Also when was the last time you had an internal PSU on a Mac fail ? Its been a very long time since I've ever seen one, and that's even thinking back to the old days with the old-school canister capacitors. Say what you like about Apple, but on the whole they do a decent job on their BOM selection.
Especially since they're not like a regular server, loud and power hungry. They're the definition of quiet and power sipping. I have one do all my background coding work when I want to keep my laptop clear (being headless helps with me moving around too).
Could you please share how you setup the host Mac? Which pmset settings, defaults, tips and tricks for it?
We have all the hosts enrolled into MDM, so we have all sorts of access ready: SSH, KVM, VNC, MDM. With Macs in the DC we found having all these ways to access the machines is really essential. Things break in unexpected ways.
In terms of settings, besides the obvous (do not sleep, do not auto-update, etc), disabling USB restricted mode helps a lot (Privacy&Security > Allow accessories to connect) - no need to approve connected devices via UI.
What are you building if it's not a secret?
I'm building a EU Mac mini hosting service. Very early stage, I have just bootstrapped the company.
Think 80% of Dockerfile syntax supported for running "docker build" against macOS guests. I do have some pieces already done here and there and basically it all comes down to actually creating a product out of a pile of various working scripts (OSS obviously). That's the hardest part.
And then comes the final boss of the game: marketing. Not sure anyone needs this at all.
With up to three tiers of cores (Super/Performance/Efficiency), how do you handle getting them split evenly to customers?
It has been used to build and and submit an app on a Linux machine, according to: https://x.com/JoshuaSWarren/status/2109068298537066710
This directly challenges the author's claim "You can't build iOS apps without a Mac."
omarchy-apple-dev is built on: https://xtool.sh which is "a cross-platform (Linux/WSL/macOS) tool that replicates Xcode functionality with open standards."
There's also: https://github.com/MarcoDotIO/Ycode - which even includes an experimental simulator.
Also it doesn't have iOS Simulator, so automated testing is out of question. I doubt that porting the Simulator would be feasible, since it requires macOS kernel.
Well, maybe with XNU in a VM and reimplementing parts of macOS userland. Would love to see someone try.
But honestly before we had that, we started with a system where nodes pulled whole images from each other (a decentralized algorithm without central coordination, but no chunking). It worked surprisingly well considering how simple the implementation was.
I could see them flooding the used Macbook market and selling like hot cakes, but also giving them back "rack" space.
1. Beefy M6 Mini as the server. UTM managed MacOS VMs with screen sharing enabled. 1 VM per project. Set the Mini and VMs to never go to sleep. 2. Secondary Macbook Air -- I screenshare into the VM for the project I want to work on and do work from there. I can close the laptop at any time and not worry about agents stopping. 3. With Tailscale on all machines I can work from anywhere.
Yesterday I saw AI in a pendrive, which got me wondering if small set-top boxes isolated from the web, connected to TVs or your local WiFi will be the norm of tomorrow
If I run just a UI test suite for an hour, that's $3.60/run, which is now higher than what I'm paying for AI inference that made the change that triggered the run. That's crazy! If I ran even just one run per night that would run me more than $100/month. A Mac Mini is $900 - I break even in nine months by buying the hardware outright. If I run multiple UI test suites per day, I break even much faster. These ARM chips are so power-light that the cost of electricity is practically irrelevant, and after all, these aren't rack-mounted monstrosities with 90 dB fans - they're meant to cool themselves sitting on your desk. Maybe it's different if you need like 60 of them?
Is anyone even running the numbers before they sign up? Or are companies really running that many Minis and are paying the premium to outsource the DCOps?
When you take resale value into account, purchase makes even more sense than renting.
Used M1 Mac Minis from 2020 are currently selling for about $500 when they originally retailed for $699.
The clue to their decision may be right there, potentially going for an increase in sales.
But nowadays it seems they suffer from a lack of vision and drive; perhaps (hopefully) they’ll come around.
I'm not sure this is true anymore.
[0] https://xtool.sh
I'm hoping AI will change that soon.
> But still, why laptops? Isn’t it a major waste of screen pixels and keyboard buttons and battery ions? Well yes, Mac Mini would be a better choice for a datacenter. Sure, it has very rudimentary remote management, no redundant power, and a beautiful case that ensures that cooling will be difficult regardless of which side you lay it on. But these are all known challenges that we have learned to manage.
> But have you actually tried to order a couple hundred Minis? We have. Ordered in January, delivered in July. Welcome to the world of 2026 when the supply chains are in disarray. Never mind that Apple doesn’t release updated CPU models to all devices at once. So Mac Mini didn’t get M5 cores for 10 months (and then suddenly got M6 before any other device model!). We promise the best and the newest to our customers so we had to be inventive. That’s how the MacBook-in-Datacenter project was born.
Yes.
Why else would you buy such an expensive machine with such a crappy operating system?
[1] https://www.reddit.com/r/hackintosh/comments/1uboij0/arm_is_...
https://old.reddit.com/r/hackintosh/comments/1x8j5h1/we_are_...
"A second ARM-based hackintosh has been confirmed, this time on an RK3588 board!"
https://old.reddit.com/r/hackintosh/comments/1xhtefq/a_secon...
Uptime.lab (see Insta) deshells M2sand runs them bare a bit like GitHub does
For real. Also disconnects if you wave a palm hair in its general direction. So much worse than screw-in DVI, mechanical design wise.
Though to be fair, they more-or-less ignored enterprise even when we weren't in a fab-constrained environment.
I hope you guys are the only ones employing this strategy otherwise you're just contributing to the same problem mac minis have.
People are outraged landlords doing that with housing.
Yet someone such rent seeking behaviour is acceptable?
Get fucked.