Short version: "It turned out our chosen base address for the FMC (FPGA Mezzanine Card) bus had a default memory type of Normal Cached."

They accidentally put an external non-memory device behind the cache. That's never going to work right, but might work some of the time.

Harkens me back to Sun/Sparc hardware and Solaris.

For the uninitiated, Service Processor (SP) is just another kind of Baseboard Management Controller.