It happens to be a standard ARM instruction, Apple pushes some bits into ACTLR_EL1 (Auxiliary Control Register, EL1, "Provides IMPLEMENTATION DEFINED configuration and control options for execution at EL1 and EL0") in the kernel on context switch. The DTKs used a proprietary register and touched it using msr, but again, no custom instructions.
Apple does in fact ship custom instructions on their silicon, but where those are used, how they work, and how ARM lets them get away with it is a story for another day :)
Lemme just look up TSO and...
https://github.com/saagarjha/TSOEnabler
...Oh. Fair enough, my mistake :P
Is there not an instruction to switch into TSO mode, though? Wouldn't that technically count? :P