I am pretty certain it was Global Illumination issues with their large always loaded world and the different indoor/outdoor transitions. They wouldn't have been using Unity or Unreal so it was probably trading off between one code only game library or writing their own, and yeah they probably just wanted to rather than fix the library.
He had help from another developer specifically for that issue.
I don’t buy that. Even back then, as an indie developer, the only reason you build your own engine is because you want to build your own engine.
You have to be doing something truly technically impressive to need a new engine and The Witness wasn’t pushing the envelope.