Hi,
Original propolis architect here, and primary engineer for much of the effort porting bhyve to illumos.
Quite a bit was changed between "illumos bhyve" and FreeBSD upstream. We in illumos decided to take a slightly different approach in how we approached the machinery for saving/restoring the state of devices emulated in-kernel, which is a necessary part of doing such a live migration. FreeBSD was first to add that functionality, but we (I) felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware. It was at that point that illumos "soft-forked" the kernel VMM component so we could take a different approach. While the two kernel VMM components are very similar, and much of the bhyve userspace emulation (a counterpart to propolis) will run on either OS, there are some key differences around instruction emulation and those kernel-emulated devices which prevent them from being drop-in replacements for each other.
All that said, I'm happy to see FreeBSD making progress on this front. They've been a friendly peer in this space, and both projects benefit from efforts in this area.
Not really familiar with bhyve or even illumos in the slightest. But reading this comment makes me a bit sad in a way. These types of conversations and comments used to be common where I worked just few years ago. Now no one even cares how something is implemented let alone a "I felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware". The "good" ones today are looking exclusively at external properties rather than internal ones. Thank you for the comment.
> Now no one even cares how something is implemented ...
> The "good" ones today are looking exclusively at external properties rather than internal ones.
To provide a little additional context here: This effectively was an external property of the system (just to the userspace VMM). One of the significant challenges for those implementing a feature live migration is how you'll deal with multiple version of the software (older and newer) interacting in the future. A primary use case for live migration is moving workloads around to permit software updates, making that multiple version issue even "worse".
Abstracting the implementation details of those in-kernel emulated devices was helpful in two ways. Firstly, it reduced (hopefully) the chances that a later change to the internal implementation would be visible in the exported/imported state. Secondly, that the act itself of doing that state abstraction required a much more intentional and thorough examination of the emulation logic itself, resulting in a cleaner and simpler artifact.
So while it may seem like it was driven by a desire to take pride in the craft (undeniably _a_ factor), the main motivation was avoiding a versioning mess in the future.
I feel you. Good programmers still care about those things, but it turns out there are a whole lot of people who never cared about the craft and were just in the profession because it was lucrative. It's a shame.
