It felt utterly hopeless but this might reignite my hope for continuing running on ~xenserver.
run virt-manager, create vm, select your graphics card, and it boots
However Xenserver (and VMware) supports GPU sharing between multiple virtual machines, essentially GPU virtualization. Both Nvidia and AMD have custom solutions for this. I believe this is what the ancestor comment is using on Xenserver.
There is work-in-progress support for GPU sharing in Linux KVM as well, although currently I think it's restricted to Intel. If you're interested in that, I would recommend going with AMD, since they maintain a high-quality driver in Linux itself (unlike Nvidia who only maintains a proprietary out-of-tree driver), and thus is much more likely to be supported by KVM.
Not quite, Nvidia actually forbids the use of their consumer level cards being used this way and actively tries to deter it by detecting the use of a hypervisor in the drivers and refusing to initialize the card. There are ways to either work around or defeat this detection, either by using patched drivers that remove the check or by. Having the virtual machine hide it's presence from the guest system which can have a performance impact. AMD does not try to prevent such uses of their consumer level cards and are more likely to work out of the box.
So I also used PCI passthrough. I have two GPUs that I pass through meaning that I can run two virtual machines with proper graphic cards (and then regular VMs as I please). The idea was that instead of dual-booting between two machines I run them both at the same time. To switch between them I just select another input on my monitors.
Unfortunately though, it is not that simple.
This gets easier for every year and it was about 2 years since I last played with this and setup my workstation so things could have changed a bit. But even for PCI passthrough you have to be very careful selecting your GPU, motherboard and CPU. The drivers need to play along and the CPU with motherboard need to be able to isolate everything adequately. Whether AMD or nVidia is best is/was up to debate, each GPU generation has its own quirks.
For CPU you ideally you wanted a i7 or a non-entry-level Xeon to be as safe as possible, this is not a requirement but it is very possible that with a "regular" CPU you might not be able to pass through the devices you need to get it to work. More here: http://vfio.blogspot.se/2015/10/intel-processors-with-acs-su...
Bottom line, it can work great but it is a lot to setup and even if you do your research you might end up in a situation where it just doesn't work. Oh, and the hypervisor you use might screw you, removing the very features you depend on.
This may change now with looking Glass giving a fast way to share the frame buffer between host and guest, https://github.com/gnif/LookingGlass
Something like that would be very appealing (from a user experience, security I wouldn't now) in Qubes OS.
I love the way it works, its overall architecture. Very clean and reliable feeling. A bit like a kubernetes for VMs.
Install in your favourite distro, assign it a network bridge (eg: br0) and a lvm volume group, and done. All nodes in a cluster do replicate the configurations, and can become the master node at a moment's notice.
would be great. the backup situation on xenserver compared to vmware/hyper-v sucks. or to say it differently veeam gui is kinda strange, but it's a really good backup tool.