Sandboxing isn’t incompatible with a highly customizable OS. Malware is really more a question of being able to install software without the users control and the inability to remove such installations after the fact.
Windows suffers from Malware in no small part due to the systems design rather than simply being common. Plenty of alternatives have more users than windows did back in the late 90’s when it was a huge target.
I think the crux of the issue is that making a system that's customizable AND sandboxing AND user friendly multiplies together to create development and testing effort that's not palatable for most right now.
Even the open source offerings that add sandboxing often drop either the customizability or the user friendliness.
> Sandboxing isn’t incompatible with a highly customizable OS.
This is true, but there is much badly designed sandboxing. Many of the sandbox systems on Linux will have some problems, including too broad permissions, assuming text (including file names) is Unicode, failure to consider permissions that differ by command-line switches and other configuration files, failing to consider other things that the user might want to connect access to devices by files and other programs, and in case it prompts the user (at run time) for the name of another program to run by using popen or something like that and that program should run using its own permissions instead, etc. (Many of my programs that have an interactive mode do run other programs with popen, which prompt the user at run time to specify what command to run, so it can vary.)
I had my own ideas of operating system design, which is different from POSIX (although there is a POSIX compatibility layer available as a C library; the kernel need not know anything about it). One of its major ideas is that it uses capability-based security with proxy capabilities; these can be used for sandboxing and other high-level features implemented in terms of proxy capabilities. All I/O (except the Yield and Quit syscalls) requires capabilities; capabilities (as well as links) can also be passed in messages through other capabilities, including the initial message. There are many other ideas too, including ways for programs to be connected together and with user codes by user specifications. These ideas can allow working without dropping customizability, etc (in fact, the way I thought to do, would actually improve customizability, even for programs that do not normally have this feature).
Windows suffers from Malware in no small part due to the systems design rather than simply being common. Plenty of alternatives have more users than windows did back in the late 90’s when it was a huge target.