The magic of VLANs then allows putting everything on one physical interface for the router. This essentially turns the switch into a port multiplexer for the firewall (10GBe comes in handy there). Also useful in case the router is a VM, since only a single NIC needs to be passed to it.
Although, if it's just the Linux box and the NAS on 10/8 (or whatever), you could connect linux:eth2 to nas, perhaps? There's a lot about this particular network that hasn't been described.
There are a number of features available to Make It Work The Way You Want but they're often nonobvious. As a sibling comment notes, you can set the arp_announce and arp_ignore (and arp_filter) sysctls to instruct the kernel to answer or not answer for arp requests based on interface specifics (the other arp_ sysctls are related to gratuitous arp sending and receipt).
As in this post, you can use the rp_filter sysctl to change the reverse path filtering (as in RFC3704).
Additionally, you can use iproute2 rules and route tables to enforce outbound paths if you want a node to have multiple interfaces on the same subnet without arp confusion.
sysctl -qw net.ipv4.conf.all.arp_announce=2
sysctl -qw net.ipv4.conf.all.arp_ignore=1
Even for failover of a cluster VIP, which is needlessly strict. That's the strangest ARP behaviour I've seen in a while.
Also, if you're on Azure, setup two VMs in the same subnet and watch ARP between them. They actually aren't in the same subnet and there's something doing Proxy ARP between them with a MAC like 01:23:45:67:89:AB or something like that. Cursed.
Also, you can't spoof source IP there, even when talking between your VMs on a private subnet. So forget about running a router inside a VM. On AWS you can. 
We haven't even got to the joys of the load balancer yet!
Just log onto the VM console to troubleshoot. Oh, they don't have a console by design, okay then.
How does VyOS do it?
modem <-> router/NAT <-> switch <-> (wifi and other stuff)
I tend to agree that if you are trying to slice an Ethernet switch (or network) for two networks that you don't want to be bridged together, you should probably use VLANs.
Setting rp_filter=1 is very standard for linux routers and almost all distros set it by default. Both Debian and Red Hat set it by default. I am not sure if OP accidentally disabled this or if he is using some silly distro.