You might be surprised. For small structs, copying is very fast and may give you a result with better locality. Passing around structs by pointer means more pointer dereferencing, which has its own overhead and can cause cache misses.
You are correct. On a much larger project with a similar software engine, if I change all structs I pass by value to pass by reference my instruction count shoots up by 40% according to cachegrind (flto + ofast).
Amazing how powerful pass by value can be when it comes to pod structs.