I seem to recall most RValue objects are actually smaller than 5 words and I only remember class objects being this large. I may be completely off base anyway.
It can obscure future performance enhancements, may take away cache memory from other parts of the code that are already optimized and need caching to reach full speed, and can be used as a replacement for seeking out improved algorithms, causing more structural changes down the line when performance needs to be improved and you can no longer "just add caching!
The low-level CPU caching is, as _delirium points out, highly specific to an architecture, so it's almost got to be the last thing on the list to optimize.
All this is to say, starting by looking at the cache is premature optimization most of the time.
For example, how many people write their own file system or memory based caching scheme or jump right to memcache to cache database queries, when they really could just cache the whole page using Varnish or Squid? Just understanding that there are those various levels of caching from processor to memory to filesystem to web server to opcode caching (or JIT'ing) to memcaching to database caching. A lot of developers just flat out don't understand that all of those things exist and what to do make the most of them in the right situations.