You understand incorrectly, for the most part. A sinking tide lowers all boats. Buffett doesn't care what the stock market prices his company at in the short term, only how the intrinsic value of his company improves over decades.
Also, the financial press loves to take pot shots, even though Buffett repeatedly and thoroughly explains that Berkshire's stock price will go through severe beatings from time to time.
Also, the financial press loves to take pot shots, even though Buffett repeatedly and thoroughly explains that Berkshire's stock price will go through severe beatings from time to time.