I've done a lot of both, and what I've found is that in person works better if the company isn't "remote first". Being one or a few remote people sucks, lots of stuff happens in the office without you. But having the right tools for remote work makes it in some cases better.
But even in the case of an office, when everyone goes remote, suddenly it's not so bad.
At reddit, since when I was there everyone else was just out of college, we always worked the last two weeks of the year from home because we had "winter break". So for those few weeks, we were a 100% remote company, and it worked out ok. BUT, we didn't make any big decisions or do any deep collaboration because even though everyone was remote we waited till we got back to the office, not because it was better but because we weren't set up to do it remotely.
It's really all about having good processes for remote work -- ie. as much async as possible. And once you do that, even if you go to an office later, you've get better systems set up for tracking what happened and seeing history for new employees.
The worst thing for a new employee is having to get up to speed when all the meetings were in person and not recorded.
We have 2 "offices" 1 in SF and 1 in tokyo. It's completely optional whether you come in or not. Most people work from home. Our office in SF and Tokyo is mainly for entertaining customers and meetings.