Maybe they are expecting something more than a worker who can keep in sync with others? If the very small team is the whole company, they may be expecting someone they can brainstorm with at random times without having to get you on Skype. Why would they want to take an extra step to keep you in sync when the rest of the in-office team already is?
Being the only person who's remote can suck, yes, and keeping one or two people in sync with an otherwise co-located team can also be a challenge (and not necessarily worthwhile for an early-stage company).
I was more wondering why more founders don't see the value in building companies where the whole team is not necessarily distributed all the time, but has the flexibility to WFH or work from the road whenever they feel like it.