You don't always have the option to opt out of an HOA, especially when buying a home. If the neighborhood said home is in, and has an HOA, you're usually grandfathered in.
That's...exactly what they just said. You claimed not having an HOA is a luxury, they claimed exactly the opposite (because HOA's are generally in wealthier neighborhoods, the people who can generally afford to be pickier).