There's a lot of information that is only available to MLS members. Zillow used to not have access to this information, but they slowly brokered deals with MLSes around the country to get it.
Yeah, each MLS org has its own data set which is a giant pain in the ass for the newspapers who are publishing listings for the MLSes in their areas. I don't know if they ever standardized it, but I know that one of the first tasks I had as a new dev for a newspaper back in the early 00s was to build a tool to take the data and normalize it into a single CSV.
