I've only ever heard it referred to in the plural sense as 'elder scrolls' series, across quite a wide variety of people. This is from casual players, console version players, d&d geeks, as well as the hardcore modding communities. In almost all cases someone refers to a specific game by its subtitle though, and noted above.
I'm not sure many people would realise what you meant if you asked "Do you like the Scrolls games?", whereas s/Scrolls/Elder Scrolls/ and you'd have much higher recognition. I might try asking a few people and seeing how it actually works out, but IMO the names are sufficiently distinct as to not cause confusion.
Hmm, then I retract my assumption that "Scrolls" was clear, I was under the impression that to some extent it was similar in clarity to saying "Warcraft" and people generally knowing what you mean, or saying "Warhammer".