• Ruby: a language; now also a VM. Its language semantics can exist on other VMs and people still consider that "Ruby"; nobody really uses the VM outside of the language. So, we call Ruby a language. However, its language semantics require a lot of the VM (the ability to redefine methods of existing classes, for example), so one could say Ruby has its own Abstract Machine that a lot of VMs just happen to be adequate implementations of already.
• Erlang: a language, and a VM (itself called BEAM.) The VM's semantics are the important part; the language is just an arbitrary syntax. Other languages on the BEAM VM (or rather, the Erlang Abstract Machine, which the BEAM VM implements) all "taste" very strongly like Erlang, because the flavour of Erlang comes from the VM, not the language. (And then "OTP" is the operating system for that VM. You could write your own, but I'm not sure anyone has. All the current BEAM VM languages assume they're running inside OTP.)
• C, C++, Rust: different languages, all for the same Abstract Machine, which has no real name but assumes a "C Runtime" with POSIX-like abstractions is available. You could call it the "C Abstract Machine": it's the effective assumptions of hardware instruction-set that allow one to talk about things like "signed pointer-length integers". When something hits "undefined behaviour" in one of these languages, it's behavior undefined by the C Abstract Machine—likely because different ISAs do different things and it's too costly to try to smooth that over. (Note that the POSIX abstractions could be said to be an "abstract operating system" of sorts, one which is implemented by multiple different real C-AM operating systems.)
• Smalltalk: its own syntax, its own OS, and its own VM. All three are just known as "Smalltalk" in general, with the name of the implementation prefixed. Simple!
• Java, C#: languages that created VMs (the JVM and the CLR) that are now more important than those languages. The JVM has no operating system, just a bunch of libraries. The CLR has an implicit dependency on a lot of Windows to be its operating system; Mono and Wine have a weird relationship because of this.
• Lua: a language. The actual VM is irrelevant (there are multiple), though the "Lua abstract machine" has the interesting requirement of being an embeddable sandbox. No operating system; everything inside the VM comes in along with the application.
I learned the word from JARGON.TXT, a text file passed around BBSes in the 90s.