I don't think it's that hard. A server, in all contexts, is a thing that "serves" information or resources. The X server serves a graphical interface. A computer can serve an environment to SSH into. Pegging the definition as "a thing that renders information or resources for use" is useful, and then when more context is needed it may be given - i.e. a web server (a server that serves resources to the web), a graphic server (a server that serves a graphical interface).
