It's called uninstalling. The whole point of having an app installed is so that its activities / services can run or other apps can use the providers and services it has. For apps on the system partition settings will let you disable the app which will have a similar effect.
No, the whole point of having an app installed is to have an icon you can tap to bring up said app when you need it. Same way as with desktop: installing != running.
