One thing I found systemd really confusing was its treatment of ExecStop in a service script. ExecStart is the command to run when systemd starts the service at system boot up (or when a user tells systemd to start the service). However, ExecStop is run when the starting command has finished running. You have to set RemainAfterExit=yes to have the desired function of running the stop command on system shutdown or on user stopping the service. ExecStop is basically the "on-cleanup" event rather than "to-shutdown-the-service" event.
It is important to keep in mind that systemd is tailored towards daemons. So if your service just runs a command that eventually exits, you need to explicitly tell systemd to treat it differently than a daemon.
Edit: As others noted, you’re probably looking for oneshot + RemainAfterExit.
It is a little asymmetric, because 'ExecStart' is actually normally 'Executable that is the service', not just script that starts the service, but I think that's a hangover from the self-daemonizing approach to init scripts.
Then ExecStop is basically on-cleanup, not to-stop. ExecStart really is to-start, not on-start. In the httpd server case, ExecStart runs the httpd command.
I think part of the problem is with how one "naturally" or "habitually" thinks of a Service. From systemd's perspective/terminology, the Service is the thing that starts and stops. But whether because it is inherently more intuitive, or because of how daemons traditionally worked on *nix, the mind tends to think of the process which the Service starts as the thing which starts and stops. I'm not able to double check currently, but I also think that systemd isn't totally consistent with the mental model it bases the choice of keywords on when it comes to ExecStop, because with that mental model one would expect ExecStop to only run when the Service is stopped from systemd, but I'm fairly sure it runs in other cases when the process started with ExecStart exits, but the service isn't necessarily stopped. I could be wrong about that though.
If that helps you understand it better, then sure.
All I am trying to say is that the name of the option makes sense as-is: it literally runs the provided command(s) on service stop (“execute on stop”).
Similarly, ExecStart literally means command(s) to execute on start.
If the command runs a blocking daemon/server (usually the case), then the server will be implicitly stopped by definition - because if you’re stopping the systemd service, you’re interrupting any commands that are still running/blocking.
This was ultimately what I needed to do when I wrote a systemd service that managed some firewall rules. It really was a footgun though, what with having essentially different meanings/purposes for ExecStop whether you’re doing a Type=forking, a Type=oneshot, or a Type=oneshot with RemainAfterExit=yes.
And relatedly, I honestly have no idea when I’d want to use ExecStartPre, or multiple ExecStarts, or ExecStartPost, and so on.
I would argue the semantics of ExecStop are always the same. It's the command that's executed to stop the service. On the other hand, what it means for a service to be "running" or "stopping" naturally depends on what type of service it is (i.e., is it a daemon or not?)
Yes, so whether the service is stopping as a result of the process exiting, or whether you requested the service to stop manually, it will run the ExecStop in either case.
That makes sense to me personally. What would be the more intuitive design in your mind?
Stopping as a result of the process exiting or requested the service to stop are two very different things. Systemd overloads the term ExecStop for different semantics, relying on different property settings. That's where the confusion comes from.
The name sounds like it means "this is how I want you to cause the service to stop" to me (and clearly to others as well). That would be symmetrical with ExecStart meaning "this is how I want you to cause the service to start". If it runs after the service stopped it should be called "ExecAfterStop" or something like that.
That is what ExecStop means. It specifies how you want to cause the service to stop. But the lifetime of the service isn't exactly the same thing as the lifetime of the process that got started in ExecStart.
Maybe think about it this way: ExecStart is what the system will run to transition the service from the "starting" state to the "started" state. ExecStop is what the system will run to transition the service from the "stopping" state to the "stopped" state.
For a service with RemainAfterExit=no (the default), you enter the stopping state right away once the processes that got started in ExecStart exit. That's useful when you are starting some long lived process as a service, and in that case there is usually no need for an ExecStop. But semantically, ExecStop has the same meaning either way -- it's what needs to be run, if anything, to transition the service from the stopping state to the stopped state.
I have now found the documentation for ExecStop (in systemd.service(5)), which hopefully improves my understanding.
It definitely seems to be both "cause to stop" and "after (unexpected) stop" in one. You can look at $MAINPID to see which case you have. This design apparently makes sense to you, but to me and several others in this thread a service that has already stopped isn't in need of being stopped and shouldn't execute commands intended for that. (There is a separate ExecStopPost for "after stopping, for any reason".)
Yes. ExecStart works the same for all the cases. ExecStop works differently though. While ExecStart is the event to kick off the command for the service, ExecStop is not. The asymmetric semantics are where the confusion comes from.
It's been enlightening to me to read through some of the distro-provided .service files to see what can be done, with services I'm more of less familiar with.
One thing I found systemd really confusing was its treatment of ExecStop in a service script. ExecStart is the command to run when systemd starts the service at system boot up (or when a user tells systemd to start the service). However, ExecStop is run when the starting command has finished running. You have to set RemainAfterExit=yes to have the desired function of running the stop command on system shutdown or on user stopping the service. ExecStop is basically the "on-cleanup" event rather than "to-shutdown-the-service" event.