Since Android signing keys are just PKCS #8, and GPG keys are supported by most HSMs, a HSM would definitely be usable (even if you just used an addon HSM card that you added to your "release terminal"). Unfortunately in order to safely use the HSM you'd need to re-generate your keys again from within the HSM -- which obviously is a problem on Android. In addition, HSMs are quite expensive and might be prohibitively so in your case. But I would definitely recommend looking into it if you're really stuck on doing distribution yourselves.
Reproducible builds are a useful thing separately, but using a HSM doesn't require reproducible builds -- after all signing a hash of a binary is the same as just signing the binary. The main benefit of reproducible builds is that people can independently verify that the published source code is actually what was used to build the binary (which means it's an additional layer of verification over signatures).
One question I have is how are going to handle the case where the release terminal fails? Will you have to (painfully) rotate the keys again?
I.e. we are already using HSMs on the build server.