>We then perform all releases from a dedicated isolated release terminal.
>We physically store the device securely.
Why didn't they go with a HSM?
> The signing keys (hardware or software) are kept exclusively on this device.
We still want to make very sure that the build environment itself hasn't been tampered with, hence keeping the build machine itself isolated too.
A much better approach would be to use reproducible builds and sign the hash of a build with a hardware key, but we didn't want to block an improved build setup on reproducibilizing everything.
Edit: we may be missing an HSM trick, though, in which case please elaborate :)
Since Android signing keys are just PKCS #8, and GPG keys are supported by most HSMs, a HSM would definitely be usable (even if you just used an addon HSM card that you added to your "release terminal"). Unfortunately in order to safely use the HSM you'd need to re-generate your keys again from within the HSM -- which obviously is a problem on Android. In addition, HSMs are quite expensive and might be prohibitively so in your case. But I would definitely recommend looking into it if you're really stuck on doing distribution yourselves.
Reproducible builds are a useful thing separately, but using a HSM doesn't require reproducible builds -- after all signing a hash of a binary is the same as just signing the binary. The main benefit of reproducible builds is that people can independently verify that the published source code is actually what was used to build the binary (which means it's an additional layer of verification over signatures).
One question I have is how are going to handle the case where the release terminal fails? Will you have to (painfully) rotate the keys again?
I.e. we are already using HSMs on the build server.