When you try to open a file with Firefox it will first try to map the file to a mimetype using the ExternalHelperAppService (https://developer.mozilla.org/en-US/docs/How_Mozilla_determi...). In case a mimetype is found, a file dialog is shown so you can open the file with the right application, in case it is not, the contents of the file will be displayed in the browser. In this case my OS provided the ExternalHelperAppService with a mimetype for one of my public keys with the .pub file extension: application/vnd.ms-publisher. Of course that's not the correct mimetype for the public key file, but that's basically what saved me by showing a file dialog because it found a mimetype. All other files had no file extension so no mimetype was found.
I also discovered that my private keys were all encrypted with a passphrase so even though they have been compromised it was not as bad as I initially believed.