when will appengine support setting environment variables with gcloud command or inside the gcloud console instead of in the app.yaml, i.e. so that each environment can have their own variables without copying app.yaml's?!
(App Engine PM here) At this time, app.yaml is the only way to set env vars. If you want to separate environments, you could use different .yaml files, I personally use app.staging.yaml and app.prod.yaml
You definitely don't want to check secrets into the repo for the same reason you don't set them as environment variables: It's not secure. The solution is to use Cloud Key Management Service. More info here: https://cloud.google.com/kms/ We use that to either store secrets directly, or to decrypt encrypted entities in a DB (e.g. Datastore).
Neat, looks like they're using a solution similar to what we're doing, except instead of using Cloud KMS to encrypt information stored in Google Cloud Storage, we're putting our information in Datastore.
The reason being that these secrets are absolutely essential for many tasks that our registry needs to perform, and the past six years of experience have shown us that Datastore has better availability than GCS. We haven't seen Datastore ever go down unless all of Cloud is down too, whereas we have seen outages isolated to just GCS. Datastore also has lower latency (since it's a DB, not bulk file storage).
One caveat is that the maximum size of a single entity in Datastore is 1 MB -- if you're encrypting stuff larger than that then you'll need to shard (ugly) or just use GCS. Since none of our secrets are anything close to that large, it works just fine.