Note that if you're encoding or decoding secret data, such as keys, you most likely want a constant-time encoder (example: https://boringssl.googlesource.com/boringssl/+/master/crypto...), not the fastest one.
Where a side channel attack studying the decryption speed/power use can reveal information, the encoding of said encrypted data can not as far as I ever heard about. (It will still be encrypted after base64 decoding has taken place).
Please correct me if I'm wrong.
EDIT: I see you updated your original message with different wording, please ignore this rant.
My contribution is here: https://github.com/paragonie/constant_time_encoding
Also, anyone wanting constant time implementation might just run a verified implementation through something like FaCT or Jasmin:
Encoding/decoding a value to/from base64 depends on the value, and many implementations use a lookup table or branches. While the table is small, so it fits in CPU cache thus making timing differences harder to detect, it's important for code dealing with secret values to avoid any timing differences by using simple branchless instructions that we know are constant-time on CPUs.
Most SSH hosts store their secret keys encoded in base64 (/etc/ssh/).
Then, there's JWK which supports unwrapped secret keys.
Basically, you can't guarantee that the whole world is only encoding encrypted keys. That's why it's a good software engineering practice to prevent possible attacks by using constant-time encoder/decoder. As you can see from the link in posted, BoringSSL implements one.
Finally, the context of this thread is encoding and decoding of secret data, so I'm not sure why you had to post your opinion stating that it never happens. 1) It happens in real world. 2) I mentioned a solution for the case when it happens.
There's also a problem with trust in CPU, and specifically, knowing which operations are constant time. See: https://bearssl.org/constanttime.html