You need to go gamma -> linear before doing any linear algebra on your color values.
There is some debate over whether the alpha should be stored linearly or not in low precision (8 bit) formats. I think the standard practice is linear, but there’s good arguments it should be gamma encoded.
And, from there it’s important to point out that while pre-mul alpha is great for the math, having a color recorded in an 8-bit per channel image pre-multiplied doesn’t leave many possible values in the low range. So, you can get a lot of banding. Dithering becomes important. You should probably store the image non-premultiplied and do the premul as part of the math at runtime. Unfortunately, almost all image editors and encoders make it impossible to control the rgb values of low or zero alpha pixels. They all assume those colors conceptually “don’t exist” and don’t matter. But, they matter here because the filtered transition between solid and transparent pixels is the whole issue we are struggling with here.
There is some debate over whether the alpha should be stored linearly or not in low precision (8 bit) formats. I think the standard practice is linear, but there’s good arguments it should be gamma encoded.
And, from there it’s important to point out that while pre-mul alpha is great for the math, having a color recorded in an 8-bit per channel image pre-multiplied doesn’t leave many possible values in the low range. So, you can get a lot of banding. Dithering becomes important. You should probably store the image non-premultiplied and do the premul as part of the math at runtime. Unfortunately, almost all image editors and encoders make it impossible to control the rgb values of low or zero alpha pixels. They all assume those colors conceptually “don’t exist” and don’t matter. But, they matter here because the filtered transition between solid and transparent pixels is the whole issue we are struggling with here.