While that is a much simpler solution and quite a nice writeup, it doesn't quite solve the problem that townscaper poses. For that it needs to be infinitely expandable, which is quite a different feat.
Let’s say we have 12 iron mines with known amounts of iron ore supply, and 10 steel foundries with known amounts of demand. For each (mine, foundry) pair, there is a known cost of shipping the ore from point A to B. Which mines should send how much ore to which foundries?
AIUI, that is the general “optimal transport problem”. Now replace ore with colors and it applies to the problem above :)
"Flowers replaced to red" looks like it also tinted the whole sky red. In general, all the image captions imply that the algorithm is able to identify objects (flowers, ice cream scoop, shirt) and only change that object's color.
I is the flattened image (i.e., a matrix where each row represents the RGB values of a single pixel)
T and N are some matrices, basically of appropriate dimensions such that the transformation you quoted results in an image of the same shape as the original and also such that we can optimize certain objectives with respect to the variables that define the transformation.
The why this is the desired transformation is a tricky question, it's definitely not the only and quite probably not the best way to formulate this problem - just the first way that I tried and worked reasonably well. I tried to explain my intuition for this in the article after the definition.
> So is "T" and "N" conventionally used in numerical optimization as the transform "variables" (like X and Y are in calculus)?
Not really, numerical optimization offers a general toolbox for various problems, which are usually defined based on domain-specific notation and concepts. So the concept of a transformation like the one described in the article is not general to numerical optimization, but specific to the optimization being done here. That being said, when talking about techniques, theorems and algorithms in numeric optimization there are conventions. So you usually move between the more theory-oriented notation to a domain-specific notation (in the article I mostly used the domain-specific, lazy notation I chose).
Bear in mind that my perspective is not very academic so I might feel less strongly about (and be less aware of) rigorous mathematical conventions.
Thanks! Yeah, I think one thing the simplistic optimization is missing is preserving the "texture" of the image. There are various numeric ways to represent textures so this is something that potentially could be added to the objective, it was out of scope for the project though. Might give it a try some day!
Unrelated, the author has a very nice take on generating the organic grid from Oskar Stålberg’s Townscaper game [2].
[1]: https://codimd.math.cnrs.fr/s/2eRBqV9zl
[2]: https://andersource.dev/2020/11/06/organic-grid.html