First, I'll assume alpha works similar to a compiler. It parses the text, and generates an AST for the input expression. Your goal is to replace the tree with the 'best' possible equivalent AST. Best here is a fitness function, that provides a score for a given AST. At first glance, we'll say the AST with the smallest number of nodes is optimal.
Then, create a list of legal operations on a subtree. Try an operation, and see if the AST gets better. Repeat.