I don't think you necessarily need to optimize before you make the code "clean" but you should at least understand what it will take to make it performant. This is a stronger requirement than it might sound like since most programmers are quite bad at predicting what will perform well without profiling first.
So knock together a prototype and profile that to understand what the bottlenecks are in your program, then you have an informed baseline of how the program needs to fit together to function well.
So knock together a prototype and profile that to understand what the bottlenecks are in your program, then you have an informed baseline of how the program needs to fit together to function well.