There's no non-arbitrary connection between any of the colors and any of the ideas they represent, so if the user's just going to have to memorize them you might as well use symbols. At least then you can make use of more preexisting memorization and scale to whatever complexity is necessary.
Also, it's hard to predict how colors will be perceived. Between the renderer, the display, and the eye there's an unfair amount of variation. Patterns are better, but they have the same memorization problem as colors.