There were also concerns about running arbitrary shader code on the GPU, which has DMA access (Yes, Direct Memory Access access is a bit redundant...) to system memory, as well as the framebuffer which is in GPU memory.
The issue is that if there does happen to be some type of bug which allows an exploit, simply pointing your webgl-enabled web browser at some page would let an attacker compromise your system. This wasn't really a problem before webgl, since to get 3d graphics running at all, an attacker would have had to have code execution privileges on a victim's computer (by which time it's already too late).