R is a different language (which is vectorized and partially functional) and has a more comprehensive stats library and better for "statsy" things, whereas matlab is better for more "mathy" things like linear algebra.
You may be thinking of Octave, not R. Unfortunately I don't know enough about Octave vs. R (or Matlab vs. S-PLUS) to comment on the similarities and differences.
How is R different than Matlab (besides the license issue)?