To put it briefly: for many programs the style is extremely elegant and compositional; for others it's just plain awkward. In those cases it's not just hard to read for beginners; it's hard to come up with the program and maintain it.
Factor has lots of great ideas, but takes them way too far! The object system is supremely awkward IMO. It's basically as if you grafted an object system onto bash. Yeah you could imagine some way it can work. But both the syntax and semantics just clash with the original paradigm. And you should just use a different language if you want the OO paradigm.
The concatenative style works better as a little language than for general purpose programming.