Sounds like you might be more interested in live coding. Overtone [1] can do what I think you are describing, and more--it's a frontend to SuperCollider, which has all the audio synthesis stuff you aren't interested in, but it can trigger samples fine. Of course, if Clojure isn't your thing, there are other live coding tools out there. I'm partial to Orca [2] myself.
[1] https://github.com/overtone/overtone
[2] https://github.com/hundredrabbits/Orca