I had the same concerns about shipping speed, but then I just said fuck it and cancelled my Prime anyways. It turns out that waiting a few more days for items actually isn't a problem for me. It also brings the added benefit that I no longer feel constrained to a single shitty store, and can buy from anywhere on the internet again.
Also, when I do buy from Amazon as a non-Prime member, I find that often (not always) items tend to ship faster than the estimates claim. A few days ago I bought some RAM for my computer, and the estimate said it'd take a week to arrive. Instead, it "shipped early" and arrived in 2 days (on a Sunday). I think they've just optimized their shipping process for Prime so much that for some items it probably is cheaper to ship as fast as they can than to artificially delay shipping orders for non Prime members.
Also, when I do buy from Amazon as a non-Prime member, I find that often (not always) items tend to ship faster than the estimates claim. A few days ago I bought some RAM for my computer, and the estimate said it'd take a week to arrive. Instead, it "shipped early" and arrived in 2 days (on a Sunday). I think they've just optimized their shipping process for Prime so much that for some items it probably is cheaper to ship as fast as they can than to artificially delay shipping orders for non Prime members.