Talk of the singularity reminds me of the furor over human cloning - it's an excellent generator of fantasy-lit plot coupons, late-night drunken dorm-room philosophy, and paranoid fundamentalist ravings, but in real life it's much more boring. I know lots of human clones - they're called "identical twins", and they are little different from anyone else. And, speaking as a strong AI that is composed of trillions of microscopic robots working in tandem, it's hard to be impressed by yet another strong AI like me. There's well over six billion of us already, and that's being awfully ungenerous to chimps, gorillas, dogs, and my pet parrot. If I really wanted another strong AI, wouldn't it be cheaper and easier just to stop using birth control? Or to go to the pound and adopt a friendly but unwanted puppy?
To pick on a different example: it is the height of human egotism to believe that our scientists can create a "grey goo" that would scare a bacterium, let alone me. Bacteria have been dividing every few minutes for several billion years. They cover every inch of the earth, and compose large portions of the soil. As I type this, billions of bacteria are actively trying to digest me. It's no big deal. We grew up with this problem and have coped with it all our lives, as did our ancestors - and I'm not talking about our African ancestors, our shrewlike ancestors, or our fishlike ancestors, but about our algal mat ancestors. We've got a lot of experience, and the bacteria have even more. In a battle between a human-designed replicator and a cubic foot of soil bacteria, I'm going to bet on the machine that's had a longer design cycle.
"it's hard to be impressed by yet another strong AI like me" - I think the point is that it won't be like you but orders of magnitude smarter.
"As I type this, billions of bacteria are actively trying to digest me. It's no big deal" - there are several bacterium that are very successful at digesting people which are a big deal and worth worrying about.
Actually it may be a big deal. Fairly recently scientists have determined that some very stupid behaviors that some animals engage in, to their own detriment, is the direct result of toxins produced by parasitical bacteria. So think of all the stupid things humans do to their own detriment as a species. Whose to say whether we do these things consciously ... or as the result of being manipulated to our detriment by bacteria ... or whether or not millions of computers are subtly affecting our thinking with electro-magnetic frequency manipulation ...