I think it makes sense in a "plain conversation" sort of way: when asking a computer for something is as plain as asking a friend to pass the remote.
To push a button, then speak in to the device, then watch it recognize it, then wait for a result is a ridiculous interaction. I Have found it useful when driving, where I can push a button and say "Navigate Home" (android) and the GPS fires up and starts directing me. But the part where I have to hit the "listen" button first is an annoyance and distraction.
I absolutely feel that a voice-only UI would be plain stupid. I'm saying using voice as an additional interface while simultaneously using other interfaces. If I could have a conversation with my computer while working, I could see it being quite useful and efficient.
Say I'm writing something, and while still finishing typing a sentence, say aloud "find me synonyms for [word]" and then keep typing while the system finds a list and opens them up in a background tab in my browser with a "ding", where I can say "go on" when I'm ready to hear them or see the tab. Even fancier if it highlights that word and shows the list, which I can click and have it replace.
Or maybe while coding, I could hit the save button, and while mousing over some files to figure out what to work on next, I can say "commit changes with message 'some commit message' and deploy". Open my next file and carry on (Of course "deploy" would include the automated tests and builds and so on).
These example aren't amazing, but the point I'm making is that it only makes sense when it works conversationally as an addition rather than a replacement.
And it has to be as if I were speaking to someone in the room. No buttons; No little microphone with a meter next to it; No recognition graphics, spelling out what I said. I just say what I need and it lets me know, subtly when there are results ready, which I can deal with when I'm ready to deal with them.
Don't interrupt me. Don't show me that you're thinking. Stay out of my way and help me do things when I ask you to and then listen intently in case I need something else.
In that respect, no, I don't think it would be silly at all, and yes, I absolutely would like to speak to computers in such a way.