I attended a conference once where the invited speaker, Raul Rojas, showed that the Z3 was a universal computer, which wasn't previously believed to be the case. The basic idea of the proof was to show that you could have a single while-loop on the Z3 containing a big case-statement in the body of the loop which basically simulates a (finite) Turing machine.
The interesting part of this talk however was this: the Z3 did support neither loops nor conditional branches per se! Instead, it could just compute a series of mathematical operations encoded on a punched tape.
To simulate the case statement without the conditional branching, the trick you do is you execute every branch of the case-statement each time, but you only allow one of the branches to write the results of its computation back to the machine's memory.
But how do you get a while-loop? Well, you just glue the ends of the punch tape together! ;-)
But was it used as a general purpose machine? I think there's a difference between something that was designed and used for general purpose computation and something that was only demonstrated to be able to do general purpose computation after decades of hindsight.
Here's the Ferranti Mk 1 (the successor to the Manchester Mk 1, which was in turn the successor to the SSEM) which was the first commercially available computer playing a medley of songs: http://www.digital60.org/media/mark_one_digital_music/
"The Z1 was a mechanical computer designed by Konrad Zuse from 1935 to 1936 and built by him from 1936 to 1938."
and
"The Atanasoff–Berry Computer (ABC) was...conceived in 1937, the machine was not programmable, being designed only to solve systems of linear equations. It was successfully tested in 1942."
MIT press has an excellent book, "The First Computers- History and Architectures"[1] that discusses a number of potential "firsts" based on varying criteria- well worth a read, if you're into this sort of thing. I'd agree that the Z1 was the first complete, working machine that I consider a computer.
see http://en.wikipedia.org/wiki/Computer